Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement][Worker] Startup parameter should have the highest priority #13274

Merged
merged 5 commits into from
Dec 29, 2022
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/en/guide/parameter/context.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ The result of Node_mysql is as follows:

![context-log03](../../../../img/new_ui/dev/parameter/context_log03.png)

Even though output is assigned a value of 1 in Node_A's script, the log still shows a value of 100. But according to the principle from [parameter priority](priority.md): `Local Parameter > Parameter Context > Global Parameter`, the output value in Node_B is 1. It proves that the output parameter is passed in the workflow with reference to the expected value, and the query operation is completed using this value in Node_mysql.
Even though output is assigned a value of 1 in Node_A's script, the log still shows a value of 100. But according to the principle from [parameter priority](priority.md): `Startup Parameter > Local Parameter > Parameter Context > Global Parameter`, the output value in Node_B is 1. It proves that the output parameter is passed in the workflow with reference to the expected value, and the query operation is completed using this value in Node_mysql.
Radeity marked this conversation as resolved.
Show resolved Hide resolved

But the output value 66 only shows in the Node_A, the reason is that the direction of value is selected as IN, and only when the direction is OUT will it be defined as a variable output.

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/en/guide/parameter/priority.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ DolphinScheduler has three parameter types:

The user can define part of the parameters when creating workflow definitions.

As there are multiple sources of the parameter value, it will raise parameter priority issues when the parameter name is the same. The priority of DolphinScheduler parameters from high to low is: `Local Parameter > Parameter Context > Global Parameter`.
As there are multiple sources of the parameter value, it will raise parameter priority issues when the parameter name is the same. The priority of DolphinScheduler parameters from high to low is: `Startup Parameter > Local Parameter > Parameter Context > Global Parameter`.

In the case of upstream tasks can pass parameters to the downstream, there may be multiple tasks upstream that pass the same parameter name:

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/zh/guide/parameter/context.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Node_mysql 运行结果如下:

![context-log03](../../../../img/new_ui/dev/parameter/context_log03.png)

虽然在 Node_A 的脚本中为 output 赋值为 1,但日志中显示的值仍然为 100。但根据[参数优先级](priority.md)的原则:`本地参数 > 上游任务传递的参数 > 全局参数`,在 Node_B 中输出的值为 1。则证明 output 参数参照预期的值在该工作流中传递,并在 Node_mysql 中使用该值完成查询操作。
虽然在 Node_A 的脚本中为 output 赋值为 1,但日志中显示的值仍然为 100。但根据[参数优先级](priority.md)的原则:`启动参数 > 本地参数 > 上游任务传递的参数 > 全局参数`,在 Node_B 中输出的值为 1。则证明 output 参数参照预期的值在该工作流中传递,并在 Node_mysql 中使用该值完成查询操作。

但是 value 的值却只有在 Node_A 中输出为 66,其原因为 value 的方向选择为 IN,只有当方向为 OUT 时才会被定义为变量输出。

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/zh/guide/parameter/priority.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ DolphinScheduler 中所涉及的参数值的定义可能来自三种类型:
* [上游任务传递的参数](context.md):上游任务传递过来的参数
* [本地参数](local.md):节点的自有变量,用户在“自定义参数”定义的变量,并且用户可以在工作流定义时定义该部分变量的值

因为参数的值存在多个来源,当参数名相同时,就需要会存在参数优先级的问题。DolphinScheduler 参数的优先级从高到低为:`本地参数 > 上游任务传递的参数 > 全局参数`
因为参数的值存在多个来源,当参数名相同时,就需要会存在参数优先级的问题。DolphinScheduler 参数的优先级从高到低为:`启动参数 > 本地参数 > 上游任务传递的参数 > 全局参数`

在上游任务传递的参数中,由于上游可能存在多个任务向下游传递参数,当上游传递的参数名称相同时:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,13 +103,11 @@ public String curingGlobalParams(Integer processInstanceId, Map<String, String>
Map<String, String> resolveMap = new HashMap<>();
for (Map.Entry<String, String> entry : entries) {
String val = entry.getValue();
if (val.startsWith(Constants.FUNCTION_START_WITH)) {
String str = "";
if (val.contains(Constants.FUNCTION_START_WITH)) {
String str = val;
// whether external scaling calculation is required
if (timeFunctionNeedExpand(val)) {
str = timeFunctionExtension(processInstanceId, timezone, val);
} else {
str = convertParameterPlaceholders(val, allParamMap);
}
resolveMap.put(entry.getKey(), str);
}
Expand Down Expand Up @@ -174,14 +172,16 @@ public Map<String, Property> paramParsingPreparation(@NonNull TaskInstance taskI
if (MapUtils.isNotEmpty(localParams)) {
globalParams.putAll(localParams);
}

if (MapUtils.isNotEmpty(cmdParam)) {
globalParams.putAll(ParamUtils.getUserDefParamsMap(cmdParam));
}
Iterator<Map.Entry<String, Property>> iter = globalParams.entrySet().iterator();
while (iter.hasNext()) {
Map.Entry<String, Property> en = iter.next();
Property property = en.getValue();

if (StringUtils.isNotEmpty(property.getValue())
&& property.getValue().startsWith(Constants.FUNCTION_START_WITH)) {
&& property.getValue().contains(Constants.FUNCTION_START_WITH)) {
/**
* local parameter refers to global parameter with the same name
* note: the global parameters of the process instance here are solidified parameters,
Expand All @@ -191,8 +191,6 @@ public Map<String, Property> paramParsingPreparation(@NonNull TaskInstance taskI
// whether external scaling calculation is required
if (timeFunctionNeedExpand(val)) {
val = timeFunctionExtension(taskInstance.getProcessInstanceId(), timeZone, val);
} else {
val = convertParameterPlaceholders(val, params);
}
property.setValue(val);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,6 @@ public void testCuringGlobalParams() {

String result6 = dolphinSchedulerCuringGlobalParams.curingGlobalParams(1, globalParamMap, globalParamList,
CommandType.START_CURRENT_TASK_PROCESS, scheduleTime, null);
Assertions.assertTrue(result6.contains("20191220"));
Assertions.assertEquals(result6, JSONUtils.toJsonString(globalParamList));
}
}