Support broadcast segments #14789
Merged
Support broadcast segments #14789
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
YongGang
reviewed
Aug 9, 2023
| @@ -34,6 +34,7 @@ public class DruidK8sConstants | |||
| public static final String TASK_JSON_ENV = "TASK_JSON"; | |||
| public static final String TASK_DIR_ENV = "TASK_DIR"; | |||
| public static final String TASK_ID_ENV = "TASK_ID"; | |||
| public static final String LOAD_BROADCAST_SEGMENTS_ENV = "LOAD_BROADCAST_SEGMENTS"; | |||
There was a problem hiding this comment.
should this be loadBroadcastSegments?
There was a problem hiding this comment.
this is a env variable so i think it should match the other env variables
| @@ -224,7 +224,11 @@ private Collection<EnvVar> getEnv(Task task) | |||
| .withValueFrom(new EnvVarSourceBuilder().withFieldRef(new ObjectFieldSelector( | |||
| null, | |||
| StringUtils.format("metadata.annotations['%s']", DruidK8sConstants.TASK) | |||
| )).build()).build() | |||
| )).build()).build(), | |||
| new EnvVarBuilder() | |||
There was a problem hiding this comment.
My understanding is we what to do things similar to this
Is setting an env variable enough?
There was a problem hiding this comment.
this needs to be passed to the java druid internal peon command as a flag, and in the PodTemplateTaskAdapter, we don't actually control how the user configures that command.
But right now there's no way to even see whether the flag should be set, since we set the environment variable here, the user can read it and choose to set --loadBroadcastSegments
suneet-s
approved these changes
Aug 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since the pod template adapter does not generate the internal peon ... run command itself (the command is passed in the pod templates), we need a way to account for the optional --load-broadcast-segments flag in CliPeon for tasks that are queryable.
Description
Based on the task being launched, pass whether the task can handle queries as a environment variable to the job launched by the pod template adapter. The operator can then configure their k8s command in the pod template to include this env variable in the internal peon run command.
Release note
Allow the PodTemplateTaskAdapter to account for queryable tasks.
Key changed/added classes in this PR
This PR has: