Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spark plugin] Fix environment variable ValueFrom for pod templates #4532

Merged

Conversation

Tom-Newton
Copy link
Contributor

@Tom-Newton Tom-Newton commented Dec 5, 2023

Tracking issue

Closes #4530

Why are the changes needed?

To support using ValueFrom for environment variables configured in pod templates.

What changes were proposed in this pull request?

Put Container.Env from the pod template into SparkPodSpec.Env instead of SparkPodSpec.EnvVars.

This means all the ValueFrom features of corev1.EnvVar should be supported.

How was this patch tested?

  • Updated the existing unittests to catch the bug.
  • I've been using these changes in a real flyte deployment.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly. - its just a bug fix so I don't think any documentation needs updating.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Dec 5, 2023
@Tom-Newton Tom-Newton changed the title Fix environment variable env from for spark pod templates [Spark plugin] Fix environment variable env from for pod templates Dec 5, 2023
Copy link

codecov bot commented Dec 6, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (aff1150) 58.89% compared to head (1849e48) 58.89%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4532   +/-   ##
=======================================
  Coverage   58.89%   58.89%           
=======================================
  Files         620      620           
  Lines       52458    52458           
=======================================
  Hits        30895    30895           
- Misses      19095    19096    +1     
+ Partials     2468     2467    -1     
Flag Coverage Δ
unittests 58.89% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@pingsutw pingsutw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 6, 2023
@Tom-Newton Tom-Newton changed the title [Spark plugin] Fix environment variable env from for pod templates [Spark plugin] Fix environment variable ValueFrom for pod templates Dec 6, 2023
@hamersaw hamersaw merged commit 94d79f5 into flyteorg:master Dec 6, 2023
40 of 41 checks passed
pvditt pushed a commit that referenced this pull request Dec 13, 2023
…4532)

* Update tests

Signed-off-by: Thomas Newton <[email protected]>

* Use Copy Container.Env to SparkPodSpec.Env instead of SparkPodSpec.EnvVars

Signed-off-by: Thomas Newton <[email protected]>

* Fix list initialisation

Signed-off-by: Thomas Newton <[email protected]>

---------

Signed-off-by: Thomas Newton <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] [Spark plugin] Environment variables in the pod_template that use ValueFrom don't work
3 participants