SparkSubmit: Adding propertyfiles option #36164
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pateash
changed the title
pateash
force-pushed
the
34838-Adding-propertyfiles-option
branch
2 times, most recently
from
425c16c to
07aaf5d
Compare
|
cc. @ghostp13409 |
Introduce a new parameter 'properties_file' in SparkSubmitHook and SparkSubmitOperator. This allows loading of extra properties from a specified file, defaulting to conf/spark-defaults.conf if nothing is specified. The goal is to provide greater flexibility for Spark configuration. The changes have been tested for the affected files.
Updated the alt text of an image in the example_twitter_README.md file for better accessibility and SEO. The change provides a more detailed description of the screenshot, enhancing understanding for visually impaired users and search engines."
pateash
force-pushed
the
34838-Adding-propertyfiles-option
branch
from
769c92d to
ca439a6
Compare
| @@ -33,7 +33,7 @@ | |||
| 1. Save Summary to MySQL | |||
|
|
|||
| ***Screenshot:*** | |||
| <img src="http://i.imgur.com/rRpSO12.png" width="99%"/> | |||
| <img src="http://i.imgur.com/rRpSO12.png" width="99%" alt="Dag for Collecting tweets for four users account"/> | |||
|
|
|||
There was a problem hiding this comment.
Is this change related to the main goal of this PR?
There was a problem hiding this comment.
No, this was a pre-commit failure.
as all the images need an alt-text, may be due to addition of this new rule
https://github.com/DavidAnson/markdownlint/blob/main/doc/md045.md
potiuk
approved these changes
Dec 11, 2023
84 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #34838
Description
There is an option to override the default conf/spark-defaults.conf properties in Spark-submit
We want to have that option in SparkSubmitOperator and Hook as well.