Append HADOOP_CONF_DIR to the tools CLASSPATH execution cmd #1308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ahmed Hussein <[email protected]> Fixes NVIDIA#1253 Fixes NVIDIA#1302 This change includes the following: - the python wrapper pulls the hadoop configuration directory `$HADOOP_CONF_DIR` env var. If the latter is not defined, the wrapper tries `$HADDOP_HOME/etc/hadoop`. - If the `hadoop_conf_dir` is defined then it is appended to the java CLASSPATH iff it is a valid local directory path - If none of the above applies, the class path will be the same.
amahussein
added
bug
Signed-off-by: Partho Sarthi <[email protected]>
Signed-off-by: Ahmed Hussein <[email protected]>
tgravescs
previously approved these changes
Aug 22, 2024
Signed-off-by: Ahmed Hussein <[email protected]> Fixes NVIDIA#1303
parthosa
reviewed
Aug 22, 2024
| except Exception as ex: # pylint: disable=broad-except | ||
| self.logger.error('Failed in processing output arguments. Output_folder must be a local directory') | ||
| raise ex | ||
| # self.output_folder = FSUtil.get_abs_path(self.output_folder) |
| deps_arr = [self.prop_container.get_jar_file()] | ||
| hadoop_cp = self._get_hadoop_classpath() | ||
| # append hadoop conf dir if any | ||
| if hadoop_cp is not None: |
There was a problem hiding this comment.
Should this throw an exception when hadoop conf dir was not found but event logs are in hdfs?
Signed-off-by: Ahmed Hussein <[email protected]>
tgravescs
reviewed
Aug 22, 2024
Signed-off-by: Ahmed Hussein <[email protected]>
tgravescs
approved these changes
Aug 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Ahmed Hussein [email protected]
Fixes #1253
Fixes #1283
Fixes #1302
Fixes #1303
This change includes the following:
The wrapper gets Hadoop's configuration directory from the environment variables. The first valid directory is added to the java cmd CLASSPATH. The order of available hadoop configuration directories are:
This PR also enforces URI to the
--output-folderargument to the java cmd. This is required to prevent the core tools from storing the output-folder on the remote storage in case HDFS defines a default FileSystem.