Fix fastparquet tests to work with HDFS #9583
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes NVIDIA#9545. This commit fixes the `fastparquet` tests to run on Spark clusters where the `fs.default.name` does not point to the local filesystem. Before this commit, the `fastparquet` tests assumed that the parquet files generated for the tests were written to local filesystem, and could be read from both `fastparquet` and Spark from the same location. However, this fails when run against clusters whose default filesystem is HDFS. `fastparquet` can only read from the local filesystem. This commit changes the tests as follows: 1. For tests where data is generated by Spark, the data is copied to local filesystem before it is read by `fastparquet`. 2. For tests where data is generated by `fastparquet`, the data is copied to the default Hadoop filesystem before reading through Spark. Signed-off-by: MithunR <[email protected]>
|
Build |
revans2
approved these changes
Oct 31, 2023
| @@ -167,6 +203,8 @@ def test_reading_file_written_with_gpu(spark_tmp_path, column_gen): | |||
| There are xfails here because of fastparquet limitations in handling Decimal, Timestamps, Dates, etc. | |||
| """ | |||
| data_path = spark_tmp_path + "/FASTPARQUET_TEST_GPU_WRITE_PATH" | |||
| local_base_path = (spark_tmp_path + "_local") | |||
There was a problem hiding this comment.
I think this is okay, but it does assume that the temp path in HDFS is going to be available in the local file system too. I don't think there is anything we can do about that though.
|
I've merged this change. With any luck, this should unblock the CDH builds. |
|
Alternative: fastparquet uses fsspec that provides a bunch of implementations out of the box https://github.com/fsspec/filesystem_spec/tree/master/fsspec/implementations . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #9545.
This commit fixes the
fastparquettests to run on Spark clusters where thefs.default.namedoes not point to the local filesystem.Before this commit, the
fastparquettests assumed that the parquet files generated for the tests were written to local filesystem, and could be read from bothfastparquetand Spark from the same location. However, this fails when run against clusters whose default filesystem is HDFS.fastparquetcan only read from the local filesystem.This commit changes the tests as follows:
fastparquet.fastparquet, the data is copied to the default Hadoop filesystem before reading through Spark.