[opt] support orc merge multi stripes io #244
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
morningman
changed the title
Merged
16 tasks
morningman
pushed a commit
to apache/doris
that referenced
this pull request
Nov 7, 2024
…ripes. (#42004) ### What problem does this PR solve? When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 #### Summary: ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; # xxx is the size in bytes ``` ### Release note Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <[email protected]> Co-authored-by: daidai <[email protected]>
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Nov 7, 2024
…ripes. (apache#42004) When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; ``` Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <[email protected]> Co-authored-by: daidai <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.