Quicker shared cache file preallocation #79447
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ywelsch
added
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
Once tests passed we should use |
| } | ||
| } | ||
| } catch (final Exception e) { | ||
| logger.warn(new ParameterizedMessage("failed to pre-allocate cache file [{}] using native methods", cacheFile), e); |
There was a problem hiding this comment.
Aren't we now running into a broken situation on Linux if e.g. we don't have enough disk space available?
The native allocation will fail, but the setLength will create a sparse file in this case and thus won't fail because of something like a full disk?
There was a problem hiding this comment.
Good point. Only Windows can guarantee us that it won't create a sparse file (as it requires passing an extra flag to do so), so the setLength variant needs to be restricted to Windows.
On the flip side, the current behavior is trappy as well on Linux, as it falls back to writing 0s through the whole file (possibly taking hours doing so).
In that case I prefer a sparse file (and the machine running out of disk later) instead of the machine spending hours to fill the file with 0s.
There was a problem hiding this comment.
Also worth noting that disk full is covered already by the check in SharedBytes.findCacheSnapshotCacheFilePath which checks available disk space before even preallocating.
hgmauri
approved these changes
Oct 19, 2021
original-brownbear
approved these changes
Oct 19, 2021
|
@elasticmachine retest this please |
ywelsch
added a commit
to ywelsch/elasticsearch
that referenced
this pull request
Oct 19, 2021
Reworks preallocation of the shared_cache file, which was very slow on Windows. Instead of manually filling the file with 0's, this new approach uses the RandomAccessFile.setLength() method, which can quickly allocate a file of the given size (tested that this took only seconds to preallocate TB-size file on Windows).
This was referenced Oct 19, 2021
ywelsch
added a commit
to ywelsch/elasticsearch
that referenced
this pull request
Oct 19, 2021
Reworks preallocation of the shared_cache file, which was very slow on Windows. Instead of manually filling the file with 0's, this new approach uses the RandomAccessFile.setLength() method, which can quickly allocate a file of the given size (tested that this took only seconds to preallocate TB-size file on Windows).
elasticsearchmachine
pushed a commit
that referenced
this pull request
Oct 19, 2021
Reworks preallocation of the shared_cache file, which was very slow on Windows. Instead of manually filling the file with 0's, this new approach uses the RandomAccessFile.setLength() method, which can quickly allocate a file of the given size (tested that this took only seconds to preallocate TB-size file on Windows).
elasticsearchmachine
pushed a commit
that referenced
this pull request
Oct 19, 2021
Reworks preallocation of the shared_cache file, which was very slow on Windows. Instead of manually filling the file with 0's, this new approach uses the RandomAccessFile.setLength() method, which can quickly allocate a file of the given size (tested that this took only seconds to preallocate TB-size file on Windows).
weizijun
added a commit
to weizijun/elasticsearch
that referenced
this pull request
Oct 20, 2021
* upstream/master: (24 commits) Implement framework for migrating system indices (elastic#78951) Improve transient settings deprecation message (elastic#79504) Remove getValue and getValues from Field (elastic#79516) Store Template's mappings as bytes for disk serialization (elastic#78746) [ML] Add queue_capacity setting to start deployment API (elastic#79433) [ML] muting rest compat test issue elastic#79518 (elastic#79519) Avoid redundant available indices check (elastic#76540) Re-enable BWC tests TEST Ensure password 14 chars length on Kerberos FIPS tests (elastic#79496) [DOCS] Temporarily remove APM links (elastic#79411) Fix CCSDuelIT for skipped shards (elastic#79490) Add other time accounting in HotThreads (elastic#79392) Add deprecation info API entries for deprecated monitoring settings (elastic#78799) Add note in breaking changes for nameid_format (elastic#77785) Use 'migration' instead of 'upgrade' in GET system feature migration status responses (elastic#79302) Upgrade lucene version 8b68bf60c98 (elastic#79461) Use Strings#EMPTY_ARRAY (elastic#79452) Quicker shared cache file preallocation (elastic#79447) [ML] Removing some code that's obsolete for 8.0 (elastic#79444) Ensure indexing_data CCR requests are compressed (elastic#79413) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reworks preallocation of the shared_cache file, which was very slow on Windows.
Instead of manually filling the file with 0's, this new approach uses the
RandomAccessFile.setLength()method, which can quickly allocate a file of the given size (tested that this took only seconds to preallocate TB-size file on Windows).