Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test] Account for auto-repairing for shard gen file #112778

Merged
merged 6 commits into from
Sep 12, 2024

Conversation

ywangd
Copy link
Member

@ywangd ywangd commented Sep 12, 2024

Since #112337, missing shard gen files are automatically reconstructed based on the existing shard snapshot files. If the list of shard snapshot files is complete, it means the repository is effectively not corrupted. This PR updates the test to account for this situation.

Resolves: #112769

Since elastic#112337, missing shard gen files are automatically reconstructed
based on the existing shard snapshot files. If the list of shard
snapshot files are completed, it means the repository is effectively not
corrupted. This PR updates the test to account for this situation.

Resolves: elastic#112769
@ywangd ywangd added >test Issues or PRs that are addressing/adding tests :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.16.0 labels Sep 12, 2024
@ywangd ywangd requested a review from DaveCTurner September 12, 2024 00:24
@elasticsearchmachine elasticsearchmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v9.0.0 labels Sep 12, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

Comment on lines 347 to 349
// Missing shard generation file is automatically repaired based on the shard snapshot files.
// See also BlobStoreRepository#buildBlobStoreIndexShardSnapshots
assertThat(anomalies, corruptedFileType == RepositoryFileType.SHARD_GENERATION ? empty() : not(empty()));
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we can add a flag to make buildBlobStoreIndexShardSnapshots not repair missing shard gen file so that the verification can fail. I am not sure whether it is necessary since this would return warnings to users for something we already have a workaround, i.e. no action is required on the user side. In addition, the warning message about the repairing is logged in this case and I think that is all we need for this type of issue.

Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bah I didn't re-run these tests enough after implementing that workaround. However I don't think this is quite right, we should still get failed to load shard generation if the shard gen file is corrupt (truncated or modified) rather than just deleted.

@ywangd
Copy link
Member Author

ywangd commented Sep 12, 2024

we should still get failed to load shard generation if the shard gen file is corrupt (truncated or modified) rather than just deleted.

Thanks! You are right. I updated the PR to account for that.

@ywangd ywangd requested a review from DaveCTurner September 12, 2024 04:23
Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ywangd ywangd added auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) auto-backport-and-merge labels Sep 12, 2024
@elasticsearchmachine elasticsearchmachine merged commit e1f7814 into elastic:main Sep 12, 2024
15 checks passed
@ywangd ywangd deleted the es-112769-fix branch September 12, 2024 07:19
@elasticsearchmachine
Copy link
Collaborator

💔 Backport failed

Status Branch Result
8.x Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 112778

ywangd added a commit to ywangd/elasticsearch that referenced this pull request Sep 12, 2024
Since elastic#112337, missing shard gen files are automatically reconstructed
based on the existing shard snapshot files. If the list of shard
snapshot files is complete, it means the repository is effectively not
corrupted. This PR updates the test to account for this situation.

Resolves: elastic#112769
(cherry picked from commit e1f7814)

# Conflicts:
#	muted-tests.yml
@ywangd
Copy link
Member Author

ywangd commented Sep 12, 2024

💚 All backports created successfully

Status Branch Result
8.x

Questions ?

Please refer to the Backport tool documentation

v1v added a commit to v1v/elasticsearch that referenced this pull request Sep 12, 2024
…tion-ironbank-ubi

* upstream/main: (302 commits)
  Deduplicate BucketOrder when deserializing (elastic#112707)
  Introduce test utils for ingest pipelines (elastic#112733)
  [Test] Account for auto-repairing for shard gen file (elastic#112778)
  Do not throw in task enqueued by CancellableRunner (elastic#112780)
  Mute org.elasticsearch.script.StatsSummaryTests testEqualsAndHashCode elastic#112439
  Mute org.elasticsearch.repositories.blobstore.testkit.integrity.RepositoryVerifyIntegrityIT testTransportException elastic#112779
  Use a dedicated test executor in MockTransportService (elastic#112748)
  Estimate segment field usages (elastic#112760)
  (Doc+) Inference Pipeline ignores Mapping Analyzers (elastic#112522)
  Fix verifyVersions task (elastic#112765)
  (Doc+) Terminating Exit Codes (elastic#112530)
  (Doc+) CAT Nodes default columns (elastic#112715)
  [DOCS] Augment installation warnings (elastic#112756)
  Mute org.elasticsearch.repositories.blobstore.testkit.integrity.RepositoryVerifyIntegrityIT testCorruption elastic#112769
  Bump Elasticsearch to a minimum of JDK 21 (elastic#112252)
  ESQL: Compute support for filtering ungrouped aggs (elastic#112717)
  Bump Elasticsearch version to 9.0.0 (elastic#112570)
  add CDR related data streams to kibana_system priviliges (elastic#112655)
  Support widening of numeric types in union-types (elastic#112610)
  Introduce data stream options and failure store configuration classes (elastic#109515)
  ...
elasticsearchmachine pushed a commit that referenced this pull request Sep 12, 2024
Since #112337, missing shard gen files are automatically reconstructed
based on the existing shard snapshot files. If the list of shard
snapshot files is complete, it means the repository is effectively not
corrupted. This PR updates the test to account for this situation.

Resolves: #112769
(cherry picked from commit e1f7814)

# Conflicts:
#	muted-tests.yml
davidkyle pushed a commit that referenced this pull request Sep 12, 2024
Since #112337, missing shard gen files are automatically reconstructed
based on the existing shard snapshot files. If the list of shard
snapshot files is complete, it means the repository is effectively not
corrupted. This PR updates the test to account for this situation.

Resolves: #112769
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport pending :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test Issues or PRs that are addressing/adding tests v8.16.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI] RepositoryVerifyIntegrityIT testCorruption failing
3 participants