Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove support for internal versioning for concurrency control #38254

Merged
merged 51 commits into from
Feb 5, 2019

Conversation

bleskes
Copy link
Contributor

@bleskes bleskes commented Feb 2, 2019

Elasticsearch has long supported compare and set (a.k.a optimistic concurrency control) operations using internal document versioning. Sadly that approach is flawed and can sometime do the wrong thing. Here's the relevant excerpt from the resiliency status page:

When a primary has been partitioned away from the cluster there is a short period of time until it detects this. During that time it will continue indexing writes locally, thereby updating document versions. When it tries to replicate the operation, however, it will discover that it is partitioned away. It won’t acknowledge the write and will wait until the partition is resolved to negotiate with the master on how to proceed. The master will decide to either fail any replicas which failed to index the operations on the primary or tell the primary that it has to step down because a new primary has been chosen in the meantime. Since the old primary has already written documents, clients may already have read from the old primary before it shuts itself down. The version numbers of these reads may not be unique if the new primary has already accepted writes for the same document

We recently introduced a new sequence number based approach that doesn't suffer from this dirty reads problem.

This PR removes support for internal versioning as a concurrency control mechanism in favor of the sequence number approach.

Notes:

  1. I'm opening this PR to allow downstream projects to test their code to see it doesn't use versioning.
  2. I will open a separate PR with deprecation warnings for 6.x
  3. I still need to add breaking changes documentation
  4. This PR contain some minor fixes I run into that I will extract to a separate PR before merging.

Relates to #1078

@bleskes bleskes added >breaking :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. v7.0.0 labels Feb 2, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@bleskes
Copy link
Contributor Author

bleskes commented Feb 5, 2019

@elasticmachine run elasticsearch-ci/default-distro

@jasontedor
Copy link
Member

@elasticmachine test this please

@bleskes
Copy link
Contributor Author

bleskes commented Feb 5, 2019

@elasticmachine run elasticsearch-ci/1

@bleskes bleskes merged commit 033ba72 into elastic:master Feb 5, 2019
@bleskes bleskes deleted the cas_remove_internal_versioning branch February 5, 2019 19:53
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Feb 11, 2019
* master:
  Add an authentication cache for API keys (elastic#38469)
  Fix exit code in certutil packaging test (elastic#38393)
  Enable logs for intermittent test failure (elastic#38426)
  Disable BWC to backport recovering retention leases (elastic#38477)
  Enable bwc tests now that elastic#38443 is backported. (elastic#38462)
  Fix Master Failover and DataNode Leave Blocking Snapshot (elastic#38460)
  Recover retention leases during peer recovery (elastic#38435)
  Set update mappings mater node timeout to 30 min (elastic#38439)
  Assert job is not null in FullClusterRestartIT (elastic#38218)
  Update ilm-api.asciidoc, point to REMOVE policy (elastic#38235) (elastic#38463)
  SQL: Fix esType for DATETIME/DATE and INTERVALS (elastic#38179)
  Handle deprecation header-AbstractUpgradeTestCase (elastic#38396)
  XPack: core/ccr/Security-cli migration to java-time (elastic#38415)
  Disable bwc tests for elastic#38443 (elastic#38456)
  Bubble-up exceptions from scheduler (elastic#38317)
  Re-enable TasksClientDocumentationIT.testCancelTasks (elastic#38234)
  Allow custom authorization with an authorization engine  (elastic#38358)
  CRUDDocumentationIT fix documentation references
  Remove support for internal versioning for concurrency control (elastic#38254)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>breaking :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. v7.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants