Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets #2900

Merged

Conversation

thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Sep 25, 2019

built on top of #2836 ([18.09 backport] Switch to go 1.11) rebased

relates to

Backports of

Cherry-pick were clean, no conflicts

# https://github.com/docker/swarmkit/pull/2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
git cherry-pick -s -S -x 5f167cab731ee75bc6cf3888fd2a4a5b5194f924

# second commit from https://github.com/docker/swarmkit/pull/2771 Allow using Configs as CredentialSpecs
git cherry-pick -s -S -x be26111c4a48c44fac04c17c69fd2504aea6db91

# https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
git cherry-pick -s -S -x 06a356671bc11e4fd5d754f257f9c5f93ec5c563

# https://github.com/docker/swarmkit/pull/2866 Swap gometalinter for golangci-lint
git cherry-pick -s -S -x 27c2d27e23e76243bb49ec5d803a6e40b3f96f7a

# https://github.com/docker/swarmkit/pull/2869 Increase max recv gRPC message size to initialize connection broker
git cherry-pick -s -S -x 127e816ed8c8de4c981e02c9d0b45c70d3baa824

# https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence
git cherry-pick -s -S -x d68ac46e3b11d7384472677d210bb0ce941284dc

@thaJeztah
Copy link
Member Author

Argh; I suspect this is running Go 1.13, and related to golang/go#34073 / golang/go#31859

#!/bin/bash -eo pipefail
make check binaries checkprotos
🐳 check
ERRO Running error: context loading failed: failed to load program with go/packages: unsupported version of go: exit status 2: flag provided but not defined: -compiled
usage: list [-e] [-f format] [-json] [build flags] [packages]
Run 'go help list' for details.
 
make: *** [check] Error 3
Exited with code 2

@thaJeztah
Copy link
Member Author

Oh... the reverse; it's still on Go 1.10 😞

@thaJeztah thaJeztah force-pushed the 18.09_backport_fix_update_out_of_sequence branch from a60a21a to cde561a Compare September 25, 2019 09:54
@thaJeztah thaJeztah changed the title [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets [WIP][18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Sep 25, 2019
@thaJeztah
Copy link
Member Author

thaJeztah commented Sep 25, 2019

rebased on top of #2836; marked as WIP and will rebase once that's merged

@codecov
Copy link

codecov bot commented Sep 25, 2019

Codecov Report

Merging #2900 into bump_v18.09 will decrease coverage by 0.03%.
The diff coverage is 31.25%.

@@               Coverage Diff               @@
##           bump_v18.09    #2900      +/-   ##
===============================================
- Coverage        61.79%   61.75%   -0.04%     
===============================================
  Files              134      134              
  Lines            21869    21875       +6     
===============================================
- Hits             13513    13508       -5     
- Misses            6894     6912      +18     
+ Partials          1462     1455       -7

@thaJeztah
Copy link
Member Author

Failure is a flaky test; #2559

--- FAIL: TestUpdaterRollback (35.31s)
    --- FAIL: TestUpdaterRollback/pause/monitor_set/spec_version_unset (3.61s)
        update_test.go:272: service was updated
	Error Trace:	testutils.go:42
			update_test.go:343
			update_test.go:20
	Error:		no task creation
            

    --- FAIL: TestUpdaterRollback/continue/spec_version_set (30.00s)
        update_test.go:272: service was updated
        update_test.go:324: test timed out before watchServiceUpdate provided an event
FAIL
coverage: 84.9% of statements
FAIL	github.com/docker/swarmkit/manager/orchestrator/replicated	37.618s
make: *** [coverage] Error 1
Exited with code 2

I'll try restarting CI

@thaJeztah thaJeztah force-pushed the 18.09_backport_fix_update_out_of_sequence branch from 1539b14 to 7c20388 Compare September 26, 2019 16:12
olljanat and others added 6 commits October 1, 2019 18:35
Signed-off-by: Olli Janatuinen <[email protected]>
(cherry picked from commit 5f167ca)
Signed-off-by: Sebastiaan van Stijn <[email protected]>
Signed-off-by: Drew Erny <[email protected]>
(cherry picked from commit be26111)
Signed-off-by: Sebastiaan van Stijn <[email protected]>
It is likely that a large portion of test flakiness, especially in CI,
comes from the fact that swarmkit components under test are started in
goroutines, but those goroutines never have an opportunity to run. This
adds code ensuring those goroutines are scheduled and run, which should
hopefully solve many inexplicably flaky tests.

Additionally, increased test timeouts, to hopefully cover a few more
flaky cases.

Finally, removed direct use of the atomic package, in favor of less
efficient but higher-level mutexes.

Signed-off-by: Drew Erny <[email protected]>
(cherry picked from commit 06a3566)
Signed-off-by: Sebastiaan van Stijn <[email protected]>
gometalinter is deprecated, and golangci-lint is its recommended
successor. This commit adds golangci-lint as the linter for swarmkit. In
addition, golangci-lint found a few issues in the code that were not
yet identified, and so those issues have been fixed.

Signed-off-by: Drew Erny <[email protected]>
(cherry picked from commit 27c2d27)
Signed-off-by: Sebastiaan van Stijn <[email protected]>
Signed-off-by: nmengin <[email protected]>
(cherry picked from commit 127e816)
Signed-off-by: Sebastiaan van Stijn <[email protected]>
A simple but old error has recently become evident. Due to the fact that
we read an object and then write it back across the boundaries of a
transaction, it is possible for the task object to have changed in
between transactions. This would cause the attempt to write out the old
task to suffer an "Update out of sequence" error.

This fix simply reads the latest version of the task back out within the
boundary of a transaction to avoid the race.

Signed-off-by: Drew Erny <[email protected]>
(cherry picked from commit d68ac46)
Signed-off-by: Sebastiaan van Stijn <[email protected]>
@thaJeztah thaJeztah force-pushed the 18.09_backport_fix_update_out_of_sequence branch from 7c20388 to bfce89e Compare October 1, 2019 16:35
@thaJeztah thaJeztah changed the title [WIP][18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Oct 1, 2019
@thaJeztah
Copy link
Member Author

rebased 👍

@thaJeztah
Copy link
Member Author

@dperny this one's green, and probably ready to go

@dperny dperny merged commit 5c86095 into moby:bump_v18.09 Oct 2, 2019
@thaJeztah thaJeztah deleted the 18.09_backport_fix_update_out_of_sequence branch October 2, 2019 21:07
thaJeztah added a commit to thaJeztah/docker that referenced this pull request Oct 9, 2019
…v18.09)

full diff: moby/swarmkit@142a737...5c86095

- moby/swarmkit#2892 [18.09 backport] Remove hardcoded IPAM config subnet value for ingress network
    - backport of moby/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network
    - fixes [ENGORC-2651](https://docker.atlassian.net/browse/ENGORC-2651)
- moby/swarmkit#2836 [18.09 backport] Switch to go 1.11
    - backport of moby/swarmkit#2752 Switch to go 1.11
- moby/swarmkit#2901 [18.09 backport] Bump to golang 1.12.9
    - backport of moby/swarmkit#2880 Bump to golang 1.12.9
- moby/swarmkit#2900 [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets
    - backport of moby/swarmkit#2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
    - backport of moby/swarmkit#2771 Allow using Configs as CredentialSpecs
        - **second commit only** (attempt to fix weirdly broken tests)
    - backport of moby/swarmkit#2808 Fix flaky tests
    - backport of moby/swarmkit#2866 Swap gometalinter for golangci-lint
    - backport of moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker
        - related / similar to moby#38103 / docker-archive#102 cluster: set bigger grpc limit for array requests
        - related / similar to moby#39306 Increase max recv gRPC message size for nodes and secrets
        - fixes moby/swarmkit#2733 Error generated when messages size is too big
    - backport of moby/swarmkit#2870 Fix update out of sequence

Signed-off-by: Sebastiaan van Stijn <[email protected]>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this pull request Oct 23, 2019
…v18.09)

full diff: moby/swarmkit@142a737...5c86095

- moby/swarmkit#2892 [18.09 backport] Remove hardcoded IPAM config subnet value for ingress network
    - backport of moby/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network
    - fixes [ENGORC-2651](https://docker.atlassian.net/browse/ENGORC-2651)
- moby/swarmkit#2836 [18.09 backport] Switch to go 1.11
    - backport of moby/swarmkit#2752 Switch to go 1.11
- moby/swarmkit#2901 [18.09 backport] Bump to golang 1.12.9
    - backport of moby/swarmkit#2880 Bump to golang 1.12.9
- moby/swarmkit#2900 [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets
    - backport of moby/swarmkit#2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
    - backport of moby/swarmkit#2771 Allow using Configs as CredentialSpecs
        - **second commit only** (attempt to fix weirdly broken tests)
    - backport of moby/swarmkit#2808 Fix flaky tests
    - backport of moby/swarmkit#2866 Swap gometalinter for golangci-lint
    - backport of moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker
        - related / similar to moby/moby#38103 / docker-archive/engine#102 cluster: set bigger grpc limit for array requests
        - related / similar to moby/moby#39306 Increase max recv gRPC message size for nodes and secrets
        - fixes moby/swarmkit#2733 Error generated when messages size is too big
    - backport of moby/swarmkit#2870 Fix update out of sequence

Signed-off-by: Sebastiaan van Stijn <[email protected]>
Upstream-commit: e06f07ef337ab890f211397d6b408b75a2512dc5
Component: engine
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants