Fix for problem where agent is stopped and does not restart #2307

kylewuolle · 2018-11-26T18:04:33Z

This fixes a problem where the agent on a controller is stopped when a node leaves a swarm and is never restarted. By adding a check to see if the agent is nil the agent init routine will run and the other checks that guard against multiple agents will still function as intended. Here's some more information about how this problem presents itself docker/for-linux#495.

GordonTheTurtle · 2018-11-26T18:04:35Z

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "fix-agent-init-problem" [email protected]:kylewuolle/libnetwork.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

thaJeztah · 2018-11-27T00:35:57Z

ping @fcrisciani @euank ptal

thaJeztah · 2018-11-27T00:38:46Z

whoops, wrong Euan. Meant @euanh 😊

controller.go

selansen · 2018-11-30T17:02:24Z

Can you pls fix below error
^@^@🐳 check-protobuf
🐳 lint
🐳 gosimple
🐳 vet
🐳 ineffassign
🐳 fmt
controller.go
👹 please format Go code with 'gofmt -s -w'
Makefile:155: recipe for target 'fmt' failed
make: *** [fmt] Error 1
Makefile:114: recipe for target 'check' failed
make: *** [check] Error 2
Exited with code 2

kylewuolle · 2019-01-09T17:40:29Z

ping @fcrisciani @euanh!

fcrisciani · 2019-01-18T17:54:26Z

controller.go

@@ -268,7 +268,7 @@ func (c *controller) SetClusterProvider(provider cluster.Provider) {
 	}
 	c.Unlock()

-	if provider == nil || sameProvider {
+	if provider == nil || (sameProvider && c.getAgent() != nil) {


if I understood well, basically when the node leave the cluster never reset the cluster provider. If so, I will find much better to just reset the cluster provider to nil when the node leave the cluster.
That way sameProvider will be false and the init will continue. What do you think? is it my understanding correct?

That might work as well. I'll test it out and see!

Not sure if you review moby or not but I've made a PR over there that fixes it in a different way.
moby/moby#38626

fcrisciani · 2019-01-25T00:35:10Z

@kylewuolle if you can rebase I have fixed that lint failure.
also can you confirm that is fixing the issue? If so, LGTM
last thing, squash all the changes into 1 single signed commit

kylewuolle · 2019-01-25T16:04:03Z

Will do thanks! It does for sure fix the issue!

Signed-off-by: Kyle Wuolle <[email protected]>

kylewuolle · 2019-01-28T19:54:27Z

@kylewuolle if you can rebase I have fixed that lint failure.
also can you confirm that is fixing the issue? If so, LGTM
last thing, squash all the changes into 1 single signed commit

All done thanks!

fcrisciani

LGTM Thanks!

fcrisciani · 2019-01-28T19:56:15Z

@thaJeztah to which branch do we need this one to go?

full diff: moby/libnetwork@1a06131...ebcade7 relevant changes: - moby/libnetwork#2349 IPVS: Add support for GetConfig/SetConfig - moby/libnetwork#2343 Revert "debian has iptables-legacy and iptables-nft now" - moby/libnetwork#2230 Moving IPVLAN driver out of experimental - moby/libnetwork#2307 Fix for problem where agent is stopped and does not restart - moby/libnetwork#2303 Touch-up error-message and godoc for ConfigVXLANUDPPort - moby/libnetwork#2325 Fix possible nil pointer exception - moby/libnetwork#2302 Use sync.RWMutex for VXLANUDPPort - moby/libnetwork#2306 Improve error if auto-selecting IP-range failed Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: moby/libnetwork@1a06131...ebcade7 relevant changes: - moby/libnetwork#2349 IPVS: Add support for GetConfig/SetConfig - moby/libnetwork#2343 Revert "debian has iptables-legacy and iptables-nft now" - moby/libnetwork#2230 Moving IPVLAN driver out of experimental - moby/libnetwork#2307 Fix for problem where agent is stopped and does not restart - moby/libnetwork#2303 Touch-up error-message and godoc for ConfigVXLANUDPPort - moby/libnetwork#2325 Fix possible nil pointer exception - moby/libnetwork#2302 Use sync.RWMutex for VXLANUDPPort - moby/libnetwork#2306 Improve error if auto-selecting IP-range failed Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: 3ab093d5670e8d59f6ae0c4604b8fcabf1582854 Component: engine

full diff: moby/libnetwork@1a06131...ebcade7 relevant changes: - moby/libnetwork#2349 IPVS: Add support for GetConfig/SetConfig - moby/libnetwork#2343 Revert "debian has iptables-legacy and iptables-nft now" - moby/libnetwork#2230 Moving IPVLAN driver out of experimental - moby/libnetwork#2307 Fix for problem where agent is stopped and does not restart - moby/libnetwork#2303 Touch-up error-message and godoc for ConfigVXLANUDPPort - moby/libnetwork#2325 Fix possible nil pointer exception - moby/libnetwork#2302 Use sync.RWMutex for VXLANUDPPort - moby/libnetwork#2306 Improve error if auto-selecting IP-range failed Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: moby/libnetwork@c902989...872f0a8 - moby/libnetwork#2354 [18.09 backport] Cleanup the cluster provider when the agent is closed - backport of moby/libnetwork#2307 Fix for problem where agent is stopped and does not restart - fixes docker/for-linux#495 Docker swarm overlay networking not working after --force-new-cluster - moby/libnetwork#2369 [18.09 BACKPORT] Pick a random host port if the user does not specify a host port - backport of moby/libnetwork#2368 (windows) Pick a random host port if the user does not specify a host port Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: moby/libnetwork@c902989...872f0a8 - moby/libnetwork#2354 [18.09 backport] Cleanup the cluster provider when the agent is closed - backport of moby/libnetwork#2307 Fix for problem where agent is stopped and does not restart - fixes docker/for-linux#495 Docker swarm overlay networking not working after --force-new-cluster - moby/libnetwork#2369 [18.09 BACKPORT] Pick a random host port if the user does not specify a host port - backport of moby/libnetwork#2368 (windows) Pick a random host port if the user does not specify a host port Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: 5354408039681020f9ad6afe4bf696fc90f9ce69 Component: engine

full diff: moby/libnetwork@1a06131...ebcade7 relevant changes: - moby/libnetwork#2349 IPVS: Add support for GetConfig/SetConfig - moby/libnetwork#2343 Revert "debian has iptables-legacy and iptables-nft now" - moby/libnetwork#2230 Moving IPVLAN driver out of experimental - moby/libnetwork#2307 Fix for problem where agent is stopped and does not restart - moby/libnetwork#2303 Touch-up error-message and godoc for ConfigVXLANUDPPort - moby/libnetwork#2325 Fix possible nil pointer exception - moby/libnetwork#2302 Use sync.RWMutex for VXLANUDPPort - moby/libnetwork#2306 Improve error if auto-selecting IP-range failed Signed-off-by: Sebastiaan van Stijn <[email protected]>

GordonTheTurtle added the dco/no label Nov 26, 2018

kylewuolle force-pushed the fix-agent-init-problem branch from e52c790 to 918fcaf Compare November 26, 2018 18:09

GordonTheTurtle removed the dco/no label Nov 26, 2018

selansen reviewed Nov 27, 2018

View reviewed changes

controller.go Outdated Show resolved Hide resolved

GordonTheTurtle added the dco/no label Nov 30, 2018

kylewuolle force-pushed the fix-agent-init-problem branch from fa3358b to 42da117 Compare November 30, 2018 16:59

GordonTheTurtle removed the dco/no label Nov 30, 2018

selansen approved these changes Nov 30, 2018

View reviewed changes

fcrisciani reviewed Jan 18, 2019

View reviewed changes

GordonTheTurtle added the dco/no label Jan 24, 2019

kylewuolle mentioned this pull request Jan 24, 2019

Fix for DNS name resolution after performing init with --force-new-cluster moby/moby#38626

Closed

Cleanup the cluster provider when the agent is closed

c7125bd

Signed-off-by: Kyle Wuolle <[email protected]>

kylewuolle force-pushed the fix-agent-init-problem branch from 35f6aa3 to c7125bd Compare January 25, 2019 17:03

GordonTheTurtle removed the dco/no label Jan 25, 2019

fcrisciani approved these changes Jan 28, 2019

View reviewed changes

fcrisciani merged commit d8d4c8c into moby:master Jan 28, 2019

This was referenced Apr 1, 2019

bump libnetwork to ebcade70ad1059b070d0040d798ecca359bc5fed moby/moby#38983

Merged

[18.03 backport] Cleanup the cluster provider when the agent is closed #2353

Merged

[18.09 backport] Cleanup the cluster provider when the agent is closed #2354

Merged

thaJeztah mentioned this pull request Apr 23, 2019

[18.09] bump libnetwork 872f0a83c98add6cae255c8859e29532febc0039 (18.09 branch) docker-archive/engine#200

Closed

andrewhsu mentioned this pull request Apr 23, 2019

[18.09] bump libnetwork 872f0a8 docker-archive/engine#201

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for problem where agent is stopped and does not restart #2307

Fix for problem where agent is stopped and does not restart #2307

kylewuolle commented Nov 26, 2018

GordonTheTurtle commented Nov 26, 2018

thaJeztah commented Nov 27, 2018

thaJeztah commented Nov 27, 2018

selansen commented Nov 30, 2018

kylewuolle commented Jan 9, 2019

fcrisciani Jan 18, 2019

kylewuolle Jan 22, 2019

fcrisciani Jan 22, 2019

kylewuolle Jan 23, 2019

fcrisciani commented Jan 25, 2019 •

edited

Loading

kylewuolle commented Jan 25, 2019

kylewuolle commented Jan 28, 2019

fcrisciani left a comment

fcrisciani commented Jan 28, 2019 •

edited

Loading

Fix for problem where agent is stopped and does not restart #2307

Fix for problem where agent is stopped and does not restart #2307

Conversation

kylewuolle commented Nov 26, 2018

GordonTheTurtle commented Nov 26, 2018

thaJeztah commented Nov 27, 2018

thaJeztah commented Nov 27, 2018

selansen commented Nov 30, 2018

kylewuolle commented Jan 9, 2019

fcrisciani Jan 18, 2019

Choose a reason for hiding this comment

kylewuolle Jan 22, 2019

Choose a reason for hiding this comment

fcrisciani Jan 22, 2019

Choose a reason for hiding this comment

kylewuolle Jan 23, 2019

Choose a reason for hiding this comment

fcrisciani commented Jan 25, 2019 • edited Loading

kylewuolle commented Jan 25, 2019

kylewuolle commented Jan 28, 2019

fcrisciani left a comment

Choose a reason for hiding this comment

fcrisciani commented Jan 28, 2019 • edited Loading

fcrisciani commented Jan 25, 2019 •

edited

Loading

fcrisciani commented Jan 28, 2019 •

edited

Loading