Fleet scale down doesn't work after an update

[markmandel]

What happened:

After doing an allocation, then a gameserver update, scaling down the Fleet had no effect.

What you expected to happen:

The fleet should scale down when requested, at all times.

How to reproduce it (as minimally and precisely as possible):

root@7551c789cd4f:/go/src/agones.dev/agones# kubectl apply -f ./examples/simple-udp/fleet.yaml
fleet.stable.agones.dev/simple-udp created
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl scale fleet simple-udp --replicas=50
fleet.stable.agones.dev/simple-udp scaled
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get flt simple-udp
NAME         SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp   Packed       50        50        0           50        43s
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl create -f ./examples/simple-udp/gameserverallocation.yaml
gameserverallocation.allocation.agones.dev/simple-udp-bt2dr-jxsxm created
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl create -f ./examples/simple-udp/gameserverallocation.yaml
gameserverallocation.allocation.agones.dev/simple-udp-bt2dr-h87n6 created
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl create -f ./examples/simple-udp/gameserverallocation.yaml
gameserverallocation.allocation.agones.dev/simple-udp-bt2dr-r226c created
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl create -f ./examples/simple-udp/gameserverallocation.yaml
gameserverallocation.allocation.agones.dev/simple-udp-bt2dr-99rj7 created
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get flt simple-udp
NAME         SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp   Packed       50        50        4           46        58s
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl edit flt simple-udp

# change the container port to another value, to force a fleet update

fleet.stable.agones.dev/simple-udp edited
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get flt
NAME         SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp   Packed       50        50        4           46        17m
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get gsset
NAME               SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp-7pvvw   Packed       46        46        0           46        16m
simple-udp-bt2dr   Packed       0         4         4           0         17m
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl scale flt simple-udp --replicas=30
fleet.stable.agones.dev/simple-udp scaled
root@7551c789cd4f:/go/src/agones.dev/agones#
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get flt
NAME         SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp   Packed       30        50        4           46        17m
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get flt
NAME         SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp   Packed       30        50        4           46        17m
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get gsset
NAME               SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp-7pvvw   Packed       46        46        0           46        16m
simple-udp-bt2dr   Packed       0         4         4           0         18m
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get gsset
NAME               SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp-7pvvw   Packed       46        46        0           46        17m
simple-udp-bt2dr   Packed       0         4         4           0         18m

Anything else we need to know?:

This needs an e2e test to ensure it doesn't continue to happen.

Environment:

• Agones version:
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Install method (yaml/helm):
• Troubleshooting guide log(s):
• Others:

[aLekSer]

@markmandel thanks for the detailed description, I will investigate this issue.

[aLekSer]

Wrote E2E test with scaling from 1 to 3 , allocate server, update Fleet and scale down to 1.
Fixed one error with UpdateStatus() failing when ResourceVersion was outdated. Found only error on Gameserver Creation:

message:  "error getting external address for GameServer simple-fleet-x7kb9-lgknt-j2xmp: error retrieving node  for Pod simple-fleet-x7kb9-lgknt-j2xmp: node "" not found"   
  stack: [
   0:  "agones.dev/agones/pkg/gameservers.(*Controller).applyGameServerAddressAndPort
	/go/src/agones.dev/agones/pkg/gameservers/controller.go:669"    
   1:  "agones.dev/agones/pkg/gameservers.(*Controller).syncGameServerStartingState
	/go/src/agones.dev/agones/pkg/gameservers/controller.go:649"    
   2:  "agones.dev/agones/pkg/gameservers.(*Controller).syncGameServer
	/go/src/agones.dev/agones/pkg/gameservers/controller.go:363"    

Need to find a root cause.

[aLekSer]

I think that this behaviour caused by rollingUpdateActive() function, return of which (active GameServerSet.Spec.Replicas) is used as target number of replicas, not Fleet.Spec.Replicas:
https://github.com/GoogleCloudPlatform/agones/blob/e6d91905ccb400b534aca70d2927836ca4043cee/pkg/fleets/controller.go#L424

So I could propose a draft PR for this.

[markmandel]

Fixed!