Build and deploy in endless loop when switching branches with different FTL versions

[matt2e]

@jiyoontbd hit an endless build loop

logs (with module names changed):

...
info:pm: Module built (14.01s)
info:pm: Deploying module
info:controller0: Stopped dpl-pm-3ap2jpizqgnebkbz
info:controller0: Deployed dpl-pm-4dk6xt07dxqlv60u
info:abf: Building module
warn:abf: /Users/jiyoon/IdeaProjects/pfi/backend/modules/abf/cryptosample.go:15:22-22: external type "github.com/tbdeng/pfi/backend/libs/cryptography/datakey.DataKeyStoreConfig" will be widened to Any
info:abf: Module built (8.07s)
info:abf: Deploying module
info:cmpl: Building module
info:cmpl: Module built (9.07s)
info:cmpl: Deploying module
info:cstm: Building module
info:cstm: Module built (8.12s)
info:cstm: Deploying module
info:tx: Building module
info:tx: Module built (7.88s)
info:tx: Deploying module
info:k: Building module
info:k: Module built (7.78s)
info:k: Deploying module
info:ofr: Building module
info:ofr: Module built (6.38s)
info:ofr: Deploying module
info:pm: Building module <==== back to the start
...

It looks like pm module did have differences because it had to be deployed whereas other modules didn't detect any changes in the build artifacts

[matt2e]

Repro'd by doing the following:

• in partner project, go to commit a20cc725
• run ftl dev --recreate --log-level=DEBUG
• while ftl is running check out commit baa8449c (this has a bumped version of ftl installed)
• ftl then starts building and deploying each module in an endless loop

So maybe this is to do with switching branches where FTL is on different versions, but that is also something people do, so maybe we should handle this in a more graceful way.

[matt2e]

debug: changed "k": *.../modules/k/go.mod

[stuartwdouglas]

So this is a 'go mod tidy' loop. PFI has a go.mod in backend/go.mod. Because the other modules depend on this if they have been set to the older version by FTL running go mod tidy will automatically upgrade them.

In the example above the exact sequence of events is:

1. Checkout FTL on 0.306
2. Start FTL
3. Move to 0.309 via checkout
4. The running 0.306 will override most of the go.mod files, but not backend/go.mod as it is not a FTL module, so it remains at 0.309.
5. go mod tidy is run as part of the deployment process, and the version is synched back to 0.309
6. FTL detects that this has changed and goes back to step 1