ci: pin image versions used in CI so it remains deterministic

[davidmurdoch]

We shouldn't use image versions in CI that can update all willy-nilly. This will result in non-deterministic test runs.

ubuntu-2404:2024.05.1 is currently the same as ubuntu-2404:2024.05.1 (see https://discuss.circleci.com/t/new-ubuntu-24-04-linux-machine-executor-image/51018#linux-1)

[github-actions]

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

[metamaskbot]

Builds ready [285f0d8]

• builds: chrome, firefox
• builds (beta): chrome
• builds (flask): chrome, firefox
• builds (MMI): chrome, firefox
• builds (test): chrome, firefox
• builds (test-flask): chrome, firefox
• build viz: Build System
• mv3: Background Module Init Stats
• mv3: UI Init Stats
• mv3: Module Load Stats
• mv3: Bundle Size Stats
• mv2: E2e Actions Stats
• code coverage: Report
• storybook: Storybook
• typescript migration: Dashboard
• all artifacts
• bundle viz:

  • background: 0, 1, 2, 3, 4, 5, 6, 7
  • common: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9
  • content-script: 0
  • offscreen: 0
  • ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (1868 ± 135 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	485	2414	1775	393	189
		domContentLoaded	1436	2396	1828	274	131
		load	1447	2423	1868	281	135
		domInteractive	16	130	43	26	13
		backgroundConnect	8	102	44	30	14
		firstReactRender	16	152	49	38	18
		getState	6	72	23	23	11
		initialActions	0	1	0	0	0
		loadScripts	1031	1735	1319	204	98
		setupStore	6	86	26	29	14
		uiStartup	1682	2999	2185	357	171

Bundle size diffs

• background: 0 Bytes (0.00%)
• ui: 0 Bytes (0.00%)
• common: 0 Bytes (0.00%)

[Gudahtt]

We've gone back and forth on this a few times. We used to pin browser versions as well, but the team decided at one point that auto-updating made it easier to keep up (and to detect issues with newer browser versions). I see that the browser images are still tags that we'd expect to be updated over time with OS updates.

Still, no objection to pinning, I don't think this is too onerous to keep up with if you see the moving target as causing problems.

[davidmurdoch]

We've gone back and forth on this a few times. We used to pin browser versions as well, but the team decided at one point that auto-updating made it easier to keep up (and to detect issues with newer browser versions). I see that the browser images are still tags that we'd expect to be updated over time with OS updates.

Still, no objection to pinning, I don't think this is too onerous to keep up with if you see the moving target as causing problems.

I think it makes sense for our browser versions to auto update because if the extension doesn't work properly on a recent/future browser version it is going to (soon) be failing in production as well [1]. These types of failures are runtime bugs that must be fixed ASAP.

I'm just firmly against having our CI infrastructure changing out from under us; when the CI image is updated the tooling (i.e., node version) is already going to be many versions old anyway; there is just no urgency, so there is no need to create a "CI IS BROKEN!" panic if it auto-updates and things fail on us. Of course current doesn't change very often, but I still appreciate a deterministic CI environment.

[1] We could automate updating our pinned version at every release of a new browser build, Dependabot style. We'd then get the best of both worlds: CI determinism and early detections of browser incompatibilities.

[Gudahtt]

I see that the browser images are still tags that we'd expect to be updated over time with OS updates.

To clarify this part, I was pointing out that this problem remained with the other executors. The Node version is fixed for them (at least to the minor version), but OS updates would still sporadically be made.

Really we need a setup-node type of step. It does seem egregiously wrong that we're using system node for some of our build steps.

[davidmurdoch]

The Node version is fixed for them (at least to the minor version), but OS updates would still sporadically be made.

Ah, I hadn't even thought about that. Digging into https://circleci.com/developer/images/image/cimg/node its not clear if they'll retag a fully-versioned (e.g., cimg/node:20.17.1-browsers) image with OS updates.

Anyway, you brought up some points I hadn't thought about. I'll mull on it over the long weekend.