util: support specifying catalog of metamorphic constant values

[jbowens]

The use of metamorphic constants helps us expand test coverage through testing varied code paths. But when we reproduce a failure, changing metamorphic constants can make it more challenging to reproduce a failure. It would be a large quality-of-life improvement if test failures wrote out a catalog of metamorphic constants and their values, and it was easy to re-run a test loading constants from that catalog.

Jira issue: CRDB-33313

[blathers-crl]

cc @cockroachdb/test-eng

[blathers-crl]

Hi @jbowens, please add a C-ategory label to your issue. Check out the label system docs.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[DarrylWong]

I agree it would certainly be nice for the set of metamorphic constants used to be more obvious. Maybe in it's own log file that's very clearly named? And then a print statement in test.log pointing the user towards that file(s)?

In the meanwhile, you can still find these constants in cockroach.stderr.log file.

it was easy to re-run a test loading constants from that catalog

#113301 will add a helpful hint on how to rerun tests with the same seed.

[DarrylWong]

Clarified with Jackson that this is directed towards unit tests, but he mentioned that it would be nice for roachtests as well.

As for UX, he mentioned it'd be nice be able to re run a failing unit test with the same metamorphic constants minus the suspected constant. A suggested idea for how to do this would be a separate file from a test failure that can be easily modified to configure a future test run.

Similar to what's blocking #113164, this seems tricky because metamorphic constants are selected at init time. We'd likely have to rework the metamorphic constants framework first to allow for this.

[pav-kv]

Can the whole list of constants be reproduced from a single rand seed? Print the seed, and add a parameter to set the seed?

[jbowens]

Can the whole list of constants be reproduced from a single rand seed? Print the seed, and add a parameter to set the seed?

Personally, I'd like the ability to re-run with all the same constants with explicitly set exceptions, which is tricky (impossible?) to do through a single seed.

[srosenberg]

...all the same constants with explicitly set exceptions,

Could you please clarify what you mean by "explicitly set exceptions"? An example is best.

[jbowens]

When a particular metamorphic constant value causes a test failure, it's helpful to be able to run all the other metamorphic constants at the same settings but override the suspected constant. This issue was originally motivated by the issues linked from #114057. These tests failed readily with a particular metamorphic seed. Being able to re-run the tests with the same constants but overriding the metamorphism of storage.experimental.eventually_file_only_snapshots.enabled and kv.snapshot_receiver.excise.enabled would've helped the investigation.

[pav-kv]

If the list of overrides is small, one can run with a seed + overrides listed in flags?

If you need to do the opposite (set a couple of variables as constants and randomize the rest), don't even need the seed: just run random generation + overrides.

[pav-kv]

If you need to automate randomization of a couple of variables, and set the rest constant, then you need 3 flags:

• The "global" seed. It defines all the values (but some of them will be overridden below).
• The (short) list of variables that you are experimenting with (optionally with values that override what the seed above generated).
• The (optional) seed to generate values just for this small list (and override what the global seed generated).

With these 3 flags, you can emulate all the use-cases:

• Provide a full list of k=v values.
• Generate a full list using the seed.
• Generate most of the list using the seed, and override some values manually.
• Generate most of the list using the seed, and generate the few values of interest using the other seed.