Make test ordering and parametrization reproducible

[pavoljuhas]

Implement deterministic ordering of pytest parameters.
Add CIRQ_TESTING_RANDOM_SEED environment variable for seeding
random numbers generation in random and numpy.random modules.
Use commit epoch time as the default seed to have a non-constant,
but deterministic seeding.

NB: Use empty CIRQ_TESTING_RANDOM_SEED to prevent seeding, for example,
CIRQ_TESTING_RANDOM_SEED="" check/pytest

This implements #4787

[MichaelBroughton]

LGTM

[vtomole]

We can also use pytest-randomly to get random a seed but it will also come with other stuff; like running the tests in random order, so I'm fine with this.

[vtomole]

Oh, we can pass in the --randomly-dont-reorganize flag. But on the other hand, pytest-randomly will be a new dependency we add to Cirq's dev tools on top of the already enormous list of dev tool packages we have..

[vtomole]

Let's add export CIRQ_TESTING_RANDOM_SEED to the incremental coverage checks as well.

[pavoljuhas]

Let's add export CIRQ_TESTING_RANDOM_SEED to the incremental coverage checks as well.

The incremental-coverage scripts run tests with check/pytest which already exports CIRQ_TESTING_RANDOM_SEED:

• Cirq/check/pytest-and-incremental-coverage

  Line 57 in 894d49e

  check/pytest --actually-quiet \

• Cirq/check/pytest-changed-files-and-incremental-coverage

  Line 93 in 894d49e

  check/pytest "${changed_python_tests[@]}" \

Is there perhaps some other script that I am missing?

[vtomole]

Ah, okay.

If you have the time to try pytest-randomly, please do. In the conftest.py, we can have

@pytest.fixture()
def global_seed(pytestconfig) -> int:
    return pytestconfig.getoption("randomly_seed")

This will shave a lot of code in this PR with the caveat of us adding pytest-randomly as a Dev tools dependency.

[pavoljuhas]

If you have the time to try pytest-randomly, please do.

They have a wontfix issue pytest-dev/pytest-randomly#75 which implies that having random parameters in @pytest.mark.parametrize (random at import time) breaks distributed run with pytest-xdist.
That is just the use case I'd like to support here as Cirq has a number of such parametrizations, for example,

Cirq/cirq-core/cirq/testing/random_circuit_test.py

Line 95 in 894d49e

@pytest.mark.parametrize('seed', [random.randint(0, 2 ** 32) for _ in range(10)])

Resetting random seed for every test case as done in pytest-randomly sounds fishy, because every test then gets the same handful of random numbers from the start of a seeded RNG sequence.

On the other hand, the randomized test order with pytest-randomly sounds useful provided it indeed works with pytest-xdist.
@vtomole - perhaps that should be opened as a separate issue?

[pavoljuhas]

@vtomole - please let me know if there are still some changes required here.

[vtomole]

@pavoljuhas Thanks for the ping. I missed the first notification. I'll finish reviewing this today.

[vtomole]

LGTM.

[vtomole]

which implies that having random parameters in @pytest.mark.parametrize (random at import time) breaks distributed run with pytest-xdist.

Okay. Thanks for looking into it.