Minimum changes to make CI workflows work again #814
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ubuntu 16.04 is no longer supported by GitHub. Updated the runner to use Ubuntu 20.04.
The current failures in the Cirq compatibility CI workflow are limited to the Address Sanitizer (ASAN) tests in `scripts/msan_test.sh`. They started happening only when we updated the version of Linux used by the workflow from Ubuntu 16.04 to 20.04, because GitHub no longer offers the Ubuntu 16 runners. After spending a ridiculous amount of time testing various combinations of TensorFlow, TensorFlow Quantum, and compiler toolchains on a more recent Linux, my conclusion is that the ASAN failures stem from differences in the toolchains used to produce the copy of TensorFlow 2.15.0 we get from PyPI, and the current toolchain used to compile TFQ on GitHub. This conclusion comes from the fact if I build a local copy of TensorFlow, and then build TFQ against that, using Clang for everything, the ASAN failures go away. Given that we can't build TensorFlow as part of this workflow (it takes 2 hours to build using 24-cores on a fast machine), it's not clear what can be done to stop the ASAN failures. I'm temporarily commenting out the leak tests in this workflow so that we can proceed on doing other updates and releasing a new version of TFQ. However, this needs to be revisited at some point.
MichaelBroughton
approved these changes
Dec 2, 2024
mhucka
added
the
area/ci
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The aim of this PR is to stop the current CI workflow failures in
ci.yamlandcirq_compatibility.yaml(both in.github/workflows/).In the case of the failure in
ci.yaml, the change is only a stopgap measure: it disables the address sanitizer tests. The failures happen when the workflow runners are updated from Ubuntu 16.04 to 20.04; this update is necessary because GitHub no longer offers the Ubuntu 16 runners.After spending a ridiculous amount of time testing various combinations of TensorFlow, TensorFlow Quantum, and compiler toolchains on a more recent Linux, my conclusion is that the ASAN failures stem from differences in the toolchains used to produce the copy of TensorFlow 2.15.0 we get from PyPI, and what we get under Ubuntu 20 when compiling TFQ on GitHub. This conclusion comes from the fact if I build a local copy of TensorFlow 2.15.0, and then build TFQ against that, using Clang for everything, the ASAN failures go away.
Given that we can't build TensorFlow as part of this workflow (it takes 2 hours to build using 24 cores on a fast machine), it's not clear what can be done to resolve the ASAN failures correctly. So, I'm temporarily commenting out the leak tests in this workflow so that we can proceed on doing other updates and releasing a new version of TFQ. However, this needs to be revisited at some point.