🏁 Windows Support

[burgholzer]

Description

This PR adds continuous testing and deployment for Windows.
To this end, it switches to the reusable MQT workflows for CI and CD.
Along the way, it fixes a couple of errors that only revealed themselves on Windows.

Checklist:

• The pull request only contains commits that are related to it.
• I have added appropriate tests and documentation.
• I have made sure that all CI jobs on GitHub pass.
• The pull request introduces no new warnings and follows the project's style guidelines.

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[codecov]

Codecov Report

Attention: Patch coverage is 65.67164% with 23 lines in your changes missing coverage. Please review.

Project coverage is 83.6%. Comparing base (361c4ae) to head (ceea10c).
Report is 80 commits behind head on main.

Files with missing lines	Patch %	Lines
src/UFDecoder.cpp	45.9%	20 Missing ⚠️
include/Code.hpp	81.2%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #266     +/-   ##
=======================================
- Coverage   83.6%   83.6%   -0.1%     
=======================================
  Files         49      49             
  Lines       4162    4159      -3     
  Branches     372     372             
=======================================
- Hits        3481    3477      -4     
- Misses       681     682      +1     

Flag	Coverage Δ
cpp	83.6% <56.6%> (-0.3%)	⬇️
python	83.5% <100.0%> (+0.1%)	⬆️

Files with missing lines	Coverage Δ
...mation_decoding/simulators/memory_experiment_v2.py	89.6% <ø> (+5.1%)	⬆️
...mation_decoding/simulators/quasi_single_shot_v2.py	69.9% <100.0%> (ø)
...alog_information_decoding/simulators/simulation.py	82.7% <100.0%> (ø)
src/mqt/qecc/cc_decoder/decoder.py	98.2% <100.0%> (ø)
include/Code.hpp	79.5% <81.2%> (-0.4%)	⬇️
src/UFDecoder.cpp	61.2% <45.9%> (-2.6%)	⬇️

[burgholzer]

Alright. This is looking much better now already.

@pehamTom I just removed the timeouts from the state prep tests and that seems to have done it. Although some of these tests (sometimes) seem to take forever. Do you see any kind of way to reduce the tests or to speed them up?
Some of the (simulation) tests also sporadically fail because a certain threshold is not reached. Could you maybe fix the random seed for these kind of tests so that the results are reproducible and less flaky?

@lucasberent could you please look at the change here: b58b604
This was necessary to fix one of the tests on Windows. However, the test now fails on Ubuntu and macOS. Do you see any kind of reason for that.
I was kind of suspicious that the following to lines are not the same despite the same comment:

• mqt-qecc/test/python/analog_info/test_memory_experiment.py

  Line 111 in 63defae

  assert np.array_equal(res[0], np.array([0, 0, 0])) # estimate is all zeros

• mqt-qecc/test/python/analog_info/test_memory_experiment.py

  Line 137 in 63defae

  assert np.array_equal(res[0], np.array([1, 0, 0])) # estimate is all zeros

[pehamTom]

@pehamTom I just removed the timeouts from the state prep tests and that seems to have done it. Although some of these tests (sometimes) seem to take forever. Do you see any kind of way to reduce the tests or to speed them up? Some of the (simulation) tests also sporadically fail because a certain threshold is not reached. Could you maybe fix the random seed for these kind of tests so that the results are reproducible and less flaky?

Well, the timeout is there, so the SAT solver doesn't spend a long time on unsatisfiable instances. I can try to trim down the tests further. But again, there are not many small code instances one can meaningfully test.

[burgholzer]

@pehamTom I just removed the timeouts from the state prep tests and that seems to have done it. Although some of these tests (sometimes) seem to take forever. Do you see any kind of way to reduce the tests or to speed them up? Some of the (simulation) tests also sporadically fail because a certain threshold is not reached. Could you maybe fix the random seed for these kind of tests so that the results are reproducible and less flaky?

Well, the timeout is there, so the SAT solver doesn't spend a long time on unsatisfiable instances. I can try to trim down the tests further. But again, there are not many small code instances one can meaningfully test.

At least on Windows that has led to basically none of the synthesis tasks returning a circuit (see the CI logs from a couple commits ago).
It's odd, but not completely unexpected that performance on windows is so different to the Unix systems.
Maybe relaxing the timeouts a little bit instead of removing them would be even better.
Just seeking a pragmatic solution here.
The current CI run shows that it works in principle given a long enough timeout.

[lucasberent]

@lucasberent could you please look at the change here: b58b604 This was necessary to fix one of the tests on Windows. However, the test now fails on Ubuntu and macOS. Do you see any kind of reason for that. I was kind of suspicious that the following to lines are not the same despite the same comment:

• mqt-qecc/test/python/analog_info/test_memory_experiment.py

  Line 111 in 63defae

  assert np.array_equal(res[0], np.array([0, 0, 0])) # estimate is all zeros

• mqt-qecc/test/python/analog_info/test_memory_experiment.py

  Line 137 in 63defae

  assert np.array_equal(res[0], np.array([1, 0, 0])) # estimate is all zeros

the test should be correct as is, the comment is wrong the estimate should be [1,0,0]..

[burgholzer]

Alright. This is looking much better now already.
@pehamTom I just removed the timeouts from the state prep tests and that seems to have done it. Although some of these tests (sometimes) seem to take forever. Do you see any kind of way to reduce the tests or to speed them up? Some of the (simulation) tests also sporadically fail because a certain threshold is not reached. Could you maybe fix the random seed for these kind of tests so that the results are reproducible and less flaky?
@lucasberent could you please look at the change here: b58b604 This was necessary to fix one of the tests on Windows. However, the test now fails on Ubuntu and macOS. Do you see any kind of reason for that. I was kind of suspicious that the following to lines are not the same despite the same comment:

• mqt-qecc/test/python/analog_info/test_memory_experiment.py

  Line 111 in 63defae

  assert np.array_equal(res[0], np.array([0, 0, 0])) # estimate is all zeros

• mqt-qecc/test/python/analog_info/test_memory_experiment.py

  Line 137 in 63defae

  assert np.array_equal(res[0], np.array([1, 0, 0])) # estimate is all zeros

the test should be correct as is, the comment is wrong the estimate should be [1,0,0]..

In that case, something is going wrong in the decoder on Windows as it always yields [0, 0, 0]. Is there anything that you could think of that could cause this?
I'll revert the test change for now.

[lucasberent]

In that case, something is going wrong in the decoder on Windows as it always yields [0, 0, 0]. Is there anything that you could think of that could cause this? I'll revert the test change for now.

perhaps something with the np arrays? Maybe using 'astype' helps otherwise unclear to me.

[burgholzer]

Ok. pymatching 2.2.1 already puts a pin on numpy < 2 so we don't have to.
Hopefully, all tests pass now and this can be merged. I'll then proceed to release a new version, which will be the first one to ship Windows wheels 🎉

Also cleaned up the PR history a little bit so it becomes clearer what changed.

[burgholzer]

@pehamTom sorry that I have to bother you again, but I can't seem to get the ft simulation tests to work under Windows 3.12; even after increasing the number of shots by a factor of 10.
The logical error rate seems to be quite off in the two ft simulation tests.
Windows 3.9 seems to be fine and is not complaining.
Any idea what could be causing this?

[pehamTom]

@pehamTom sorry that I have to bother you again, but I can't seem to get the ft simulation tests to work under Windows 3.12; even after increasing the number of shots by a factor of 10. The logical error rate seems to be quite off in the two ft simulation tests. Windows 3.9 seems to be fine and is not complaining. Any idea what could be causing this?

It was another timeout issue with the state preparation. Since some optimal circuits are synthesized for the simulation, they timed out and returned a non-verified circuit.

But now the time to run the tests is 77 minutes on windows 3.12. I don't think that is an acceptable amount of time. I think the only way to circumvent this is to either not test the SAT solution for the state preparation and verification circuit synthesis or test the individual SAT formulas used in the encoding separately.

[burgholzer]

@pehamTom sorry that I have to bother you again, but I can't seem to get the ft simulation tests to work under Windows 3.12; even after increasing the number of shots by a factor of 10. The logical error rate seems to be quite off in the two ft simulation tests. Windows 3.9 seems to be fine and is not complaining. Any idea what could be causing this?

It was another timeout issue with the state preparation. Since some optimal circuits are synthesized for the simulation, they timed out and returned a non-verified circuit.

But now the time to run the tests is 77 minutes on windows 3.12. I don't think that is an acceptable amount of time. I think the only way to circumvent this is to either not test the SAT solution for the state preparation and verification circuit synthesis or test the individual SAT formulas used in the encoding separately.

Yeah. That's somewhat rough..
What about the most pragmatic solution of not running these tests under Windows but running them under all other operating systems?
It's not ideal, but at least we don't completely give up testing the functionality.

[pehamTom]

@pehamTom sorry that I have to bother you again, but I can't seem to get the ft simulation tests to work under Windows 3.12; even after increasing the number of shots by a factor of 10. The logical error rate seems to be quite off in the two ft simulation tests. Windows 3.9 seems to be fine and is not complaining. Any idea what could be causing this?

It was another timeout issue with the state preparation. Since some optimal circuits are synthesized for the simulation, they timed out and returned a non-verified circuit.
But now the time to run the tests is 77 minutes on windows 3.12. I don't think that is an acceptable amount of time. I think the only way to circumvent this is to either not test the SAT solution for the state preparation and verification circuit synthesis or test the individual SAT formulas used in the encoding separately.

Yeah. That's somewhat rough.. What about the most pragmatic solution of not running these tests under Windows but running them under all other operating systems? It's not ideal, but at least we don't completely give up testing the functionality.

That would be fine by me.

[burgholzer]

Everything is passing and I am glad that this is finally in 🎉
Nice step forward!