test(python,rust): Refactor failing test

[stinodego]

Related to #9122

Changes:

• Refactor the failing test and run all OOC tests on one worker.
• Comment out a Rust test that doesn't compile. See issue Fix Rust test for logical plan optimizer for categoricals #9828 for followup.

[ritchie46]

Out of curiousity? How does that solve the failing test? Was there a race condition?

[stinodego]

Out of curiousity? How does that solve the failing test? Was there a race condition?

Not sure if this will fix it. But generally running multiple tests in one test function makes things a bit more unstable. While this might not help, at least the test will be factored a bit better.

What we really need is some way to control where the disk spill happens, and make sure things are written to a properly controlled temporary directory. Can we configure disk spill somehow?

[ritchie46]

I think it can be race conditions now that I think of it. The ooc sort/groupby runs a garbage collection job that deletes previous runs. Can we force those tests to not run in parallel?

[stinodego]

I think it can be race conditions now that I think of it. The ooc sort/groupby runs a garbage collection job that deletes previous runs. Can we force those tests to not run in parallel?

Right, so one runner might delete data on disk that a test on another runner relies on.

Sure, we can have all streaming tests on one runner. Let me figure out how to configure that nicely.

[stinodego]

This should do it. Place your OOC tests in the test_ooc.py module and they should all run on the same worker. Or if your test really must live somewhere else, add the pytest marker @pytest.mark.xdist_group("ooc").

[stinodego]

@ritchie46 The tests now run sequentially on a single worker, but it only made things worse 😕 now other tests are failing as well.

Looks like there might be a deeper issue with the garbage collection, or something else that causes subsequent streaming operations to fail?

[ritchie46]

Alright, let's leave it for now. I can take a look at this later.

[stinodego]

Sure. I'll merge the initial refactor part, we can address the underlying issues later - I'll open a draft PR for that.