scikit: random segmentation faults in hydra (

[nbren12]

Describe the bug

Hydra builds for scikitlearn give a segmentation fault intermittently in darwin and linux.

The issue happens 100% of the time for Darwin, but only rarely for Linux, and occurs about 72% of the way through the test suite. Since it's a segfault, the offending test is not shown. I suspect we'll need to enable verbose logging to figure out which test to disable.

To Reproduce

Unfortunately (?), the builds work locally in a sandbox so this seems specific to the hydra environment, which I have no idea how to debug.

Expected behavior

scikitlearn builds in hydra and is in the cache.

Notify maintainers

@rmcgibbo

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[nbren12]

From counting "." in the pytest logs, I think the segfault is in the 4313rd test in test_common.py. The name of this test is

sklearn/tests/test_common.py::test_estimators[NuSVC()-check_estimators_fit_returns_self(readonly_memmap=True)]

readonly_memmap seems fishy 🐟 . Exactly the kind of thing that would raise a segfault.

[rmcgibbo]

Good find with the '.' counting! Were you able to determine if it happens only on certain hydra machines? They might be old machines missing certain x86_64 ISA features?

[nbren12]

I'm not too familiar with hydra. I really just figured out what URL to check for mac packages.

[rmcgibbo]

This would be a great ZHF: #122042 for someone.

[tricktron]

Seems to be the same problem scikit-learn/scikit-learn#17582.

@TomDLT wrote: May be related to misaligned arrays in joblib joblib/joblib#563

[rmcgibbo]

It seems like an imminent fix for that issue in joblib is not in the cards, so I'd say that we should disable that test and include a comment that points to the relevant GitHub issues.

[tricktron]

@rmcgibbo I agree.

[nbren12]

@rmcgibbo I just pushed a PR (#122687) fixing this.