BUG: segmentation fault with pd.NA and np.ndarray.__contains__

[jorisvandenbossche]

Testing for membership in an ndarray segfaults:

In [1]: pd.NA in np.array(["a"], dtype=object) 
Segmentation fault (core dumped)

Now, this might be something that needs to be fixed in numpy, as for plain lists it "works":

In [3]: pd.NA in ['a']  
...
TypeError: boolean value of NA is ambiguous

but numpy might not expect an error instead of True/False.

[jorisvandenbossche]

cc @seberg

[seberg]

Thanks, looks like a missing error check in ndarray.__contains__ (that method is broken for other reasons):

    any = PyArray_Any((PyArrayObject *)res, NPY_MAXDIMS, NULL);
    Py_DECREF(res);
    ret = PyObject_IsTrue(any);
    Py_DECREF(any);

must test for any == NULL.

[seberg]

I hope this is fixed in master now, the whole issue is a bit more complex, but I think it should be generally safe now. But we have more issues around casting and ufunc execution when such errors occur, I hope you just do not notice it in practice, because those would not be fixed easily.

[jorisvandenbossche]

Thanks for the quick follow-up!
We run tests with numpy master, so we can add a test that is only run on that build, to check it is actually fixed.

[jreback]

reopening, it appears that this should be skipped on older numpies.

[jreback]

@ylin00 would you like to re-vise the PR to do this?

[jreback]

hmm this looks ok now. closing as fixed in #36283