isna does not work with explicit MultiIndex nan-representation

[coroa]

MultiIndex normally use -1 entries in .codes, which is correctly checked by isna, but .groupby(..., dropna=False) adds NaN to the end of .levels and uses their code instead.

Minimal demonstration:

>>> from numpy import nan
>>> from pandas import Series, MultiIndex
>>> s = (
...     Series(
...         3,
...         MultiIndex.from_tuples(
...             [(1, nan), (1, nan), (1, 2)], names=["a", "b"]
...         ),
...     )
...     .groupby(["a", "b"], dropna=False)
...     .sum()
...     .index
... )
>>> s
MultiIndex([(1, 2.0),
            (1, nan)],
           names=['a', 'b'])
>>> s.levels
FrozenList([[1], [2.0, nan]])
>>> s.codes
FrozenList([[0, 0], [0, 1]])

while all the regular MultiIndex constructors consolidate the NaN values to -1

>>> s2 = MultiIndex(s.levels, s.codes)
>>> s2.codes
FrozenList([[0, 0], [0, -1]])

It looks like this leads to all sorts of subtle bugs in pandas itself: pandas-dev/pandas#29111 , pandas-dev/pandas#36060 , pandas-dev/pandas#30750 ,
pandas-dev/pandas#43814 .

[coroa]

It looks like this happened, because pandas-dev/pandas#30584 was implemented as a hack to factorize that effectively hid nan multi-index entries by defining them as a new category and afterwards no-one cleaned up :/. Would be fixed by pandas-dev/pandas#43943 , which made the grouper remove the added nan's after grouping!