Using .loc with MultiIndex containing np.nan unexpected behavior

[deponovo]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        "temp_playlist": [0, 0, 0, 0],
        "objId": ["o1", np.nan, "o1", np.nan],
        "x": [1, 2, 3, 4],
    }
)

agg_df = df.groupby(by=['temp_playlist', 'objId'], dropna=False)["x"].agg(list)
print(agg_df.loc[agg_df.index[-1]])  # KeyError: because it is (0, np.nan), wanted to get [2, 4]

Issue Description

This issue is a follow-up of the discussion in this SO question.
It appears to be a bug, but if not, meaning, if this is desired behavior it should be documented.
As shown in the Reproducible Example, after grouping x data on the temp_playlist and objId columns, there is a MultiIndex (0, nan). This index is meaningful and I wanted to access the data via it as I can perform with any other index from agg_df.index as agg_df.loc[<index_pos>]. This is not possible for the index containing the nan (agg_info_df.loc[agg_info_df.index[-1]]). However, it does work if that same index is provided in a list of indices. So this seems at least inconsistent if not a bug entirely.
For more info, please consult the SO question, especially this answer.

Expected Behavior

agg_info_df.loc[(0, np.nan)] should return [2, 4]

Installed Versions

python 3.8.5, pandas 1.3.1, numpy 1.20.3

[phofl]

Please provide a minimal reproducible example, see https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[deponovo]

Edited the original report.

[phofl]

Are that many rows necessary? Is the groupby necessary to reproduce? An example is minimal if you can't remove anything without causing the bug to disappear

[deponovo]

I got to the finding via the use of groupby. This answer states also that the finding can only be reproduced on a df obtained by the groupby and not on one explicitly created with the MultiIndex (see 'Original Attempt to Reproduce Error' part).

Are that many rows necessary?

Maybe not, that's at least the minimal example I prepared to report this.

[phofl]

Please trim your example down then.

I can only speak for myself but my motivation decreases significantly if the example is unnecessary complicated and contains a lot of other function calls.

[deponovo]

I updated the original report again.

[phofl]

It would be great if you could remove the groupby if its not necessary. Otherwise this looks good now

[deponovo]

The groupby, as referred previously, is required.

[phofl]

Sorry misread your previous answer

[AlexKirko]

I have checked this on the latest version, same KeyError. @deponovo , would you be interested in investigating the cause or contributing a fix?

[phofl]

This is a regression caused by #35852

[CloseChoice]

take

[simonjayhawkins]

This is a regression caused by #35852

#35852 was merged for 1.1.2. AFAICT the same KeyError is raised in 1.1.1. In 1.0.5 and earlier, the code sample gives TypeError: groupby() got an unexpected keyword argument 'dropna'

[phofl]

Can't reproduce this either now. I might have focused only on the factorize function.

[simonjayhawkins]

removing milestone