REF: Store metadata in an attrs dict

[TomAugspurger]

This aids in the implementation of
#28394. Over there, I'm having
issues with using NDFrame.__finalize__ to copy attributes, in part
because getattribute on NDFrame is so complicated.

This simplifies that PR because we only need to look in NDFrame.attrs (name borrowed from xarray),
which is just a plain dictionary.

Aside from the addition of a public NDFrame.attrs dictionary, there
aren't any user-facing API changes.

[jbrockmendel]

are attrs linked to _metadata in any way?

[TomAugspurger]

Not yet, though perhaps eventually if we can find a suitable way to deprecate _metadata for subclasses. With this PR, pandas doesn't use _metadata itself anymore (previously, we just used it for Series.name.

[jreback]

wait, then we should deprecate _metadata right?

[TomAugspurger]

Most likely? I have no idea who is using it or what they're using it for though.

[jreback]

ok can you open an issue to deprecate that

[jreback]

wait, then we should deprecate _metadata right?

[jreback]

lgtm.

[jreback]

ok can you open an issue to deprecate that

[jreback]

self.attrs.update(other) ?

[TomAugspurger]

That'd work for now, but we'll need a for loop in my other PR since we'll have per-attr logic to resolve these.

[jreback]

do you want to add docs anywhere / whatsnew? or just keep private mention for now?

[TomAugspurger]

do you want to add docs anywhere / whatsnew? or just keep private mention for now?

I pushed some basics API docs. Will want a larger section once things settle I think.

[jreback]

thanks @TomAugspurger very nice!

[shoyer]

In xarray, we currently use duck-typing to check for .attrs attributes in the xarray.DataArray constructor. So this PR broke our tests (see pydata/xarray#3440), but that's probably our own fault.

My bigger concern is that I noticed that attrs is used in different way in pandas than xarray: name is a member of attrs vs. in xarray where DataArray.name is a separate attribute. This is not an unreasonable thing to do, but it does introduce some minor incompatible: in xarray (and h5py) attrs really is an arbitrary mapping, but that isn't quite true in pandas (because name needs to be hashable). Also, what happens if you try to delete name from the attrs dict?

[jorisvandenbossche]

Also, what happens if you try to delete name from the attrs dict?

From a quick test, it seems this deletes the name (s.name becomes None); which I suppose is the expected behaviour?

But you can indeed set it to a non-hashable value that way, which doesn't directly raise an error, but can confusingly give errors in later operations:

In [9]: s = pd.Series([1, 2, 3], name='a')  

In [10]: s.attrs  
Out[10]: {'name': 'a'}

In [12]: s.attrs['name'] = [1, 2] 

In [13]: s 
Out[13]: 
0    1
1    2
2    3
Name: [1, 2], dtype: int64

In [14]: s.to_frame()  
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-78ea8628cc3c> in <module>
----> 1 s.to_frame()

~/scipy/pandas/pandas/core/series.py in to_frame(self, name)
   1533         """
   1534         if name is None:
-> 1535             df = self._constructor_expanddim(self)
   1536         else:
   1537             df = self._constructor_expanddim({name: self})

~/scipy/pandas/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    456                 data_columns = list(data.dtype.names)
    457                 data = {k: data[k] for k in data_columns}
--> 458                 if columns is None:
    459                     columns = data_columns
    460                 mgr = init_dict(data, index, columns, dtype=dtype)

TypeError: unhashable type: 'list'

[jorisvandenbossche]

Do we need to further discuss @shoyer's concern?

I was planning to try this out in GeoPandas (to see if it can replace our usage of _metadata), to be able to give feedback on this feature that way, but didn't get to it yet.

[TomAugspurger]

Sorry I missed @shoyer's comment. I'll take a look.

[buhrmann]

Hi, not sure this is the best issue for my comments below, or too early, but I've been wanting for a while to store metadata along with Series or Dataframes. I've tried the subclassing approach, with _constructor_expanddim, _constructor etc., but eventually there was always some method that lead to the loss of my metadata. It seems that this new approach doesn't really solve the problem either (not sure if it's supposed to work like this):

In [17]: s = pd.Series([0,1,2], name="x")                                                                                                                                                                                                                                       
In [18]: s.attrs["mydata"] = "test"                                                                                                                                                                                                                                             
In [19]: s.attrs                                                                                                                                                                                                                                                                
Out[19]: {'mydata': 'test'}

In [20]: df = s.to_frame()                                                                                                                                                                                                                                                      
In [21]: df.x.attrs                                                                                                                                                                                                                                                             
Out[21]: {}

Is this supposed to work? Or do I misinterpret the purpose of attrs?

Edit:
For reference, it seems it may only be the "expanddim" direction from Series to DF doesn't seem to work (using s.to_frame() or the Dataframe constructor directly), while setting a column's attrs that already exists in a DF, and indexing or manipulating etc. seems to maintain the metadata correctly (though I haven't tested more complicated cases like aggregations etc.).

[TomAugspurger]

Pandas will need to call finalize in more places to ensure the metadata is propagated.

[jorisvandenbossche]

@buhrmann can you open a new issue with this specific use case?

[dotsdl]

Is it appearing likely that DataFrame.attrs is here to stay? I'm considering using this for metadata used to differentiate DataFrames of two forms in alchemlyb, but wanted to check the temperature of the pandas devs on this question before relying on it. So far it works as expected (e.g. it survives such things as DataFrame.copy())!