jupyter_client >= 7.0.0 breaks ipydatagrid

[ibdafna]

Hi team,

This issue surfaced during a routine PR in the ipydatagrid repo. There were no code changes to trigger this on the ipydatagrid side, and our CI only shows this as failing for Python >= 3.7, Python 3.6 seems to work fine with >=7.0.0. Wondering if there's anything we need to change on the ipydatagrid side, or if it's an issue with jupyter_client.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_2726/322807178.py in <module>
     42     layout={"height": "180px"},
     43     header_renderer=header_renderer,
---> 44     default_renderer=default_renderer,
     45 )
     46 

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/ipydatagrid/datagrid.py in __init__(self, dataframe, **kwargs)
    355 
    356         self.data = dataframe
--> 357         super().__init__(**kwargs)
    358         self._cell_click_handlers = CallbackDispatcher()
    359         self._cell_change_handlers = CallbackDispatcher()

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/ipywidgets/widgets/widget.py in __init__(self, **kwargs)
    413 
    414         Widget._call_widget_constructed(self)
--> 415         self.open()
    416 
    417     def __del__(self):

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/ipywidgets/widgets/widget.py in open(self)
    436                 args['comm_id'] = self._model_id
    437 
--> 438             self.comm = Comm(**args)
    439 
    440     @observe('comm')

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/ipykernel/comm/comm.py in __init__(self, target_name, data, metadata, buffers, **kwargs)
     55             if self.primary:
     56                 # I am primary, open my peer.
---> 57                 self.open(data=data, metadata=metadata, buffers=buffers)
     58             else:
     59                 self._closed = False

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/ipykernel/comm/comm.py in open(self, data, metadata, buffers)
     92                               data=data, metadata=metadata, buffers=buffers,
     93                               target_name=self.target_name,
---> 94                               target_module=self.target_module,
     95                               )
     96             self._closed = False

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/ipykernel/comm/comm.py in _publish_msg(self, msg_type, data, metadata, buffers, **keys)
     69             parent=self.kernel.get_parent("shell"),
     70             ident=self.topic,
---> 71             buffers=buffers,
     72         )
     73 

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/jupyter_client/session.py in send(self, stream, msg_or_type, content, parent, ident, buffers, track, header, metadata)
    828         if self.adapt_version:
    829             msg = adapt(msg, self.adapt_version)
--> 830         to_send = self.serialize(msg, ident)
    831         to_send.extend(buffers)
    832         longest = max([len(s) for s in to_send])

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/jupyter_client/session.py in serialize(self, msg, ident)
    702             content = self.none
    703         elif isinstance(content, dict):
--> 704             content = self.pack(content)
    705         elif isinstance(content, bytes):
    706             # content is already packed, as in a relayed message

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/jupyter_client/session.py in json_packer(obj)
     97         default=json_default,
     98         ensure_ascii=False,
---> 99         allow_nan=False,
    100     )
    101 

~/work/ipydatagrid/ipydatagrid/.tox/py37/lib/python3.7/site-packages/zmq/utils/jsonapi.py in dumps(o, **kwargs)
     23     Keyword arguments are passed along to :py:func:`json.dumps`.
     24     """
---> 25     return json.dumps(o, **kwargs).encode("utf8")
     26 
     27 

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    236         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    237         separators=separators, default=default, sort_keys=sort_keys,
--> 238         **kw).encode(obj)
    239 
    240 

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/json/encoder.py in encode(self, o)
    197         # exceptions aren't as detailed.  The list call should be roughly
    198         # equivalent to the PySequence_Fast that ''.join() would do.
--> 199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
    201             chunks = list(chunks)

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot)
    255                 self.key_separator, self.item_separator, self.sort_keys,
    256                 self.skipkeys, _one_shot)
--> 257         return _iterencode(o, 0)
    258 
    259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

TypeError: keys must be str, int, float, bool or None, not tuple

=========================== short test summary info ============================
FAILED examples/ConditionalFormatting.ipynb::Cell 5
FAILED examples/Nested Hierarchies.ipynb::Cell 0
=================== 2 failed, 66 passed in 205.59s (0:03:25) ===================
ERROR: InvocationError for command /home/runner/work/ipydatagrid/ipydatagrid/.tox/py37/bin/pytest --nbval examples (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py37: commands failed
Error: Process completed with exit code 1.

[davidbrochart]

I feel this could be related to @martinRenou's changes in #698 or ipython/ipykernel#708.

[martinRenou]

I couldn't reproduce with the examples from the ipydatagrid repo.

It seems that the example you are running is trying to serialize a dictionary with tuples as dictionary keys, which is not valid JSON. Is it something specific to the example you're running, or is it ipydatagrid itself which generates this dictionary?

If ipydatagrid generates this dictionary, I could suggest to have a custom serializer that makes it valid JSON.

[ibdafna]

Thank for the input @martinRenou - I am taking a look again. What puzzles me is that we didn't change anything in ipydatagrid, it is the jupyter_client version which breaks this. It's also odd that this is happening in python >= 3.7, but not 3.6.

[ibdafna]

Opened jupyter-widgets/ipydatagrid#273 for reference. This issue does not seem to be caused by any specific ipydatagrid code.

[martinRenou]

Thank for the input @martinRenou - I am taking a look again. What puzzles me is that we didn't change anything in ipydatagrid, it is the jupyter_client version which breaks this

Yes, more specifically it might be the PR in ipykernel: ipython/ipykernel#708 this makes json_clean a no-op for jupyter-client 7, relying entirely on jupyter-client for the serialization. I'm looking into this.