Handle buffers in `Widget._should_send_property` #1595

vidartf · 2017-08-06T11:09:49Z

Previous check in _should_send_property() did not handle buffers. After this change, the comparison is performed in the serialized state which also prevents creation of new widgets/instances during comparison.

Fixes #1592.

Previous test did not handle buffers.. After this change, the comparison is performed in the serialized state which also prevents creation of new widgets/instances during comparison.

Ensure that Widget.set_state sends the expected update messages

Also, fix the case when the state is serialized with keys in a different order, by parsing the json we generated with jsondumps.

Also add copyright, change tests to use nose assert for better error messages.

…ay, or python 3 bytes. We disregard the python 2 buffer type - it is obsolete, even on python 2. See jupyter-widgets#1194

jasongrout · 2017-08-09T09:12:50Z

@maartenbreddels - the last commit (7d56205) changes the definition of what is a binary object in python for the custom serializers - basically it adds bytearray and gets rid of the python 2 buffer (which is superseded by memoryview, even on python 2).

What do you think?

jasongrout · 2017-08-09T09:14:33Z

@vidartf - what do you think of my changes? I feel like this is ready to merge now, so if you and @maartenbreddels approve it, let's merge.

maartenbreddels · 2017-08-09T09:23:03Z

LGTM 👍

vidartf · 2017-08-09T09:52:03Z

ipywidgets/widgets/widget.py

+            split_value = _remove_buffers({ key: to_json(value, self)})
+            split_lock = _remove_buffers({ key: self._property_lock[key]})
+            # Compare state and buffer_paths
+            if (jsonloads(jsondumps(split_value[0])) == jsonloads(jsondumps(split_lock[0]))


I would rather pass sort_keys=True to dumps rather than pass it back through loads, but it is not critical and I have not profiled the two.

Done. I also factored out this comparison into a top-level function.

vidartf · 2017-08-09T09:55:48Z

ipywidgets/widgets/widget.py

@@ -483,7 +483,7 @@ def _compare(self, a, b):
        else:
            return a == b

-    def _compare_buffers(self, a, b, key):
+    def _buffer_list_equal(self, a, b):


If we have this as an extension point, keeping the key could be very informative as to which kind of comparison to perform. If anything, it's a problem with the current _compare that it does not have it.

E.g. for certain keys you might want to do a floating point comparison with a given precision, while for another you need exact equality. Without the key, you will not be able to determine which comparison to apply.

Can you elaborate on this being an extension point? I don't understand how you're thinking of this as an extension point.

I mean extension point as in "something an inheriting class might want to override".

Actually, this comparison depends on the top-level put/remove buffer function implementations, so I factored out the comparison to also be a top-level function.

I mean extension point as in "something an inheriting class might want to override".

Ah. Since it depends on the implementation of the top-level functions (i.e., what exactly can be in the buffers list), I think it makes more sense to not override it, and instead put it at the top level too.

The current implementation works for me, but if someone uses a serializer/deserialize set that might introduce noise when data is passed through a loop (a != serializer(deserialize(a))), e.g. floating point inaccuracies, they will want to override this comparison.

It is still possible to do that by overriding the _should_send_property function, but it will be more brittle as it needs to take into account the internals of that method (e.g. _property_lock).

Actually, I might be over-engineering this one. Other fields in the state other than buffers might also have this issue, so there is no obvious reason to give buffers preferential treatment.

vidartf · 2017-08-09T10:01:36Z

Went through it, and only found some minor points commented above. Neither are sticking points for me, so feel free to make the final decision on them as you see fit and merge.

Compare deterministic json strings instead of rehydrating them into objects.

jasongrout · 2017-08-09T13:33:33Z

One more iteration to try to avoid copying if possible (in python 3, or python 2 with byte memoryviews). @maartenbreddels, @vidartf - does 0f0761c look good to you?

Edit: actually, 0f0761c

jasongrout · 2017-08-09T14:09:26Z

@vidartf said on gitter:

@jasongrout Does memoryview.tobytes() actually copy the buffer? I think your changes are ok either way, but just FMI

vidartf added 3 commits August 6, 2017 13:06

Fix change test in _should_send_property

6445ea9

Previous test did not handle buffers.. After this change, the comparison is performed in the serialized state which also prevents creation of new widgets/instances during comparison.

Add tests for Widget.set_state

afe6ae6

Ensure that Widget.set_state sends the expected update messages

Fix comment typos

40cd07c

vidartf force-pushed the fix-should-send branch from f830665 to 40cd07c Compare August 6, 2017 13:04

jasongrout modified the milestone: 7.0 Aug 7, 2017

Add test for when the serializer returns bytes.

349f556

jasongrout force-pushed the fix-should-send branch from be6df90 to 349f556 Compare August 9, 2017 02:40

Fix the property lock when the deserializer returns bytes or buffer

db04817

Also, fix the case when the state is serialized with keys in a different order, by parsing the json we generated with jsondumps.

jasongrout force-pushed the fix-should-send branch from 0dbb6b7 to db04817 Compare August 9, 2017 06:56

jasongrout added 2 commits August 9, 2017 03:23

Fix test for python 2

4d912e4

Also add copyright, change tests to use nose assert for better error messages.

Change the binary types for custom serializers to memoryview, bytearr…

7d56205

…ay, or python 3 bytes. We disregard the python 2 buffer type - it is obsolete, even on python 2. See jupyter-widgets#1194

vidartf commented Aug 9, 2017

View reviewed changes

Factor out json and buffer list compare.

b9eb8c8

Compare deterministic json strings instead of rehydrating them into objects.

Compare without copying if possible.

0f0761c

jasongrout force-pushed the fix-should-send branch from 1c4d8a6 to 0f0761c Compare August 9, 2017 13:35

jasongrout merged commit ee00a61 into jupyter-widgets:master Aug 9, 2017

vidartf deleted the fix-should-send branch August 9, 2017 20:59

github-actions bot added the resolved-locked Closed issues are locked after 30 days inactivity. Please open a new issue for related discussion. label Feb 11, 2021

github-actions bot locked as resolved and limited conversation to collaborators Feb 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle buffers in `Widget._should_send_property` #1595

Handle buffers in `Widget._should_send_property` #1595

vidartf commented Aug 6, 2017 •

edited

Loading

jasongrout commented Aug 9, 2017 •

edited

Loading

jasongrout commented Aug 9, 2017

maartenbreddels commented Aug 9, 2017

vidartf Aug 9, 2017

jasongrout Aug 9, 2017

vidartf Aug 9, 2017

vidartf Aug 9, 2017

jasongrout Aug 9, 2017

vidartf Aug 9, 2017

jasongrout Aug 9, 2017 •

edited

Loading

jasongrout Aug 9, 2017 •

edited

Loading

vidartf Aug 9, 2017 •

edited

Loading

vidartf Aug 9, 2017

vidartf commented Aug 9, 2017

jasongrout commented Aug 9, 2017 •

edited

Loading

jasongrout commented Aug 9, 2017

Handle buffers in Widget._should_send_property #1595

Handle buffers in Widget._should_send_property #1595

Conversation

vidartf commented Aug 6, 2017 • edited Loading

jasongrout commented Aug 9, 2017 • edited Loading

jasongrout commented Aug 9, 2017

maartenbreddels commented Aug 9, 2017

vidartf Aug 9, 2017

Choose a reason for hiding this comment

jasongrout Aug 9, 2017

Choose a reason for hiding this comment

vidartf Aug 9, 2017

Choose a reason for hiding this comment

vidartf Aug 9, 2017

Choose a reason for hiding this comment

jasongrout Aug 9, 2017

Choose a reason for hiding this comment

vidartf Aug 9, 2017

Choose a reason for hiding this comment

jasongrout Aug 9, 2017 • edited Loading

Choose a reason for hiding this comment

jasongrout Aug 9, 2017 • edited Loading

Choose a reason for hiding this comment

vidartf Aug 9, 2017 • edited Loading

Choose a reason for hiding this comment

vidartf Aug 9, 2017

Choose a reason for hiding this comment

vidartf commented Aug 9, 2017

jasongrout commented Aug 9, 2017 • edited Loading

jasongrout commented Aug 9, 2017

Handle buffers in `Widget._should_send_property` #1595

Handle buffers in `Widget._should_send_property` #1595

vidartf commented Aug 6, 2017 •

edited

Loading

jasongrout commented Aug 9, 2017 •

edited

Loading

jasongrout Aug 9, 2017 •

edited

Loading

jasongrout Aug 9, 2017 •

edited

Loading

vidartf Aug 9, 2017 •

edited

Loading

jasongrout commented Aug 9, 2017 •

edited

Loading