Compressed JsonProperty (or Blob) exception

[eyalhei]

Hi,

I have an issue with saving and reading compressed json properties using the lib. I think the problem is on BlobProperty level.

My versions:
python 3.7
google-cloud-core==1.0.3
google-cloud-datastore==1.7.3
google-cloud-ndb==0.0.1

Reproduction code:

from google.cloud import ndb
class Test(ndb.Model):
    json_prop = ndb.JsonProperty(compressed=True)

client = ndb.Client()
with client.context():
    t = Test(json_prop={'hello': 'world'})
    k = t.put()
    t2 = k.get()
    t2.json_prop

When I try to save a new object I get an exception:

Traceback (most recent call last):
  File "venv/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-5737434b3fa6>", line 3, in <module>
    k = t.put()
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/_options.py", line 93, in wrapper
    return wrapped(*pass_args, _options=_options, **kwargs)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 4691, in _put
    return self._put_async(_options=_options).result()
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 190, in result
    self.check_success()
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 137, in check_success
    raise self._exception
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 291, in _advance_tasklet
    yielded = self.generator.send(send_value)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 4748, in put
    entity_pb = _entity_to_protobuf(self)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 602, in _entity_to_protobuf
    return helpers.entity_to_protobuf(ds_entity)
  File "venv/lib/python3.7/site-packages/google/cloud/datastore/helpers.py", line 221, in entity_to_protobuf
    _set_protobuf_value(value_pb, value)
  File "venv/lib/python3.7/site-packages/google/cloud/datastore/helpers.py", line 451, in _set_protobuf_value
    attr, val = _pb_attr_value(val)
  File "venv/lib/python3.7/site-packages/google/cloud/datastore/helpers.py", line 366, in _pb_attr_value
    raise ValueError("Unknown protobuf attr type", type(val))
ValueError: ('Unknown protobuf attr type', <class 'google.cloud.ndb.model._CompressedValue'>)

I also tried reading an object written in python27 and got the following exception:

File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 5444, in _to_dict
    values[name] = prop._get_for_dict(self)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 1849, in _get_for_dict
    return self._get_value(entity)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 1691, in _get_value
    return self._get_user_value(entity)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 1345, in _get_user_value
    return self._apply_to_values(entity, self._opt_call_from_base_type)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 1661, in _apply_to_values
    new_value = function(value)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 1402, in _opt_call_from_base_type
    value = self._call_from_base_type(value.b_val)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 1456, in _call_from_base_type
    return call(value)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 1625, in call
    new_value = method(self, value)
  File "venv/lib/python3.7/site-packages/google/cloud/ndb/model.py", line 2671, in _from_base_type
    return json.loads(value.decode("ascii"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0x9c in position 1: ordinal not in range(128)

Thanks!

[eyalhei]

Hi Again,

The read issue I solved with the following code change in BlobProperty:

    def _from_base_type(self, value):
        """Convert a value from the "base" value type for this property.

        Args:
            value (bytes): The value to be converted.

        Returns:
            Optional[bytes]: The converted value. If the current property is
            a (wrapped) compressed value, this will unwrap the value and return
            the decompressed form. Otherwise, it will return :data:`None` to
            indicate that the value didn't need to be unwrapped and
            decompressed.
        """
        if self._compressed and not isinstance(value, _CompressedValue):
            value = _CompressedValue(value)

        if isinstance(value, _CompressedValue):
            return zlib.decompress(value.z_val)

[cguardia]

Thanks a lot for this report. We'll look into it this week.

[eyalhei]

Hi, anything I can do to help?
Is there a schedule for when versions are released? (I know this is OSS so general no, just really hoping to use this in migrating my code from 2.7 to 3.7)

[cguardia]

Thanks a lot for your interest. We are planning to make a new release very soon. We'll try to figure this one out before then. It would be very helpful if you can keep testing your code and see if you spot any other issues.

[eyalhei]

Another issue that might be related, but I don't have a standalone reproduction:

1. I created an entity with JsonProperty(compressed=true) and some string field in python 2.7
2. In python 3.7 I:
   2.a. queried the entity
   2.b. updated the string field
   2.c. saved the entity using put() (all worked great)
3. When reading the entity again in python2.7 it failed saying ValueError: No JSON object could be decoded

From what I saw in the code, I guess there is some flag to let datastore know this is a compressed value and I guess it was deleted (kinda new to the entire app engine world)

[cguardia]

Thanks for following up. We'll see if there's something that can be done about the compressed flag, but for now I'll close this.