Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StructuredProperty promotions expected to find properties separated by periods at a depth of 1; received ['promotions'] #178

Closed
chmoder opened this issue Aug 28, 2019 · 3 comments · Fixed by #184
Assignees
Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. 🚨 This issue needs some love. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@chmoder
Copy link
Contributor

chmoder commented Aug 28, 2019

Hi, this issue is related to the very well documented #122.

This error happened when trying to access the entity using appengine27 after saving it with appengine37.

My question is, should we consider the entities to be not backward compatible with the old NDB library after using this one?

Steps to reproduce

  1. use python-ndb to update an entity on appengine37 with a property like: promotions = ndb.StructuredProperty(Promotion, repeated=True)
  2. try to read this entity using the appengine27 ndb library.

Stack trace

StructuredProperty promotions expected to find properties separated by periods at a depth of 1; received ['promotions'] (/base/data/home/apps/p~csgapi-qa/qa.420639802264340764/base.py:219)
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/p~csgapi-qa/qa.420639802264340764/auth.py", line 381, in post
    use_memcache=False
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1253, in get
    return self.get_async(**q_options).get_result()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
    self.check_success()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1266, in _get_async
    res = yield self.fetch_async(1, **q_options)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 995, in _run_to_list
    batch = yield rpc
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion
    result = rpc.get_result()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 615, in get_result
    return self.__get_result_hook(self)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2951, in __query_result_hook
    self.__results = self._process_results(query_result.result_list())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2984, in _process_results
    for result in results]
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 194, in pb_to_query_result
    return self.pb_to_entity(pb)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 691, in pb_to_entity
    entity = modelclass._from_pb(pb, key=key, set_key=False)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3216, in _from_pb
    _property_map[property_map_key]._deserialize(ent, p)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/941d77da994078b1/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 2439, in _deserialize
    (self._name, depth, parts))
RuntimeError: StructuredProperty promotions expected to find properties separated by periods at a depth of 1; received ['promotions']
@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Aug 29, 2019
@chrisrossi chrisrossi self-assigned this Aug 29, 2019
@chrisrossi chrisrossi added 🚨 This issue needs some love. and removed triage me I really want to be triaged. labels Aug 29, 2019
@chrisrossi
Copy link
Contributor

I'll work on updating how we store StructuredProperties to be backwards compatible.

@yoshi-automation yoshi-automation added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. 🚨 labels Sep 3, 2019
chrisrossi pushed a commit that referenced this issue Sep 4, 2019
The NDB rewrite, when storing values for structured properties, has been taking advantage of the native Datastore capability to arbitrarily nest entities as property values. Data written using the new NDB, however, would not be compatible with the legacy (Python 2) version, which always only stores a single entity in Datastore, using dotted property names to store the values of structured properties.

This patch introduces a new context flag, legacy_data, which, when True, causes NDB to store data in such a way as to be backwards compatible with the old version. The flag is True by default, requiring users to make a conscious decision to break backwards compatibility.

Fixes #178
@chmoder
Copy link
Contributor Author

chmoder commented Sep 6, 2019

I just wanted to let you know we got some time this week to test the update and it appears to serialize the repeated StructuredProperty back to period separated properties.

Thank you!

@andrewsg
Copy link
Contributor

andrewsg commented Sep 6, 2019 via email

@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Apr 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. 🚨 This issue needs some love. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants