Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address big integers bug #892

Merged
merged 16 commits into from
Mar 3, 2021

Conversation

ssteinbach
Copy link
Collaborator

@ssteinbach ssteinbach commented Feb 20, 2021

Closes #884

  • Asserts that the only integer data type that OTIO serializes/deserializes is int64_t
  • ...which means that OTIO serializes into JSON:
    • bool
    • int64_t
    • string
    • IEEE754 double (including NaN, Inf, -Infinity support)
    • list
    • dictionary
    • things that derive from SerializableObject
    • RationalTime, TimeTransform and TimeRange
  • An exception is raised if an invalid type or value is inserted into PyAny:
    • TypeError: A value of type '<type 'set'>' is incompatible with OpenTimelineIO. OpenTimelineIO only supports the following value types in AnyDictionary containers (like the .metadata dictionary): ('int', 'float', 'str', 'bool', 'list', 'dictionary', 'opentime.RationalTime', 'opentime.TimeRange', 'opentime.TimeTransform').
    • ValueError: A value of 36893488147419103232 is outside of the range of integers that OpenTimelineIO supports, [-9223372036854775808, 9223372036854775807], which is the range of C++ int64_t.
  • Adds documentation about this behavior and a unit test that hopefully makes this behavior explicit
  • Adds an explicit test for the C++ SDK to the python unit test suite (just one function, but its a start)

TODO:

  • make the exception that is raised when an invalid type is inserted more readable, and have it explain what the invalid types are
  • update the documentation to reflect where we settled on for this behavior

@ssteinbach ssteinbach added this to the Public Beta 14 milestone Feb 20, 2021
@ssteinbach
Copy link
Collaborator Author

I'm hoping to not do that in this PR... I think that entails a bunch of other small changes to true the whole class up to int64_t, and can be done in a later PR

@reinecke
Copy link
Collaborator

Ahh, gotcha. Sounds good.

src/opentimelineio/deserialization.cpp Outdated Show resolved Hide resolved
tests/test_timeline.py Show resolved Hide resolved
- to remove integer ambiguity around 32/64bit integers, we coerce all
  ints to int64_t.
- When a uint64_t is serialized, it is bitwise & with INT64_MAX first to
  avoid overflowing the integer.  In order to do that comparison at
  serialization time, the uint64_t needs to be stored as a uint64_t all
  the way through the python bindings into the any mapping.
- the next step is to move the & operation up into the python bindings
  and prevent people from storing invalid numbers in OTIO files.
 Please enter the commit message for your changes. Lines starting
- this is to allow parity with the values in python files
- add support in the serializer/deserializer for NaN, Inf, -Infinity etc
  from rapidjson
- block pybind11 from automatically casting numbers that don't fit into
  double or int64_t into bool
- add a unit test and test data to test the serialization and behavior
@ssteinbach ssteinbach added the bug A problem, flaw, or broken functionality. label Mar 2, 2021
- Adds specific messages that test if something is an out of range
  integer, or an invalid type
- includes a list of supported types
- The 'Writer' part of SerializableObject needed the same tightening as
  the RapidJSON dispatcher
- this is used for things that serialize and compare JSON (like cloning
  and the `is_equivalent_to` method.
- Because the ImageSequenceReference schema has `int` in it (which
  should at some point be changed to int64_t), it was tripping this
  check on certain platforms
Copy link
Collaborator

@reinecke reinecke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great clarification!

@codecov-io
Copy link

Codecov Report

Merging #892 (5cac58f) into master (5676338) will decrease coverage by 0.04%.
The diff coverage is 80.40%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #892      +/-   ##
==========================================
- Coverage   86.15%   86.10%   -0.05%     
==========================================
  Files         183      184       +1     
  Lines       17719    17850     +131     
  Branches     1970     1982      +12     
==========================================
+ Hits        15265    15369     +104     
- Misses       1956     1977      +21     
- Partials      498      504       +6     
Flag Coverage Δ
unittests 86.10% <80.40%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/opentimelineio/safely_typed_any.cpp 80.76% <0.00%> (-15.07%) ⬇️
src/opentimelineio/serializableObject.h 90.38% <ø> (+0.38%) ⬆️
src/opentimelineio/serialization.cpp 82.31% <20.00%> (-1.83%) ⬇️
...ntimelineio/opentimelineio-bindings/otio_utils.cpp 82.55% <25.00%> (-4.03%) ⬇️
tests/test_image_sequence_reference.py 97.59% <63.63%> (-2.41%) ⬇️
...-opentimelineio/opentimelineio/core/_core_utils.py 78.48% <71.42%> (-0.60%) ⬇️
tests/test_cxx_sdk_bindings.py 71.42% <71.42%> (ø)
tests/test_timeline.py 93.93% <87.50%> (-4.09%) ⬇️
src/opentimelineio/deserialization.cpp 60.48% <100.00%> (+1.12%) ⬆️
src/opentimelineio/imageSequenceReference.cpp 94.18% <100.00%> (+0.67%) ⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5676338...5cac58f. Read the comment docs.

@meshula meshula merged commit 8a7bd25 into AcademySoftwareFoundation:master Mar 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A problem, flaw, or broken functionality.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Large positive integers in OTIO JSON file misinterpreted as negative
5 participants