-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
speed up (de)serialization of Base bits types in abstract containers #30221
Conversation
Isn't this breaking though (because of the new list of special objects), so can only be partially backported? |
I thought we gave no guarantees about being able to deserialize data from any previous version of Julia? |
No this should be 100% backwards compatible. It just uses if statements instead of dispatch, and additional specialized methods. There were already special tags for those types. |
@@ -748,6 +756,9 @@ function handle_deserialize(s::AbstractSerializer, b::Int32) | |||
return unwrap_unionall(tname.wrapper) | |||
elseif b == OBJECT_TAG | |||
t = deserialize(s) | |||
if t === Missing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe do the same for Nothing
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nothing
already has a special representation; in a future version I'll add one for missing
and then this code can be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we just do isdefined(t, :instance) && return t.instance
here? Since deserialize
is required to return that object, I don't think there's anything very meaningful that a deserializing implementation would be permitted to do after dispatch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, but I would consider that slightly breaking so I'll do it separately.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm also inclined to say that custom deserialize
methods aren't called for primitive types (and possibly more things). Of course that's even more breaking, but it's hard to get performance otherwise (e.g. #14678).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello
The example and the fix seems to be only focusing on vector{union{int, missing}}
How about other primitive types. Eg vector {union {float64,missing}} or float32 float16 bool char?
Thank you
helps #30148
Before:
Serialize array of
Union{Missing,Int}
:Deserialize that:
Serialize array of
Union{Missing,UInt}
:Deserialize that:
Same tests with this PR:
The times are a bit variable but the basic story is clear. I tested both Int and UInt, since Int already had some special-case code. Other Base number types (Int8, UInt8, ...) should have roughly the same performance characteristics as UInt.