Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated Serializer to be more efficient #874

Merged
merged 4 commits into from
Oct 1, 2020

Conversation

egbertbouman
Copy link
Member

This PR:

  • Updates Serializer to make it more efficient
  • Updates EZPackOverlay so that there is less string slicing and fewer lists are being created
  • Renames the (un)packing functions that take a Serializable as an argument
  • Removes pack(_multiple)/unpack(_multiple) functions
  • Removes optional_format_list as its use is limited, and it's hurting unpacking performance. I've intentionally put this in a separate commit, so it's easy to undo in case we decide to keep it.

With the updated serializer, the time spent (un)serializing typically decreases by 20+%.
For instance, for intro-request/responses the decrease varies between 21-38%. Currently:

Serializing 1,000,000 intro-request packets: 5.6937067 s
Serializing 1,000,000 intro-response packets: 7.4839985 s
Unserializing 1,000,000 intro-request packets: 7.712827699999998 s
Unserializing 1,000,000 intro-response packets: 9.7632044 s

With this PR:

Serializing 1,000,000 intro-request packets: 4.4311537 s
Serializing 1,000,000 intro-response packets: 5.894732000000001 s
Unserializing 1,000,000 intro-request packets: 4.850328299999999 s
Unserializing 1,000,000 intro-response packets: 6.008891500000001 s

@egbertbouman egbertbouman force-pushed the serializer branch 2 times, most recently from e037269 to 4775867 Compare September 30, 2020 13:21
@qstokkink
Copy link
Collaborator

It seems we are experiencing technical difficulties: Failure: 'pylint is not installed'.

@egbertbouman
Copy link
Member Author

@qstokkink Nice catch! I didn't even notice, considering the check still passed.

@qstokkink
Copy link
Collaborator

@egbertbouman I noticed a hanging parenthesis and I wondered why Pylint didn't catch it. Turns out, it wasn't running.

@qstokkink
Copy link
Collaborator

Our sincere apologies for this technical intermission, please rebase onto the latest master.

@egbertbouman
Copy link
Member Author

@qstokkink Thanks, the pylint check is working now!

@qstokkink
Copy link
Collaborator

As this is hyper-critical functionality, I'll perform an extended check beyond the PR test suite (running through the documentation, periodic network health scripts, etc.) when this PR is done. The final review will take a bit longer than usual.

@egbertbouman egbertbouman force-pushed the serializer branch 7 times, most recently from 5195cd2 to 329c7bd Compare October 1, 2020 12:12
@egbertbouman egbertbouman marked this pull request as ready for review October 1, 2020 12:15
@egbertbouman egbertbouman requested a review from qstokkink October 1, 2020 12:15
Copy link
Collaborator

@qstokkink qstokkink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have finished the first half of my review: please address these two minor points.

I will now move into the second half of this review and run this code through all of our documentation and scripts.

ipv8/REST/isolation_endpoint.py Outdated Show resolved Hide resolved
ipv8/messaging/serialization.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@qstokkink qstokkink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution, fixes and your patience. I ran through all of our scripts and documentation and all of it still works as expected.

@qstokkink qstokkink merged commit 88e32af into Tribler:master Oct 1, 2020
@egbertbouman egbertbouman deleted the serializer branch November 26, 2020 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants