Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark comparing read and write throughput #479

Merged
merged 1 commit into from
Feb 3, 2022

Conversation

markmandel
Copy link
Member

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation

/kind feature

/kind hotfix

What this PR does / Why we need it:

Wanted to be able to highlight if we had bottlenecks in performance on read vs write operations on the proxy.

This adds an extra benchmark to throughput.rs called "readwrite" and follows a similar pattern as the overall throughput benchmark, with both direct and proxies traffic utilised as extra comparison values.

Which issue(s) this PR fixes:

Work on #410

Special notes for your reviewer:

This is my first benchmark in Rust - feedback appreciated.

Would be interested if we think the new benchmark should be in a readwrite.rs ? Since it reused some machinery and was still essentially about throughput I figured it would make sense to stay in the same file, but wasn't 100% on it.

Wanted to be able to highlight if we had bottlenecks in performance on
read vs write operations on the proxy.

This adds an extra benchmark to throughput.rs called "readwrite" and
follows a similar pattern as the overall throughput benchmark, with both
direct and proxies traffic utilised as extra comparison values.

Work on googleforgames#410
@markmandel markmandel added kind/feature New feature or request area/performance Anything to do with Quilkin being slow, or making it go faster. area/build-tools Development tooling. labels Feb 1, 2022
@markmandel markmandel requested a review from XAMPPRocky February 1, 2022 22:49
@github-actions github-actions bot added the size/m label Feb 1, 2022
@markmandel
Copy link
Member Author

Noting some interesting results from the benchmark. I've attached one of the readwrite runs as a PDF.
readwrite Summary - Criterion.rs.pdf

On average, throughput looks about the same:

image

Were I see some differences (and it's not 100% consistent, so we should triple check the results, and I should run it without other processes on the machine, for sure - but it does align with what I've seen previously on Xonotic+grafana graphs), is that the outliers on read tend to be much larger than that of write.

I'm cherry picking here, but to illustrate the general trend I'm seeing here is read/508 bytes:
image

And the corresponding write/508 bytes:
image

This can fluctuate between runs though, so I should run this in a cleaner environment and do multiple runs before definitely drawing conclusions.

The other thought I had too was it might be worth doing benchmarks with multiple udp clients (maybe more than current cpu on the machine?) and multiple servers, and look at the same things.

Thoughts?

@quilkin-bot
Copy link
Collaborator

Build Succeeded 🥳

Build Id: 9fe53ea8-bc27-47ab-bc16-2b98c6a9ea5a

To build this version:

git fetch [email protected]:googleforgames/quilkin.git pull/479/head:pr_479 && git checkout pr_479
cargo build

@XAMPPRocky
Copy link
Collaborator

Would be interested if we think the new benchmark should be in a readwrite.rs ?

I think it's fine for now, and maybe split it up once we have even more benchmarks.

The other thought I had too was it might be worth doing benchmarks with multiple udp clients (maybe more than current cpu on the machine?) and multiple servers, and look at the same things.

I think this would be a good variable to benchmark, another variable worth benchmarking in my opinion is the amount of filters in a given configuration. For example, how does it look with 5 filters vs 10? We'd probably just want to use the Pass filters for that because we really just want to benchmark the internal machinary rather the performance of an individual filter (though I could also adding benchmarks for specific filters as well [routing, compression, etc]).

@XAMPPRocky XAMPPRocky merged commit 2f6b6ba into googleforgames:main Feb 3, 2022
@markmandel markmandel deleted the benchmark/readwrite branch February 3, 2022 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build-tools Development tooling. area/performance Anything to do with Quilkin being slow, or making it go faster. kind/feature New feature or request size/m
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants