Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added options to benchmark orders vs certs #106

Merged
merged 3 commits into from
Dec 30, 2021

Conversation

oxade
Copy link
Contributor

@oxade oxade commented Dec 30, 2021

147423677-f4d0546c-1280-4de3-9285-0d00c9f907ba

Reasoning:
The image above roughly shows the execution flow for a FastX transfer request.
Current benchmark measures B-C-D and F-G-H and returns the sum.
From #93, we determined that measuring one or the other might be important.

Solution
This PR provides the option to measure B-C-D and F-G-H separately, or both via CLI selector.
Will enhance this in future to optionally measure end-to-end (B-H and B-L)

Example (outputs are compacted):

Run for OrdersAndCerts: default benchmark of Certs and Orders (B-C-D and F-G-H)

mbp> ./bench --benchmark-type OrdersAndCerts                     
[2021-12-30T16:05:21Z INFO  bench] Starting benchmark of: OrdersAndCerts
.
.
.
[2021-12-30T16:05:29Z INFO  fastpay::network] Done sending Udp requests to 127.0.0.1:9563 (shard 8)
[2021-12-30T16:05:29Z INFO  bench] Received 80000 responses.
[2021-12-30T16:05:29Z WARN  bench] Completed benchmark for OrdersAndCerts
    Total time: 1548267us, items: 40000, tx/sec: 25835.337186673874

Run for OrdersOnly benchmark (B-C-D)

mbp> ./bench --benchmark-type OrdersOnly
[2021-12-30T16:05:33Z INFO  bench] Starting benchmark of: OrdersOnly
[2021-12-30T16:05:33Z INFO  bench] Preparing accounts.
.
.
.
[2021-12-30T16:05:39Z INFO  fastpay::network] Done sending Udp requests to 127.0.0.1:9562 (shard 7)
[2021-12-30T16:05:39Z INFO  bench] Received 40000 responses.
[2021-12-30T16:05:39Z WARN  bench] Completed benchmark for OrdersOnly
    Total time: 463116us, items: 40000, tx/sec: 86371.44905380078

Run for CertsOnly benchmark (F-G-H)

mbp> ./bench --benchmark-type CertsOnly
[2021-12-30T16:05:44Z INFO  bench] Starting benchmark of: CertsOnly
[2021-12-30T16:05:44Z INFO  bench] Preparing accounts.
.
.
.
[2021-12-30T16:05:51Z INFO  fastpay::network] Done sending Udp requests to 127.0.0.1:9567 (shard 12)
[2021-12-30T16:05:51Z INFO  bench] Received 40000 responses.
[2021-12-30T16:05:51Z WARN  bench] Completed benchmark for CertsOnly
   Total time: 1149791us, items: 40000, tx/sec: 34788.931205758265

timeOrders: 463116 + timeCerts: 1149791 = 1612907 is greater than timeOrdersAndCerts: 1548267 as expected

@oxade oxade changed the title Added CLI options to benchmark orders vs certs Added options to benchmark orders vs certs Dec 30, 2021
@lxfind
Copy link
Contributor

lxfind commented Dec 30, 2021

Cool. Thanks! Could we allow specify these benchmark types by name instead of "benchmark-type 1/2/3"? It will be difficult to remember which one is which when using the command line.

@oxade
Copy link
Contributor Author

oxade commented Dec 30, 2021

Cool. Thanks! Could we allow specify these benchmark types by name instead of "benchmark-type 1/2/3"? It will be difficult to remember which one is which when using the command line.

@lxfind of course!
Btw if you specify the --help, it tells you what the numbers mean.
However what names do you recommend, if thats easier?

mbp> ./bench --help            
FastPay Benchmark 0.1.0
Local end-to-end test and benchmark of the FastPay protocol
    throughput
......
OPTIONS:
        --benchmark-type <benchmark-type>      Which execution path to track. 1 = Orders + Certs, 2 = Orders Only, 3 =
                                               Certs Only [default: 1]

@lxfind
Copy link
Contributor

lxfind commented Dec 30, 2021

I see. If it's not too much code/effort, ideally I think it would be nice to be able to just:

./bench OrdersAndCerts

or

./bench --type OrdersAndCerts

@oxade oxade marked this pull request as draft December 30, 2021 16:36
@oxade
Copy link
Contributor Author

oxade commented Dec 30, 2021

I see. If it's not too much code/effort, ideally I think it would be nice to be able to just:

./bench OrdersAndCerts

or

./bench --type OrdersAndCerts

Done

@oxade oxade marked this pull request as ready for review December 30, 2021 21:52
@oxade oxade merged commit 0c802fc into main Dec 30, 2021
@huitseeker huitseeker deleted the selectively-benchmark-different-code-paths branch February 2, 2022 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants