[Merged by Bors] - simple tool to compare traces between executions #4628

mockersf · 2022-04-28T19:59:07Z

Objective

Have an easy way to compare spans between executions

Solution

Add a tool to compare spans from chrome traces

> cargo run --release  -p spancmp -- --help
   Compiling spancmp v0.1.0
    Finished release [optimized] target(s) in 1.10s
     Running `target/release/spancmp --help`
spancmp

USAGE:
    spancmp [OPTIONS] <TRACE> [SECOND_TRACE]

ARGS:
    <TRACE>
    <SECOND_TRACE>

OPTIONS:
    -h, --help                     Print help information
    -p, --pattern <PATTERN>        Filter spans by name matching the pattern
    -t, --threshold <THRESHOLD>    Filter spans that have an average shorther than the threshold
                                   [default: 0]

for each span, it will display the count, minimum duration, average duration and max duration. It can be filtered by a pattern on the span name or by a minimum average duration.

just displaying a trace

comparing two traces

superdump · 2022-04-28T20:03:28Z

Ooooooo
Aaaaahhh
???
Profit!
Maybe a terminal UI and some more and selectable metrics and comparisons :)

hymm · 2022-04-28T20:05:22Z

This is something that tracy does too (most of it at least). This seems like a nice command line tool, but what is the argument for it to be included in bevy?

superdump · 2022-04-28T20:26:58Z

Wait what? How did I miss that Tracy had that?? Awesome!

mockersf · 2022-04-29T08:52:15Z

This is something that tracy does too (most of it at least). This seems like a nice command line tool, but what is the argument for it to be included in bevy?

While I like tracy, it's an extra tool to install and more complicated to use.

I would like to promote more for PR the ability to compare spans before/after, but that's currently something that need a somewhat complicated setup. This tool was made in the idea of critcmp. Somewhere in the future, I would like a bot that post results for example trace comparison automatically to PRs, but it could be done manually for now.

I added it in Bevy because it's not a general chrome trace comparison tool, it only supports the traces emitted by Bevy, and we can make it evolve based on the stats we want to monitor.

alice-i-cecile · 2022-04-30T01:33:48Z

bors try

bors · 2022-04-30T01:36:16Z

try

Build failed:

build-wasm (nightly, ubuntu-latest)

superdump · 2022-05-03T12:16:09Z

This is something that tracy does too (most of it at least). This seems like a nice command line tool, but what is the argument for it to be included in bevy?

While I like tracy, it's an extra tool to install and more complicated to use.

Honestly, I agree. It's already clunky to switch branches to run the profiles. The Tracy workflow then involves clicking around in a GUI to open the traces, find the relevant systems in the two traces to compare, noting down the statistics to write into a discussion / issue / PR / whatever. If I could just capture the two profiles and then run a command that spits out the information I want, that's going to be a faster workflow.

I added it in Bevy because it's not a general chrome trace comparison tool, it only supports the traces emitted by Bevy, and we can make it evolve based on the stats we want to monitor.

I'm in. And you know I'll make use of this so...

alice-i-cecile · 2022-05-03T19:22:25Z

I like the justification here. I'm on board with the design direction here, although I haven't reviewed the code in depth yet.

superdump · 2022-05-03T20:10:54Z

tools/spancmp/src/main.rs

+    /// name
+    name: String,
+    /// phase
+    ph: String,
+    /// timestamp
+    ts: f32,


Suggested change

/// name

name: String,

/// phase

ph: String,

/// timestamp

ts: f32,

name: String,

phase: String,

timestamp: f32,

and fix references. I see no reason to shorten the member names, nor to document with what the member names are/should be.

The fields have the same name as in the json for deserialization. They can be annotated with the serde annotation to rename them, but I tend to prefer to stick to the same name as the json being deserialised when not part of the public api

tools/spancmp/src/main.rs

superdump · 2022-05-03T20:14:41Z

tools/spancmp/src/main.rs

+    second_trace: Option<String>,
+}
+
+const MARGIN_PERCENT: f32 = 0.02;


I just hacked this in. It should probably be a CLI argument with a default. :) Later we can base it on the standard deviation of the distributions or something statistically sensible.

tools/spancmp/src/main.rs

mockersf · 2022-05-03T23:36:42Z

new look thanks to @superdump

alice-i-cecile · 2022-05-04T00:26:59Z

Can we use short names for the systems here? We already have a tool for this (get_short_name in bevy_reflect), that I refactored in #4299

mockersf · 2022-05-04T00:39:05Z

Almost, but not exactly: here the name is not directly a rust type name, it's a string, and we want to keep part of that string.

It's also useful to keep the full type to be able to filter all spans from bevy_render for example.

System names in spans are not easy to read, but most of the info in their name is useful. That's also painful when using other traces tool 🙁

mockersf · 2022-05-04T01:08:57Z

@alice-i-cecile in fact, it's possible with some regex

I added a flag --short that will simplify system names. It has issues on some system names, we'll see once your updated version is merged if it's better

Co-Authored-By: Robert Swain <[email protected]>

mockersf · 2022-05-04T19:07:06Z

now with everything aligned and time units thanks to @superdump

superdump

I think we should merge this then continue iterating on it. :)

alice-i-cecile

I'm in agreement with Rob; this looks fine to me and I agree with the direction.

Because there's no user impact I'm happy to take a merge and iterate approach.

superdump · 2022-05-06T08:48:47Z

bors r+

# Objective - Have an easy way to compare spans between executions ## Solution - Add a tool to compare spans from chrome traces ```bash > cargo run --release -p spancmp -- --help Compiling spancmp v0.1.0 Finished release [optimized] target(s) in 1.10s Running `target/release/spancmp --help` spancmp USAGE: spancmp [OPTIONS] <TRACE> [SECOND_TRACE] ARGS: <TRACE> <SECOND_TRACE> OPTIONS: -h, --help Print help information -p, --pattern <PATTERN> Filter spans by name matching the pattern -t, --threshold <THRESHOLD> Filter spans that have an average shorther than the threshold [default: 0] ``` for each span, it will display the count, minimum duration, average duration and max duration. It can be filtered by a pattern on the span name or by a minimum average duration. just displaying a trace ![Screenshot 2022-04-28 at 21 56 21](https://user-images.githubusercontent.com/8672791/165835310-f465c6f2-9e6b-4808-803e-884b06e49292.png) comparing two traces ![Screenshot 2022-04-28 at 21 56 55](https://user-images.githubusercontent.com/8672791/165835353-097d266b-a70c-41b8-a8c1-27804011dc97.png) Co-authored-by: Robert Swain <[email protected]>

bors · 2022-05-06T08:51:43Z

Build failed:

build-wasm (nightly, ubuntu-latest)

alice-i-cecile · 2022-05-06T19:15:48Z

bors r+

# Objective - Have an easy way to compare spans between executions ## Solution - Add a tool to compare spans from chrome traces ```bash > cargo run --release -p spancmp -- --help Compiling spancmp v0.1.0 Finished release [optimized] target(s) in 1.10s Running `target/release/spancmp --help` spancmp USAGE: spancmp [OPTIONS] <TRACE> [SECOND_TRACE] ARGS: <TRACE> <SECOND_TRACE> OPTIONS: -h, --help Print help information -p, --pattern <PATTERN> Filter spans by name matching the pattern -t, --threshold <THRESHOLD> Filter spans that have an average shorther than the threshold [default: 0] ``` for each span, it will display the count, minimum duration, average duration and max duration. It can be filtered by a pattern on the span name or by a minimum average duration. just displaying a trace ![Screenshot 2022-04-28 at 21 56 21](https://user-images.githubusercontent.com/8672791/165835310-f465c6f2-9e6b-4808-803e-884b06e49292.png) comparing two traces ![Screenshot 2022-04-28 at 21 56 55](https://user-images.githubusercontent.com/8672791/165835353-097d266b-a70c-41b8-a8c1-27804011dc97.png) Co-authored-by: Robert Swain <[email protected]>

bors · 2022-05-06T19:53:38Z

Pull request successfully merged into main.

Build succeeded:

# Objective - Have an easy way to compare spans between executions ## Solution - Add a tool to compare spans from chrome traces ```bash > cargo run --release -p spancmp -- --help Compiling spancmp v0.1.0 Finished release [optimized] target(s) in 1.10s Running `target/release/spancmp --help` spancmp USAGE: spancmp [OPTIONS] <TRACE> [SECOND_TRACE] ARGS: <TRACE> <SECOND_TRACE> OPTIONS: -h, --help Print help information -p, --pattern <PATTERN> Filter spans by name matching the pattern -t, --threshold <THRESHOLD> Filter spans that have an average shorther than the threshold [default: 0] ``` for each span, it will display the count, minimum duration, average duration and max duration. It can be filtered by a pattern on the span name or by a minimum average duration. just displaying a trace ![Screenshot 2022-04-28 at 21 56 21](https://user-images.githubusercontent.com/8672791/165835310-f465c6f2-9e6b-4808-803e-884b06e49292.png) comparing two traces ![Screenshot 2022-04-28 at 21 56 55](https://user-images.githubusercontent.com/8672791/165835353-097d266b-a70c-41b8-a8c1-27804011dc97.png) Co-authored-by: Robert Swain <[email protected]>

github-actions bot added the S-Needs-Triage This issue needs to be labelled label Apr 28, 2022

alice-i-cecile added C-Performance A change motivated by improving speed, memory usage or compile times A-Log C-Usability A targeted quality-of-life change that makes Bevy easier to use and removed S-Needs-Triage This issue needs to be labelled labels Apr 29, 2022

bors bot added a commit that referenced this pull request Apr 30, 2022

Try #4628:

d455e80

superdump requested changes May 3, 2022

View reviewed changes

mockersf and others added 11 commits May 4, 2022 20:56

simple tool to compare traces

a6dc92e

clippy

48de83e

fix order for comparison

934edd2

new format

67af174

Co-Authored-By: Robert Swain <[email protected]>

cleanup

84657a9

Co-Authored-By: Robert Swain <[email protected]>

fix value

083ed6f

Co-Authored-By: Robert Swain <[email protected]>

additional comment

4361ff7

can simplify system names

d0d22b8

spancmp: Improved formatting

09ad43a

spancmp: Fix single-trace and span-only-in-second formatting

4802e02

add missing newlines

9ebc428

mockersf added 4 commits May 4, 2022 20:56

bold span name

461dccc

read as a stream

86b05fa

split into modules

0c34ef7

making sure spancmp is checked by CI

001f75a

mockersf force-pushed the spancmp branch from b186cb5 to 001f75a Compare May 4, 2022 19:03

revert changes on number of frames

75017bd

superdump approved these changes May 5, 2022

View reviewed changes

alice-i-cecile added the S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it label May 5, 2022

alice-i-cecile approved these changes May 5, 2022

View reviewed changes

bors bot changed the title ~~simple tool to compare traces between executions~~ [Merged by Bors] - simple tool to compare traces between executions May 6, 2022

bors bot closed this May 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - simple tool to compare traces between executions #4628

[Merged by Bors] - simple tool to compare traces between executions #4628

mockersf commented Apr 28, 2022

superdump commented Apr 28, 2022

hymm commented Apr 28, 2022

superdump commented Apr 28, 2022

mockersf commented Apr 29, 2022 •

edited

Loading

alice-i-cecile commented Apr 30, 2022

bors bot commented Apr 30, 2022

superdump commented May 3, 2022

alice-i-cecile commented May 3, 2022

superdump May 3, 2022

mockersf May 3, 2022 •

edited

Loading

superdump May 3, 2022

mockersf commented May 3, 2022

alice-i-cecile commented May 4, 2022 •

edited

Loading

mockersf commented May 4, 2022 •

edited

Loading

mockersf commented May 4, 2022 •

edited

Loading

mockersf commented May 4, 2022

superdump left a comment

alice-i-cecile left a comment

superdump commented May 6, 2022

bors bot commented May 6, 2022

alice-i-cecile commented May 6, 2022

bors bot commented May 6, 2022

[Merged by Bors] - simple tool to compare traces between executions #4628

[Merged by Bors] - simple tool to compare traces between executions #4628

Conversation

mockersf commented Apr 28, 2022

Objective

Solution

superdump commented Apr 28, 2022

hymm commented Apr 28, 2022

superdump commented Apr 28, 2022

mockersf commented Apr 29, 2022 • edited Loading

alice-i-cecile commented Apr 30, 2022

bors bot commented Apr 30, 2022

try

superdump commented May 3, 2022

alice-i-cecile commented May 3, 2022

superdump May 3, 2022

Choose a reason for hiding this comment

mockersf May 3, 2022 • edited Loading

Choose a reason for hiding this comment

superdump May 3, 2022

Choose a reason for hiding this comment

mockersf commented May 3, 2022

alice-i-cecile commented May 4, 2022 • edited Loading

mockersf commented May 4, 2022 • edited Loading

mockersf commented May 4, 2022 • edited Loading

mockersf commented May 4, 2022

superdump left a comment

Choose a reason for hiding this comment

alice-i-cecile left a comment

Choose a reason for hiding this comment

superdump commented May 6, 2022

bors bot commented May 6, 2022

alice-i-cecile commented May 6, 2022

bors bot commented May 6, 2022

mockersf commented Apr 29, 2022 •

edited

Loading

mockersf May 3, 2022 •

edited

Loading

alice-i-cecile commented May 4, 2022 •

edited

Loading

mockersf commented May 4, 2022 •

edited

Loading

mockersf commented May 4, 2022 •

edited

Loading