Small artifacts for bloat reports #9331
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kpschoedel
force-pushed
the
nubloat
branch
5 times, most recently
from
e16da0a to
0a266db
Compare
kpschoedel
changed the title
kpschoedel
force-pushed
the
nubloat
branch
2 times, most recently
from
379650a to
a569bdd
Compare
pullapprove
bot
requested review from
andy31415,
bzbarsky-apple,
chrisdecenzo,
Damian-Nordic,
hawk248,
jepenven-silabs and
msandstedt
msandstedt
approved these changes
Sep 8, 2021
|
Note: the bloat check workflow is expected to run this with something like : (in this example highlighting size increases over 1%) |
|
Do you have an example of the reports will look like? |
mspang
reviewed
Sep 9, 2021
A small fake-data example is below. For actual runs, the table would contain all the available builds. I'll try to add a fuller example later. PR #1234: Size comparison from d571fb9 to e1ea910 Increases above 1.0% from d571fb9 to e1ea910:
1 build
|
Nice.. I've really missed the baseline in the current reports. |
mspang
approved these changes
Sep 9, 2021
|
Here's an example of what a full report would look like, given the workflows in this PR. (This is a comparison of two handy builds on my fork, and doesn't match an actual commit or PR.) Size comparison from a184f57 to df7771f Increases above 1.0% from a184f57 to df7771f:
40 builds
|
#### Problem
The current bloat report requires preserving large binaries, and has
trouble matching tree parents for comparison.
#### Change overview
This change makes example builds generate small artifacts containing
json file(s) containing build item identification and section sizes.
Artifacts names have the form “Size,PR,WORKFLOW,CURRENT_SHA,PARENT_SHA”;
these contain one or more json files, each containing a report on one
build target.
The reporting scripts then examines the list of available artifacts to
list of artifacts to identify pairs with the same $WORKFLOW where one's
$CURRENT_SHA is the other's $PARENT_SHA, and only then downloads and
processes those artifacts to generate comparisons for matching builds.
- Changes to `examples-…` workflows:
- Adds some `GH_EVENT_…` variables to workflow `env`, to identify
the current run.
- Uses the `gh_sizes.py` script to generate size report json files.
- Uploads size report artifacts.
- Changes to `scripts/tools/memory`:
- Adds minimal platform config files for recently-added platforms.
- Adds markdown output options (required for github comments) along
with some associated cleanup.
- Adds a script `scripts/tools/memory/gh_sizes.py` for use by workflows;
this is similar to `report_summary.py` with a suitable consistent set
of arguments.
- Adds a script `scripts/tools/memory/gh_report.py` to analyze size
report artifacts.
- Modifies `bloat_check.py` to ignore the size report artifacts.
The github comments produced by `gh_report.py` are slightly different
fromt the existing reports. Since this change enables reports for many
more builds, a full report will be over a hundred lines, and is placed
inside a details tag. Only size changes above a configurable threshold
are called out ‘above the fold’.
Note that this PR does _not_ include a change to actually send size
report comments on github; this is left to a followup after that code
has been verified on real-world size artifacts.
#### Testing
Fork-CI runs and offline verification on resulting artifacts.
bzbarsky-apple
approved these changes
Sep 10, 2021
- restore original scripts/examples/esp_example.sh and scripts/examples/gn_efr32_example.sh - avoid a problem using pyelftools `describe_p_type`
|
Size increase report for "esp32-example-build" from 2addf00
Full report output |
kpschoedel
added a commit
to kpschoedel/connectedhomeip
that referenced
this pull request
Sep 30, 2021
#### Problem The current bloat report requires preserving large binaries, and has trouble matching tree parents for comparison. In a24a6c3 (PR project-chip#9331), many workflows started uploading small json artifacts contains size information for their builds, but these have not yet been used to generate reports. #### Change overview This change adds a run of `scripts/tools/memory/gh_report.py` to the periodic bloat check workflow, with tight rate limiting in case there are unexpected problems. #### Testing Manually run offline with various data sets, and minimally on the live repository.
andy31415
pushed a commit
that referenced
this pull request
Sep 30, 2021
#### Problem The current bloat report requires preserving large binaries, and has trouble matching tree parents for comparison. In a24a6c3 (PR #9331), many workflows started uploading small json artifacts contains size information for their builds, but these have not yet been used to generate reports. #### Change overview This change adds a run of `scripts/tools/memory/gh_report.py` to the periodic bloat check workflow, with tight rate limiting in case there are unexpected problems. #### Testing Manually run offline with various data sets, and minimally on the live repository.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Small-artifact bloat reports
Problem
The current bloat report requires preserving large binaries, and has
trouble matching tree parents for comparison.
Change overview
This change makes example builds generate small artifacts containing
json file(s) containing build item identification and section sizes.
Artifacts names have the form “Size,PR,WORKFLOW,CURRENT_SHA,PARENT_SHA”;
these contain one or more json files, each containing a report on one
build target.
The reporting scripts then examines the list of available artifacts to
list of artifacts to identify pairs with the same $WORKFLOW where one's
CURRENT_SHA is the other's PARENT_SHA, and only then downloads and
processes those artifacts to generate comparisons for matching builds.
Changes to
examples-…workflows:GH_EVENT_…variables to workflowenv, to identifythe current run.
gh_sizes.pyscript to generate size report json files.Changes to
scripts/tools/memory:with some associated cleanup.
scripts/tools/memory/gh_sizes.pyfor use by workflows;this is similar to
report_summary.pywith a suitable consistent setof arguments.
scripts/tools/memory/gh_report.pyto analyze sizereport artifacts.
Modifies
bloat_check.pyto ignore the size report artifacts.The github comments produced by
gh_report.pyare slightly differentfromt the existing reports. Since this change enables reports for many
more builds, a full report will be over a hundred lines, and is placed
inside a details tag. Only size changes above a configurable threshold
are called out ‘above the fold’.
Note that this PR does not include a change to actually send size
report comments on github; this is left to a followup after that code
has been verified on real-world size artifacts.
Testing
Fork-CI runs and offline verification on resulting artifacts.