Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(cmdstate): don't store execSetup and environment in state #478

Merged
merged 1 commit into from
Aug 12, 2024

Conversation

benhoyt
Copy link
Contributor

@benhoyt benhoyt commented Aug 9, 2024

Per #411, the environment (and other task data) can get quite large for exec objects. Given that exec commands are one-shot and only relevant for the current run of Pebble, there's no need to persist them.

Note that none of this data is in "api-data" so it's not accessible via the API at all right now, and so this is a non-breaking change.

Later if we want to make exec tasks a bit more introspectable we can add the command line or other fields to "api-data" (but we shouldn't add the entire environment).

Fixes #411.

Per canonical#411, the environment (and other task data) can get quite large for
exec objects. Given that exec commands are one-shot and only relevant
for the current run of Pebble, there's no need to persist them.

Note that none of this data is in "api-data" so it's not accessible via
the API at all right now, and so this is a non-breaking change.

Later if we want to make exec tasks a bit more introspectable we can
add the command line or other fields to "api-data" (but we shouldn't
add the entire environment).

Fixes canonical#411.
@dmitry-lyfar
Copy link
Contributor

dmitry-lyfar commented Aug 9, 2024

A very naive comparison does see the difference, it seems. Every run was for a clean, empty state.

#!/usr/bin/bash

for i in {1..50}; do
  pebble exec -- echo "perftest" > /dev/null &
done

wait
echo "Done."

Before:
bash ./naive.sh > /dev/null 0.13s user 0.21s system 43% cpu 0.780 total
After:
bash ./naive.sh > /dev/null 0.13s user 0.19s system 105% cpu 0.311 total

After growing the number of simultaneous execs to 500, the difference becomes quite dramatic as the old version that keeps the exec setup in the state file gives up and starts to fail requests with a timeout whilst the new one can handle them all (i9-12900HK, fast SSD). Given that the state is clean for every test as with repetitive runs a growing state file degrades the performance seriously.

@benhoyt
Copy link
Contributor Author

benhoyt commented Aug 12, 2024

Thanks for the review and the performance numbers, @dmitry-lyfar!

@benhoyt benhoyt merged commit d56c1b8 into canonical:master Aug 12, 2024
16 checks passed
@benhoyt benhoyt deleted the shrink-exec branch August 12, 2024 00:46
jujubot added a commit to juju/juju that referenced this pull request Aug 29, 2024
#17991

Per discussion with @hpidcock, update Juju 3.6 to use the just-released Pebble v1.16.0 ([changelog](https://github.com/canonical/pebble/releases/tag/v1.16.0)). Summary of main changes:

* Improvements to how services with dependencies are started and stopped (using "lanes"), so that independent services are started in parallel, and dependent services start up serially. Error handling is also improved. Fix in canonical/pebble#437.
* A fix for a tricky bug in pebble exec, so it doesn't lose output in interactive mode. Fix in canonical/pebble#466!
 prdesc Reduce the size of exec tasks by not storing the execSetup (which includes all environment variables) in state. This speeds up (and shrinks the data) when serialising state to disk during state.Unlock. Fix is in canonical/pebble#478.
 prdesc Ensure Pebble doesn't hang (with no error message) when the state file directory is read-only or otherwise inaccessible. Fix in canonical/pebble#481.
* Re-implement warnings using notices. This was always the intention since the notices feature was added (it was designed as a superset of warnings), but we'd never gotten to it. Lots of nice code deletion in canonical/pebble#486.

## QA steps

Deploy a K8s charm, like `snappass-test`, and ensure `pebble version --client` on the workload reports v1.16.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Every exec stores several KB of JSON in state (due to environment)
2 participants