[Podman] Supports SQLite containers database back-end to support newer versions (4.8+) #24373

tbavelier · 2024-04-04T09:40:50Z

What does this PR do?

Introduces a lightweight SQLite database client sqlite_db_client.go for podman having exact parity with the current BoltDB client db_client.go to support newer Podman versions
Modifies the detection of the podman feature : podman_db_path is provided by the user OR /var/lib/containers/storage exists (previously, only if /var/lib/containers exists)
Modifies config.go to default podman_db_path to an empty string instead of /var/lib/containers/storage/libpod/bolt_state.db
Modifies the podman workloadmeta collector behaviour :
- previously, it would be created if /var/lib/containers exists on the system : this could cause a loop of logs about /var/lib/containers/storage/libpod/bolt_state.db not being accessible to the host Agent (running under dd-agent)
- Instead, the collector now ensures the db file exists (is accessible to the Agent) and ends with a known extension .sql or .db to select the client to use

Motivation

Anticipate feature request when customers eventually upgrade podman to newer versions that default to the SQLite backend (with the BoltDB backend being deprecated in 5.0+)

Additional Notes

Similar to the current BoltDB client, we use a slightly tweaked AllContainers function from the podman Go package returning the same type, an array of containers with their configuration and their state.

Possible Drawbacks / Trade-offs

~~Introduces a new Go dependency~~
Unlike podman, we use a pure-Go SQLite implementation (modernc.org/sqlite) as the CGo implementation (github.com/mattn/go-sqlite3) breaks a test (inv -e integration-tests --race --remote-docker) as this test environment is based on docker/compose:debian-1.29.2 which uses an old glibc version. Per the library author, this implementation is a bit slower (16% in their benchmark) but this should be acceptable considering the low podman usage out-there, nor do we rely much on it as we only open a connection to run a single query
Increases the size of the dogstatsd stand-alone binary which uses the podman build tag : 38338.890625 kB on 182707b to 41726.609375 kB on 5b73966.
- This does not apply to the full Agent (e.g. inv -e agent.build) where the increase is only half a MB : this affects the DSD binary as it has a very limited amount of build tags, and thus imports.

Describe how to test/QA your changes

2 environments/versions of podman will be needed :

A ubuntu 22.04 vm where podman 3.4.x (BoltDB) can be installed easily with sudo apt-get install podman -yq (.deb releases are very far behind main)
A CentOS Stream VM (e.g. stream 8) to benefit from the latest binaries with sudo yum -y install podman (4.8.x+)

podman containers can be run both in rootless and rootfull mode : both behaviours should be tested.

Rootfull :

sudo podman run -d --name dd-agent \
    --cgroupns host --pid host \
    -v /var/lib/containers/storage:/var/lib/containers/storage/:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e DD_API_KEY=<> \
    -e DD_HOSTNAME=rootfull-podman \
    <REPLACE ME WITH CUSTOM IMAGE>

Rootless (e.g. run by the ubuntu user) :

podman run -d --name dd-agent \
    --cgroupns host --pid host \
    -v /home/ubuntu/.local/share/containers/storage:/var/lib/containers/storage/:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e DD_API_KEY=<> \
    -e DD_HOSTNAME=rootless-podman \
    <REPLACE ME WITH CUSTOM IMAGE>

Ensure the podman workloadmeta collector is working in both rootfull and rootless by running agent workload-list -v : assess containers with runtime podman are present matching ids from the (sudo) podman ps command
Assess podman feature is automatically detected as /var/lib/containers/storage is mounted inside your Agent
Test the usage of a different podman_db_path, e.g. using -v /var/lib/containers/storage:/abc:ro (mounting the storage file in /abc) and -e DD_PODMAN_DB_PATH=/abc/libpod/bolt_state.db (providing the path to the DB location inside the Agent)
Assess the matrix below (feature parity to current podman support) (depends on the cgroups version on the VM) :
- cgroups v1 :
  - Rootless :
    - container metrics are NOT available
    - live containers are collected
  - Rootfull :
    - container metrics are available
    - live containers are collected
- cgroups v2 :
  - Rootless :
    - container metrics are available
    - live containers are collected
  - Rootfull :
    - container metrics are available
    - live containers are collected

cit-pr-commenter · 2024-04-04T09:52:28Z

Go Package Import Differences

Baseline: 5c66b7c
Comparison: ba21d7f

binary	os	arch	change
dogstatsd	linux	amd64	+29, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/libc/wctype +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib
dogstatsd	linux	arm64	+28, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib
process-agent	linux	amd64	+29, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/libc/wctype +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib
process-agent	linux	arm64	+28, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib
security-agent	linux	amd64	+29, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/libc/wctype +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib
security-agent	linux	arm64	+28, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib
trace-agent	linux	amd64	+29, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/libc/wctype +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib
trace-agent	linux	arm64	+28, -0 +github.com/remyoudompheng/bigfft +modernc.org/libc +modernc.org/libc/errno +modernc.org/libc/fcntl +modernc.org/libc/fts +modernc.org/libc/grp +modernc.org/libc/honnef.co/go/netdb +modernc.org/libc/langinfo +modernc.org/libc/limits +modernc.org/libc/netdb +modernc.org/libc/netinet/in +modernc.org/libc/poll +modernc.org/libc/pthread +modernc.org/libc/pwd +modernc.org/libc/signal +modernc.org/libc/stdio +modernc.org/libc/stdlib +modernc.org/libc/sys/socket +modernc.org/libc/sys/stat +modernc.org/libc/sys/types +modernc.org/libc/termios +modernc.org/libc/time +modernc.org/libc/unistd +modernc.org/libc/uuid/uuid +modernc.org/mathutil +modernc.org/memory +modernc.org/sqlite +modernc.org/sqlite/lib

pr-commenter · 2024-04-04T10:43:33Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=31645091 --os-family=ubuntu

pr-commenter · 2024-04-04T11:08:47Z

Regression Detector

Regression Detector Results

Run ID: 51dcfc62-12d5-4542-bda1-bde48a790d45
Baseline: 5c66b7c
Comparison: ba21d7f

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_to_blackhole	% cpu utilization	+2.23	[-4.02, +8.48]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_tree	memory utilization	+2.23	[+2.11, +2.36]
➖	file_to_blackhole	% cpu utilization	+2.23	[-4.02, +8.48]
➖	idle	memory utilization	+1.08	[+1.03, +1.12]
➖	process_agent_standard_check	memory utilization	+0.88	[+0.81, +0.94]
➖	process_agent_real_time_mode	memory utilization	+0.85	[+0.80, +0.90]
➖	basic_py_check	% cpu utilization	+0.78	[-1.74, +3.30]
➖	process_agent_standard_check_with_stats	memory utilization	+0.61	[+0.55, +0.67]
➖	otel_to_otel_logs	ingress throughput	+0.07	[-0.39, +0.52]
➖	trace_agent_json	ingress throughput	+0.01	[-0.02, +0.03]
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.20, +0.20]
➖	trace_agent_msgpack	ingress throughput	-0.00	[-0.00, +0.00]
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.02	[-0.07, +0.02]
➖	tcp_syslog_to_blackhole	ingress throughput	-0.15	[-0.24, -0.06]
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.30	[-3.17, +2.58]
➖	pycheck_1000_100byte_tags	% cpu utilization	-2.00	[-6.85, +2.85]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

pkg/config/config_template.yaml

aliciascott

Just a small suggestion to add thew word Note in bold, but approved, also your discretion if you prefer to have that entire line italicized instead.

releasenotes/notes/podman-sqlite-backend-support-8437c6d5254b39ef.yaml

Co-authored-by: Alicia Scott <[email protected]>

…ents

carlosroman

Happy to bump the DogStatsD binary size check to 42 Mb

tbavelier · 2024-04-12T07:48:18Z

/merge

dd-devflow · 2024-04-12T07:48:25Z

🚂 MergeQueue

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

dd-devflow · 2024-04-12T09:02:15Z

🚂 MergeQueue

Pull request added to the queue.

This build is going to start soon! (estimated merge in less than 24m)

Use /merge -c to cancel this operation!

…r versions (4.8+) (#24373) * go import + licenses + release note * sqlite client + modifies wlm podman init * new podman detection and config * Doc suggestion on release note formatting Co-authored-by: Alicia Scott <[email protected]> * Use pure-go SQLite driver instead of CGo implementatioon * Increase max_dsd binary size to 42 MB --------- Co-authored-by: Alicia Scott <[email protected]>

hkrutzer · 2024-05-17T10:09:51Z

#22387

tbavelier added 3 commits April 4, 2024 11:34

go import + licenses + release note

a570c43

sqlite client + modifies wlm podman init

8d3d251

new podman detection and config

dc7f2e2

tbavelier mentioned this pull request Apr 4, 2024

[Podman] Supports SQLite containers database back-end to support newer versions (4.8+) #24194

Closed

8 tasks

tbavelier added team/containers team/container-app team/agent-shared-components labels Apr 4, 2024

tbavelier added this to the 7.54.0 milestone Apr 4, 2024

tbavelier marked this pull request as ready for review April 4, 2024 11:20

tbavelier requested review from a team as code owners April 4, 2024 11:20

pgimalac reviewed Apr 4, 2024

View reviewed changes

pkg/config/config_template.yaml Show resolved Hide resolved

aliciascott approved these changes Apr 4, 2024

View reviewed changes

releasenotes/notes/podman-sqlite-backend-support-8437c6d5254b39ef.yaml Outdated Show resolved Hide resolved

tbavelier and others added 4 commits April 5, 2024 13:07

Doc suggestion on release note formatting

491c840

Co-authored-by: Alicia Scott <[email protected]>

Use pure-go SQLite driver instead of CGo implementatioon

5b73966

Merge branch 'main' into tbavelier/podman_sqlite_support_and_improvem…

023e032

…ents

Increase max_dsd binary size to 42 MB

ba21d7f

tbavelier requested review from a team as code owners April 8, 2024 15:55

pgimalac approved these changes Apr 8, 2024

View reviewed changes

mfpierre approved these changes Apr 9, 2024

View reviewed changes

vboulineau approved these changes Apr 9, 2024

View reviewed changes

carlosroman self-requested a review April 11, 2024 09:06

carlosroman reviewed Apr 11, 2024

View reviewed changes

FlorentClarret approved these changes Apr 11, 2024

View reviewed changes

chouetz approved these changes Apr 12, 2024

View reviewed changes

carlosroman self-requested a review April 12, 2024 09:01

carlosroman approved these changes Apr 12, 2024

View reviewed changes

dd-mergequeue bot merged commit 295c091 into main Apr 12, 2024
185 checks passed

dd-mergequeue bot deleted the tbavelier/podman_sqlite_support_and_improvements branch April 12, 2024 09:32

tbavelier mentioned this pull request May 17, 2024

[BUG] Podman 4.8+ no longer uses boltdb by default, breaking integration #22387

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Podman] Supports SQLite containers database back-end to support newer versions (4.8+) #24373

[Podman] Supports SQLite containers database back-end to support newer versions (4.8+) #24373

tbavelier commented Apr 4, 2024 •

edited

Loading

cit-pr-commenter bot commented Apr 4, 2024 •

edited

Loading

pr-commenter bot commented Apr 4, 2024 •

edited

Loading

pr-commenter bot commented Apr 4, 2024 •

edited

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

aliciascott left a comment •

edited

Loading

carlosroman left a comment

tbavelier commented Apr 12, 2024

dd-devflow bot commented Apr 12, 2024

dd-devflow bot commented Apr 12, 2024

hkrutzer commented May 17, 2024

[Podman] Supports SQLite containers database back-end to support newer versions (4.8+) #24373

[Podman] Supports SQLite containers database back-end to support newer versions (4.8+) #24373

Conversation

tbavelier commented Apr 4, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

cit-pr-commenter bot commented Apr 4, 2024 • edited Loading

Go Package Import Differences

pr-commenter bot commented Apr 4, 2024 • edited Loading

Test changes on VM

pr-commenter bot commented Apr 4, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

aliciascott left a comment • edited Loading

Choose a reason for hiding this comment

carlosroman left a comment

Choose a reason for hiding this comment

tbavelier commented Apr 12, 2024

dd-devflow bot commented Apr 12, 2024

dd-devflow bot commented Apr 12, 2024

hkrutzer commented May 17, 2024

tbavelier commented Apr 4, 2024 •

edited

Loading

cit-pr-commenter bot commented Apr 4, 2024 •

edited

Loading

pr-commenter bot commented Apr 4, 2024 •

edited

Loading

pr-commenter bot commented Apr 4, 2024 •

edited

Loading

aliciascott left a comment •

edited

Loading