SOCI Snapshotter is unable to get credentials from the credentials file

[ollypom]

Describe the bug
The SOCI Snapshotter does not retrieve the registry credentials from the client (finch / nerdctl), instead the snapshotter attempts to use a credentials file, by default looking in /root/.docker/config.json. We do set a DOCKER_CONFIG variable in the VM, however this is set as a user's environment variable not the root environment variable (where we currently start the snapshotter as a background process).

If you attempt to lazy load a container image in Finch today from a registry that requires authentication (even if you have done a finch login), you will see that the container image is downloaded in full before the container is started. When checking the soci-snapshotter logs you will find that it does not have registry credentials.

Oct 26 12:18:23 lima-finch soci-snapshotter-grpc[8165]: {"error":"cannot unpack the layer: cannot fetch layer: unable to fetch descriptor (sha256:b3c399da943c0747be26ad2d7858e7c1eac894c51592dfe10c98b0737b07609d) from remote store: GET \"https://111222333444.dkr.ecr.eu-west-1.amazonaws.com/v2/nginxdemo2/blobs/sha256:b3c399da943c0747be26ad2d7858e7c1eac894c51592dfe10c98b0737b07609d\": credential required for basic auth","key":"finch/805/extract-665613406-Nxi4 sha256:2b81e2af1b742b8668716c321e420de4c844f14b7a67c16d46c238660f3acdf8","level":"warning","msg":"failed to prepare snapshot; deferring to container runtime","parent":"finch/804/sha256:ab18cb8eb19742f211a6f4cd6bc8a2ded56750b2b71a310d29a86998f3db0830","time":"2023-10-26T12:18:23.728717552Z"}

If we go down the approach of managing the SOCI snapshotter with systemd (#660) we could then pass in a environment variable to use the credential file mounted into the VM.

...
[Service]
Type=notify
ExecStart=/usr/local/bin/soci-snapshotter-grpc
Environment="DOCKER_CONFIG=/Users/<username>/.finch"
Restart=always
RestartSec=5
...

Steps to reproduce

$ export AWS_REGION=eu-west-1
$ export AWS_ACCOUNT_ID=111222333444
$ aws ecr get-login-password --region $AWS_REGION | finch login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
WARNING: Your password will be stored unencrypted in /Users/<username>/.finch/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

$ finch run --snapshotter soci $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/nginxdemo2:latest
111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginxdemo2:latest:                   resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:fd96e40d576375699bd94093a2a5005d857d252e25ab35e03294069e90d856da: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:7cbe3f4c79232396f3d55fafefb47f23aba5dee91934c68be4fc6a7e497a0b22:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5aca968bda346aa3f3ae7e781a45d10a1f17df3d45a4bc05f201b7261e127c36:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2b92a4a464539d6c28ffd6b40875226086ace1e24d6598d771d8a65a6938acb1:    downloading    |++++++++++++++++++++++----------------| 35.0 MiB/59.6 MiB
layer-sha256:b3c399da943c0747be26ad2d7858e7c1eac894c51592dfe10c98b0737b07609d:    downloading    |++++++--------------------------------| 28.8 MiB/179.7 MiB
elapsed: 12.0s                                                                    total:  63.8 M (5.3 MiB/s)

This image is being downloaded in full (see the downloading next to each layer).

Expected behavior
Image to be lazy loaded.

To help debug the issue as quickly as possible, we recommend generating a support bundle with finch support-bundle generate and attaching it to this issue. This packages all Finch-related configs and logs into one file.

[coderbirju]

Verified that soci-snapshotter is able to lazily load images now.

$ sudo finch run --snapshotter soci 020023120753.dkr.ecr.us-west-2.amazonaws.com/rabbitmq:latest
020023120753.dkr.ecr.us-west-2.amazonaws.com/rabbitmq:latest:                     resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:e0d542c428854958d1d580f9f1f8ecab59e538743fa3e2c5f4be2221e2c7835c:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:32e16c01355c4af22da981ede96fe357ff5d33ee68478ba389db8a4ddbd7c937: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:152ffd7f55d350d98ec491206a0ecaae9707be48b10ec47fe4179f81e7c0cba8:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.7 s                                                                    total:  12.0 K (7.1 KiB/s)                                       
=INFO REPORT==== 22-Nov-2024::19:58:44.781404 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}
2024-11-22 19:58:48.122836+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2024-11-22 19:58:48.130691+00:00 [notice] <0.254.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
.
.
.

Documentation here

• enable soci snapshotter as a systemd service
• lazily load images