Pulumi updates docker image with no changes even with Awsx.Ecr.Image.CacheFrom set

[snebjorn]

What happened?

Pulumi wants to update the docker image even though it didn't change

     Type                                  Name                Plan        Info
     pulumi:pulumi:Stack                   stack-name
     ├─ awsx:ecr:Image                     docker-image
 ~   │  └─ docker:index:Image              latest              update      [diff: ~build]

I tried using CacheFrom but to no avail. It still wants to update the image.

var ecrRepo = new Ecrx.Repository("ecr");

var imageTag = "latest";
var dockerImage = new Ecrx.Image(
    $"docker-image",
    new()
    {
        ImageTag = imageTag,
        CacheFrom = { ecrRepo.Url.Apply(url => $"{url}:{imageTag}") },
        RepositoryUrl = ecrRepo.Url,
        Context = "../App",
        Platform = "linux/amd64",
    }
);

Note that it works when you run pulumi from the same machine. However if you switch to a different machine and deploy from that it wants to update the image.
Sometimes it also wants to update on the same machine, but I couldn't reproduce this every time.

Example

Machine A:

• git pull
• pulumi up
• observe the image is uploaded/updated as expected ✅

Machine B:

• git pull
• pulumi up
• observe the image is updated even though no code was changed ❌

Output of pulumi about

CLI
Version      3.117.0
Go Version   go1.22.3
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  aws     6.35.0
resource  awsx    2.10.0
resource  docker  4.5.3
language  dotnet  unknown
resource  random  4.16.2

Host
OS       Microsoft Windows 11 Enterprise
Version  10.0.22631 Build 22631
Arch     x86_64

Dependencies:
NAME                  VERSION
Pulumi.Awsx           2.10.0
Pulumi.Random         4.16.2
PulumiCSharpAnalyzer  0.1.0

Additional context

For CacheFrom to work you need to docker login to the ecr.
See https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html

It would be neat if Pulumi could handle this docker login

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

Sorry for the trouble @snebjorn. Thanks for opening the issue. This looks like an issue with AWSX (or potentially the docker provider). Transferring this to the AWSX repo and someone will take a look soon.

[corymhall]

@snebjorn there could be several things going on here that make it difficult to diagnose (and I do not have multiple machines to test on). Even though you are running the build in docker, the build environment between builds of the same base image could be different.

It might help if you could provide a more complete example including

• Dockerfile that can be built
• Any debug logs that go along with the diff

[snebjorn]

@corymhall I made a small repo that should demonstrate the issue. Again I don't know how to reproduce this on a single machine.
https://github.com/snebjorn/Pulumi-cache

However I encountered a related issue.
I manually deleted the image from the ECR in AWS and ran pulumi refresh pulumi up and it didn't care the image was gone. It didn't even detect it was gone nor wanted to upload it again.

See no image in the ECR

$ aws ecr list-images --repository-name ecr-<random-pulumi-id>
{
    "imageIds": []
}

pulumi refresh

$ pulumi refresh
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing refresh (dev):
     Type                           Name              Plan
     pulumi:pulumi:Stack            Pulumi-cache-dev
     ├─ awsx:ecr:Image              docker-image
     │  └─ docker:index:Image       latest
     └─ awsx:ecr:Repository         ecr
        ├─ aws:ecr:LifecyclePolicy  ecr
        └─ aws:ecr:Repository       ecr
Resources:
    6 unchanged

Do you want to perform this refresh?
No resources will be modified as part of this refresh; just your stack's state will be.
 yes
Refreshing (dev):
     Type                           Name              Status
     pulumi:pulumi:Stack            Pulumi-cache-dev
     ├─ awsx:ecr:Image              docker-image
     │  └─ docker:index:Image       latest
     └─ awsx:ecr:Repository         ecr
        ├─ aws:ecr:LifecyclePolicy  ecr
        └─ aws:ecr:Repository       ecr
Resources:
    6 unchanged

Duration: 4s

pulumi up

$ pulumi up     
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing update (dev):
     Type                 Name              Plan
     pulumi:pulumi:Stack  Pulumi-cache-dev
Resources:
    6 unchanged

Do you want to perform this update? yes
Updating (dev):
     Type                 Name              Status
     pulumi:pulumi:Stack  Pulumi-cache-dev
Resources:
    6 unchanged

Duration: 6s

As you can see Pulumi didn't notice the image was gone. But I have set the cache to use the ECR in AWS.
What is it checking the cache against?

After pulumi up was run from a different machine it wanted to update the image again.

$ pulumi up          
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing update (dev):
     Type                      Name              Plan       Info
     pulumi:pulumi:Stack       Pulumi-cache-dev
     └─ awsx:ecr:Image         docker-image
 ~      └─ docker:index:Image  latest            update     [diff: ~build]
Resources:
    ~ 1 to update
    5 unchanged

Do you want to perform this update? no
confirmation declined, not proceeding with the update

[snebjorn]

Are you able to reproduce this? Do you need any additional information?

[ffMathy]

We also have this issue in our team.

[frederickmannings]

Bumping this issue - seems to occur on my side in a similar manner to OP.