Pull of container freeze fails

[pditommaso]

In some conditions, the pull of a container freeze fails with the following error message.

  Unexpected response for 'https://wave.stage-tower.net/v2/wt/6d4a095d2606/pditommaso/wave-demo/manifests/0c53d27e6581ae02628c83fd391276e8': [404] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"repository 'docker.io/pditommaso/wave-demo:0c53d27e6581ae02628c83fd391276e8' not found"}]}

The problem seems related to the registry where the container is going to be hosted, in this test docker.io. The same test work correctly using AWS ECR as the host registry.

Hopw to replicate

1. Build the latest nextflow version from master

2. Creare a repository on docker.io where upload the freeze containers

3. Add in the nextflow config those settings

   wave.freeze = true
   build.repository = 'docker.io/<USER>/<REPO>'
   

4. Run a simple script that triggers the wave container freeze.

It looks like the problem is that the nextflow client receives a 404 http error code, here

WaveClient

However, that should not happen because the container should being built, and this snippet should make it await instead of returning 404

wave/src/main/groovy/io/seqera/wave/service/builder/BuildStore.groovy

Lines 71 to 76 in ce338d5

	default CompletableFuture<BuildResult> awaitBuild(String imageName) {
	final result = getBuild(imageName)
	if( !result )
	return null
	return CompletableFuture<BuildResult>.supplyAsync(() -> Waiter.awaitCompletion(this,imageName,result))
	}

[pditommaso]

The complete log file

nextflow.log

[munishchouhan]

Digest is the same when we are using ECR:

Resolve (1): image wave/build/dev:7ecb6e02471aa686a70885f2f98d3348 => digest=sha256:3f56bdf32a53245472eef8b20381f245617fe8bbb3655fabef63c535565cc90a

But it's different in the case of docker.io

Resolve (1): image hrma017/dev:76d6f979c74e82606124d6144b27c7b6 => digest=sha256:a9d4f30a3251807845fd9725b363b711571dcc8496ee9a2771f55f4ecae968de

[munishchouhan]

Maybe related to this issue
docker/hub-feedback#1925

[munishchouhan]

I tried using postman and also got the different digest
I have raised a query to docker hub support

[pditommaso]

I believe there's an inconsistency how the freeze is handled that may cause this, but it should not be the cause of the 404 issue. Looking into it

[munishchouhan]

Response from docker :

The discrepancy you're experiencing with the digest values is likely due to the difference in the layers of the image. Docker image digests are computed based on the content of each layer within the image, so if there are any changes or updates to the image, the digest will be different.

The Docker image registry (in this case, registry-1.docker.io) allows you to fetch the manifest of an image using the HTTP HEAD request, but it won't provide you with the exact digest used in the Docker Hub repository. The digest you obtained (sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4) is the digest of the manifest itself, not the specific image digest.

To obtain the digest for a specific image tag from the Docker Hub repository, you can use the Docker API.

https://registry-1.docker.io/v2/hrma017/dev/manifests/sha256:8b44e394e3cea79ae83a686002c1829bc162032d4122326a9b9a0f8282550b07

Remember that if the image has been updated or changed in any way, the digest will be different, which might explain why the digest obtained from the registry is different from the one you see in the Docker Hub repository.

[pditommaso]

Found! the problem was the lack of HTTP headers in the request made by Nextflow

nextflow-io/nextflow@d39866e