Remove the base URI for docker.io images

[ggrebert]

This feature allow users to use a mirror registry
to bypass the docker hub rate limit.

fix #37912

[quarkus-bot]

/cc @gsmet (hibernate-search), @yrodiere (hibernate-search)

[quarkus-bot]

Thanks for your pull request!

The title of your pull request does not follow our editorial rules. Could you have a look?

• title should preferably start with an uppercase character (if it makes sense!)

_{This message is automatically generated by a bot.}

[gastaldi]

Shouldn't the same apply to other registries as well?

[ggrebert]

For quay.io it is more complicated: the --registry-mirror option works only with relative image.

But we can had a property to replace this value. But in an other PR,

[gsmet]

@cescoffier @holly-cummins @Sanne could you have a look at this one? I remember we added the docker.io registry specifically at some point. Don't remember exactly why but I wonder if it was related to Podman.

[holly-cummins]

I'd forgotten all about this, but 57332c3 confirms the decision to be explicit about the name was related to Podman, and short names. It was an issue that @Sanne found.

#31253 (comment) has more details:

Short names of containers. Testcontainers and Quarkus Dev Services also expect the container service they make requests against to be non-interactive. If someone has multiple registries configured, and when using short image names, Podman responds with a prompt asking which registry should be used to pull images. Testcontainers defaults to using short names internally, so users need to change this default this prompt by setting the short-name-mode="disabled" configuration property of Podman in /etc/containers/registries.conf. containers/podman#16400 may be tracking this, or it may be a related testcontainers + podman issue. I can't quite tell.

@n1hility was going to check if it was fixed (he thinks it might be), but I can't find the response. A possible workaround is "to pull them by full name before running any tests."

I agree that not hardcoding the registry everywhere seems desirable ... this might be something where the podman CI is useful for validating the impact of this PR. (Although it looks like the Podman CI is suffering from another issue and needs some attention anyway.)

[cescoffier]

Yes, the registry name was required by podman. I'm finding references about something also required on fedora 35+.

Also, aren't all these images customizable by the user? Using the prefix also guarantees the origin of the image (and somewhat avoids having someone publishing a tainted image on a mirror).

[ggrebert]

Yes, the registry name was required by podman. I'm finding references about something also required on fedora 35+.

Also, aren't all these images customizable by the user? Using the prefix also guarantees the origin of the image (and somewhat avoids having someone publishing a tainted image on a mirror).

Image are customizable by the user (actual workaround).
But (to my known and searches) there is not a prefix property to change the DNS of devservices images.

[cescoffier]

I'm not sure a prefix would help:

• should it be added to all images without prefix?
• should all images be prefix-less, but what do we do for images from quay or google?

[ggrebert]

• should it be added to all images without prefix?

For me no. Because images without prefix are hosted on docker.io and can be mirrored directly by docker or podman.

• should all images be prefix-less, but what do we do for images from quay or google?

Images hosted on quay.io and gcr.io have no rate limit. And some are not replicated on docker.io, so we cannot use prefix-less for those.
But it could be interesting to have a property to change easily the DNS (for internet less networks).

quarkus.devservices.registry.quay.dns=quay.io
quarkus.devservices.registry.gcr.dns=gcr.io

[cescoffier]

To do that, we need to parse the image names and replace the DNS (by the way, I'm not sure of the wording here; I would prefer something like registry-hostname (and even hostname is not perfect as you can pass IPs).

The next question is about the scope. Should it be global (like in your proposal) or per image (which is literally what we have today, as you can replace the image)? I like the global approach as it's more in phase with the idea of a mirror. However, as I'm not an expert on image registry mirroring, don't we have the risk of having a mirror containing some images but not others (like if some images are private, the mirror may not have them)?

Finally, when the user overrides the image name, should we still apply the hostname? I would say no, but let's make sure we all agree.

[ggrebert]

new proposal:

quarkus.devservices.registry.quay.host=localhost:5000
quarkus.devservices.registry.gcr.host=gcr.mirror.internal

For private images, we already have to override the image property. So I am in tune with you:

When we override the image name
And the value use a full path
Then pull the image form this location

When we override the image name
And the value not use a full path
Then use the docker/podman configuration to pull the image
   (`docker.io` if no mirror is set)

So:

• if a user wants to pull a custom image from the quay.io registry, he must use a full path.
• if a user wants to pull a custom image from the docker.io registry, he can use full path ou relative path.

The good: this solution don't change the current behaviour)

[cescoffier]

Completely agree with this new proposal.

[ggrebert]

cool,

What do you prefer ? :

1. I rebase this PR (and check if no new image have been haded) and merge it to implement the docker.io/mirror part and I do a new one for the other registries ?
2. edit this PR which implement all the features

[ggrebert]

to me, I prefer the fisrt one

[cescoffier]

I would have done everything in this PR, but we can start with docker.io only and then do another PR with the other registries (less critical as it does not have rate limits)

[ggrebert]

ready to merge for me.

Issue created for the other registries: #38146

[cescoffier]

Why didn't you implement what we discussed:

• a new property to change the host name of the docker.io registry
• keep the docker.io name and apply the new host

[cescoffier]

@ggrebert ping?

[ggrebert]

problem of understanding on my part.

I thought that for docker.io we remove the host to allow the usage of the native --registry-mirror option.

When we override the image name
And the value not use a full path
Then use the docker/podman configuration to pull the image
(docker.io if no mirror is set)

If I understand correctly what you want, we never use the native configuration for a mirror. We always use a quarkus property to overide a host, even for docker.io which is a default regardless of the engine used

[cescoffier]

Yes, I think it's better, as it makes things more homogenous with the other registries.

[ggrebert]

OK, thx @cescoffier for clarifying the subject.

It can be take time to me due to all the different way the docker images are set in all the extensions and i work on it only on my free time.
But I am on it 🤘

[gsmet]

@ggrebert any progress on this one?

[ggrebert]

Hello @cescoffier and @gsmet,

Sorry, I really haven't had time to work on this as much as I would have hoped
(a complicated divorce that takes up all my free time...).

However, I still studied @cescoffier solution.
But it is not so easy to implement:

I was going to create a helper in ConfigureUtil.java which replaces the host.

But we then have to modify all the DevServices in each extension.
The real problem with this solution is for external extensions such as Quarkiverse.
And has you know, the Quarkiverse extentions don't use the latest Quarkus version,
So all extensions must update Quarkus before implementing the solution.

Note:
Tthere already exist docker.io images without host (solution based on registry-mirror) as proposed in the original PM.
example for MySQL

[Sanne]

I agree that not hardcoding the registry everywhere seems desirable

I'm actually not sure of this being a good idea. The registry identifies which image one intends to use, and leaving it undefined opens up to various forms of supply chain attacks.

Podman was insisting in doing the right thing by refusing images whose source is unspecified; this is clearly the better behaviour, unfortunately since it breaks stuff intended for Docker in which this level of precision isn't necessary this created friction in migration and the podman team ultimately gave up and changed the default behaviour on Fedora.

But the option for the more secure behaviour is available still and I'd recommend to use it, and so do security guides on production systems. We should still support it.

So I'd prefer it we could keep the prefixes. Allowing an override would be welcome of course.

Regarding rate control limits, shouldn't people use proxies like they are intended to? Proxys can be quite flexible AND secured.

• https://access.redhat.com/documentation/en-us/red_hat_quay/3/html/use_red_hat_quay/quay-as-cache-proxy