Support pulling zstd-compressed base layers

[rquinio1A]

Environment:

• Jib version: 0.20.0
• Build tool: Maven, via quarkus-container-jib 2.7.6 extension
• OS: Windows

Description of the issue:

Jib should support pulling image layers that use the new application/vnd.oci.image.layer.v1.tar+zstd (cf https://github.com/opencontainers/image-spec/blob/main/media-types.md).
Currelty only application/vnd.oci.image.layer.v1.tar+gzip is supported

When trying to build from an image like https://hub.docker.com/r/gscrivano/zstd-chunked, Jib will fail with:

[error]: Build step io.quarkus.container.image.jib.deployment.JibProcessor#buildFromJar threw an exception: java.lang.RuntimeException: Unable to create container image
        at io.quarkus.container.image.jib.deployment.JibProcessor.containerize(JibProcessor.java:235)
        at io.quarkus.container.image.jib.deployment.JibProcessor.buildFromJar(JibProcessor.java:164)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:882)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
        at java.base/java.lang.Thread.run(Thread.java:833)
        at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: java.util.concurrent.ExecutionException: java.util.zip.ZipException: Not in GZIP format
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88)
        at com.google.cloud.tools.jib.builder.steps.StepsRunner.realizeFutures(StepsRunner.java:112)
        at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$buildImage$9(StepsRunner.java:449)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException
         at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:880)
         at com.google.common.io.ByteStreams.copy(ByteStreams.java:104)
         at com.google.cloud.tools.jib.hash.Digests.lambda$computeDigest$2(Digests.java:85)
         at com.google.cloud.tools.jib.hash.Digests.computeDigest(Digests.java:102)
         at com.google.cloud.tools.jib.hash.Digests.computeDigest(Digests.java:86)
         at com.google.cloud.tools.jib.registry.AbstractManifestPuller.handleResponse(AbstractManifestPuller.java:96)
         at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:140)
         at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:114)
         at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:625)
         at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest(RegistryClient.java:436)
         at com.google.cloud.tools.jib.registry.RegistryClient.pullManifest(RegistryClient.java:441)
         at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.pullBaseImages(PullBaseImageStep.java:290)
         at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:179)
         at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call(PullBaseImageStep.java:69)

Additional information:

More info on zstd-chunked compression: https://www.redhat.com/sysadmin/faster-container-image-pulls
PR for zstd support in docker: moby/moby#41759. Note: the support is only in Beta version, with 20.10.16 docker pull gscrivano/zstd-chunked:fedora fails with Error processing tar file(exit status 1): archive/tar: invalid tar header
The ability to create zstd-compressed layers with Jib itself is likely a separate feature, in a first step Jib should not crash when pulling such layers.

[elefeint]

Thank you; we'll keep the feature request open to gauge community interest in the new format.

[rquinio]

The stacktrace is a bit clearer with jib-core 0.21.0:

Stacktrace

[error]: Build step io.quarkus.container.image.jib.deployment.JibProcessor#buildFromJar threw an exception: java.lang.RuntimeException: Unable to create container image
        at io.quarkus.container.image.jib.deployment.JibProcessor.containerize(JibProcessor.java:256)
        at io.quarkus.container.image.jib.deployment.JibProcessor.buildFromJar(JibProcessor.java:182)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:977)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:281)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: java.util.concurrent.ExecutionException: java.util.zip.ZipException: Not in GZIP format
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567)
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
        at com.google.cloud.tools.jib.builder.steps.StepsRunner.realizeFutures(StepsRunner.java:112)
        at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$buildImage$9(StepsRunner.java:449)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.zip.ZipException: Not in GZIP format
        at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:166)
        at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:80)
        at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:92)
        at com.google.cloud.tools.jib.cache.CacheStorageWriter.getDiffIdByDecompressingFile(CacheStorageWriter.java:163)
        at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeCompressedLayerBlobToDirectory(CacheStorageWriter.java:396)
        at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeCompressed(CacheStorageWriter.java:226)
        at com.google.cloud.tools.jib.cache.Cache.writeCompressedLayer(Cache.java:130)
        at com.google.cloud.tools.jib.builder.steps.ObtainBaseImageLayerStep.call(ObtainBaseImageLayerStep.java:141)
        at com.google.cloud.tools.jib.builder.steps.ObtainBaseImageLayerStep.call(ObtainBaseImageLayerStep.java:39)

Looking at the moby implementation, they look at the magic bytes of the input stream to detect the type of compression (Bzip2, Gzip, Xz, Zstd).
I see jib-core already depends on org.apache.commons:commons-compress which has a utility class to do that. However
zstd compressor comes via an optional third-party dependency https://github.com/luben/zstd-jni

I'm going to open a PR so you can review the feasibility / design / impacts.

[mpeddada1]

Thank you for your contribution! jib-gradle-plugin3.3.0, jib-maven-plugin 3.3.0 and jib-core 0.22.0 have been released with this feature!