Skip to content

Commit

Permalink
[SPARK-42462][K8S] Prevent docker-image-tool.sh from publishing OCI…
Browse files Browse the repository at this point in the history
… manifests

### What changes were proposed in this pull request?

This is found during Apache Spark 3.3.2 docker image publishing. It's not an Apache Spark but important for `docker-image-tool.sh` to provide backward compatibility during cross-building. This PR targets for all **future releases**, Apache Spark 3.4.0/3.3.3/3.2.4.

### Why are the changes needed?

Docker `buildx` v0.10.0 publishes OCI Manifests by default which is not supported by `docker manifest` command like the following.
docker/buildx#1509
```
$ docker manifest inspect apache/spark:v3.3.2
no such manifest: docker.io/apache/spark:v3.3.2
```

Note that the published images are working on both AMD64/ARM64 machines, but `docker manifest` cannot be used. For example, we cannot create `latest` tag.

### Does this PR introduce _any_ user-facing change?

This will fix the regression of Docker `buildx`.

### How was this patch tested?

Manually builds the multi-arch image and check `manifest`.

```
$ docker manifest inspect apache/spark:v3.3.2
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:30ae5023fc384ae3b68d2fb83adde44b1ece05f926cfceecac44204cdc9e79cb",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:aac13b5b5a681aefa91036d2acae91d30a743c2e78087c6df79af4de46a16e1b",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}
```

Closes apache#40051 from dongjoon-hyun/SPARK-42462.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Feb 16, 2023
1 parent 7ee8a32 commit 2ac70ae
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions bin/docker-image-tool.sh
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ function build {
error "Failed to build Spark JVM Docker image, please refer to Docker build output for details."
fi
if [ "${CROSS_BUILD}" != "false" ]; then
(cd $(img_ctx_dir base) && docker buildx build $ARCHS $NOCACHEARG "${BUILD_ARGS[@]}" --push \
(cd $(img_ctx_dir base) && docker buildx build $ARCHS $NOCACHEARG "${BUILD_ARGS[@]}" --push --provenance=false \
-t $(image_ref spark) \
-f "$BASEDOCKERFILE" .)
fi
Expand All @@ -194,7 +194,7 @@ function build {
error "Failed to build PySpark Docker image, please refer to Docker build output for details."
fi
if [ "${CROSS_BUILD}" != "false" ]; then
(cd $(img_ctx_dir pyspark) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push \
(cd $(img_ctx_dir pyspark) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push --provenance=false \
-t $(image_ref spark-py) \
-f "$PYDOCKERFILE" .)
fi
Expand All @@ -208,7 +208,7 @@ function build {
error "Failed to build SparkR Docker image, please refer to Docker build output for details."
fi
if [ "${CROSS_BUILD}" != "false" ]; then
(cd $(img_ctx_dir sparkr) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push \
(cd $(img_ctx_dir sparkr) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push --provenance=false \
-t $(image_ref spark-r) \
-f "$RDOCKERFILE" .)
fi
Expand Down

0 comments on commit 2ac70ae

Please sign in to comment.