[Buildx] Che-dashboard && che-workspace-loader build is failing in some cases

[skabashnyuk]

Describe the bug

For unknown to me reasons Buildx build is failing some times.

several times like this
https://github.com/eclipse/che-dashboard/runs/695674547?check_suite_focus=true

#7 ERROR: failed to copy: httpReaderSeeker: failed open: server message: invalid_token: authorization failed
------
 > [stage-1 2/4] RUN sed -i 's|    AllowOverride None|    AllowOverride All|' /usr/local/apache2/conf/httpd.conf &&     sed -i 's|Listen 80|Listen 8080|' /usr/local/apache2/conf/httpd.conf &&     mkdir -p /var/www && ln -s /usr/local/apache2/htdocs /var/www/html &&     chmod -R g+rwX /usr/local/apache2 &&     echo "ServerName localhost" >> /usr/local/apache2/conf/httpd.conf:
------
failed to solve: rpc error: code = Unknown desc = failed to load cache: failed to copy: httpReaderSeeker: failed open: server message: invalid_token: authorization failed
##[error]Process completed with exit code 1.

several times like this
https://github.com/eclipse/che-workspace-loader/runs/695731886?check_suite_focus=true
#29 exporting manifest

sha256:9eb16121ec8338a95f42e79bfe80630ecf3a36f1b981238145a470f77f48caa8 done
#29 exporting config sha256:c9a275485c8d93b80104c669d2425a640bdd965159f03f9312984419ffb39b58 done
#29 exporting manifest sha256:cf42d3821b55aa33b3c6ee2f9d0e867ac65aacacca5e2027bff8f6bc9e820855 done
#29 exporting config sha256:e55e574af113d99980c90b39e5ab05f0c54b2fe968539ab956e978847c66e8ae
#29 exporting config sha256:e55e574af113d99980c90b39e5ab05f0c54b2fe968539ab956e978847c66e8ae done
#29 exporting manifest list sha256:81fd17e4f315439dd71107ae21eb2127c63465d48db5f3d56649d0eba509b372 done
#29 pushing layers
#29 pushing layers 0.1s done
#29 pushing manifest for quay.io/eclipse/che-workspace-loader:next
#29 pushing manifest for quay.io/eclipse/che-workspace-loader:next 0.2s done
#29 ERROR: failed commit on ref "manifest-sha256:cf42d3821b55aa33b3c6ee2f9d0e867ac65aacacca5e2027bff8f6bc9e820855": unexpected status: 401 UNAUTHORIZED
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:cf42d3821b55aa33b3c6ee2f9d0e867ac65aacacca5e2027bff8f6bc9e820855": unexpected status: 401 UNAUTHORIZED
##[error]Process completed with exit code 1.

[skabashnyuk]

At this moment I think it might be related to:

• the way how we do cache layers. We use docker hub to push cache and quay.io to push real images
• quay.io/docker hub might not support the necessary API used by buildx
• multi-platform build

[skabashnyuk]

CC @shahidhs-ibm @benoitf @nickboldt

[skabashnyuk]

at this moment I want to make several experiments with/without cache and using the matrix for multi-platform build

[skabashnyuk]

build with matrix

      matrix:
        platform: [linux/amd64, linux/s390x]
        layerscaching: [true, false]

fails only linux/s390x+ without layerscaching

[skabashnyuk]

che-workspce-loader

#16 [linux/s390x builder 7/7] RUN yarn build && yarn test
#16 12.51 Error: Unexpected token ] in JSON at position 356 in /workspace-loader/tslint.json
#16 12.51     at readConfigurationFile (/workspace-loader/node_modules/tslint/lib/configuration.js:187:19)
#16 12.51     at Object.loadConfigurationFromPath (/workspace-loader/node_modules/tslint/lib/configuration.js:160:29)
#16 12.51     at /workspace-loader/node_modules/typescript-formatter/lib/provider/tslintjson.js:134:43
#16 12.51     at step (/workspace-loader/node_modules/typescript-formatter/lib/provider/tslintjson.js:32:23)
#16 12.51     at Object.next (/workspace-loader/node_modules/typescript-formatter/lib/provider/tslintjson.js:13:53)
#16 12.51     at fulfilled (/workspace-loader/node_modules/typescript-formatter/lib/provider/tslintjson.js:4:58)
#16 12.51     at <anonymous>
#16 12.65 error Command failed with exit code 1.
#16 12.66 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
#16 12.82 error Command failed with exit code 1.
#16 12.82 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
#16 ERROR: executor failed running [/bin/sh -c yarn build && yarn test]: buildkit-runc did not terminate successfully

[ericwill]

I was running into this for sidecar container builds as well, but ended up just disabling caching for now. Not sure if it helps but @benoitf managed to get it working here.

[skabashnyuk]

che loader

#29 pushing layers
#29 pushing layers 0.1s done
#29 pushing manifest for quay.io/eclipse/che-workspace-loader:next
#29 pushing manifest for quay.io/eclipse/che-workspace-loader:next 0.2s done
#29 ERROR: failed commit on ref "manifest-sha256:c6e2b61b24d6e9cce631a20165a8bc391330bbc6db4173f67e723ffc8aa86d0b": unexpected status: 401 UNAUTHORIZED
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:c6e2b61b24d6e9cce631a20165a8bc391330bbc6db4173f67e723ffc8aa86d0b": unexpected status: 401 UNAUTHORIZED
child_process.js:669
    throw err;
    ^

Error: Command failed: docker buildx build --platform linux/amd64,linux/s390x --cache-from="type=registry,ref=${CACHE_IMAGE_FULL}" --cache-to="type=registry,ref=${CACHE_IMAGE_FULL},mode=max" -t ${IMAGE_FULL}  -f apache.Dockerfile --push .

https://github.com/eclipse/che-workspace-loader/runs/698909038

[shahidhs-ibm]

@skabashnyuk We have checked the github action logs of your work and the phantomjs failures in s390x will need a fix. We will work on this and will keep you posted. Most likely will need a new PR.

[skabashnyuk]

@shahidhs-ibm thx.

[skabashnyuk]

workspace loader new failure

9 pushing manifest for quay.io/eclipse/che-workspace-loader:next
#29 pushing manifest for quay.io/eclipse/che-workspace-loader:next 0.2s done
#29 ERROR: failed commit on ref "manifest-sha256:c6e2b61b24d6e9cce631a20165a8bc391330bbc6db4173f67e723ffc8aa86d0b": unexpected status: 401 UNAUTHORIZED
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:c6e2b61b24d6e9cce631a20165a8bc391330bbc6db4173f67e723ffc8aa86d0b": unexpected status: 401 UNAUTHORIZED
child_process.js:669
    throw err;
    ^

Error: Command failed: docker buildx build --platform linux/amd64,linux/s390x --cache-from="type=registry,ref=${CACHE_IMAGE_FULL}" --cache-to="type=registry,ref=${CACHE_IMAGE_FULL},mode=max" -t ${IMAGE_FULL}  -f apache.Dockerfile --push .
    at checkExecSyncError (child_process.js:630:11)
    at execSync (child_process.js:666:15)
    at run (/home/runner/work/_actions/nick-invision/retry/v1/dist/exec.js:5:3)
    at Object.<anonymous> (/home/runner/work/_actions/nick-invision/retry/v1/dist/exec.js:8:1)
    at Module._compile (internal/modules/cjs/loader.js:1133:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1153:10)
    at Module.load (internal/modules/cjs/loader.js:977:32)

[sleshchenko]

that's a quite general. Currently, we have a pretty stable build of workspace and che-dashboard.
If we'll face any failure - we'll register a new issue.