[k8s plugin] buildah push command doesn't work #15035

azatsarynnyy · 2019-10-30T14:47:39Z

Describe the bug

Kubernetes Che plugin provides Kubernetes: Run command that needs to call buildah push command to push the built image to a remote registry. It reports an error in Theia Output panel:

[Buildah 2019-10-30 14:17:51.459]
Getting image source signatures
Copying blob sha256:687890749166493fb9958058143592c3aadf73b6f914f1b15093e8108d47dcd8
Copying blob sha256:799e7111d6d49ecc3b71cb8ecd0211b3ca99d7b618f6ad039a7b0181af8ad9cc
Copying blob sha256:2f77733e9824e18b358f20bd33ad569f21e3136c2a22ef502c15b627be3c9f5e
Copying blob sha256:97041f29baff775daa9b4f608c11db724fbe406c6478f69721a85df4006ef7c0
Copying blob sha256:b8f8aeff56a83bdb007111d2a746c5819d592c6c0a0ddbd4d2e60f5ed5d71dea
Copying blob sha256:a72a7e555fe1a736d152fd9170494c0a1ca73003a61297f0d39ddb547d444ec0
Copying blob sha256:2da66389fd947d5840d4d2214e5f9fd5266490eb82a3748891e8d213dde00d1a
Copying blob sha256:2f6a6589e295ae19c4a165c504a4e021ebba2116f68a4520d8b7ae69b87961e2
Copying blob sha256:4c5e2c02e279bf20e4e5e97bb13eb6f5cbd4531f2e024f5dadfd6acc90d952cb
Copying blob sha256:aa8fc267d1f60f520efe120bdeb871382d0d6460728003fd485b8da13c703251
error copying layers and metadata from "containers-storage:[overlay@/home/theia/.local/share/containers/storage+/var/tmp/theia:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.ignore_chown_errors=true]localhost/azatsarynnyy/simple-nodejs-weather-app:error" to "docker://azatsarynnyy/simple-nodejs-weather-app:error": Error writing blob: Error initiating layer upload to /v2/azatsarynnyy/simple-nodejs-weather-app/blobs/uploads/ in registry-1.docker.io: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
level=error msg="exit status 1"

Che version

latest
nightly
other: please specify

Steps to reproduce

Start a Workspace with Kubernetes plugin included.
Clone some project with Dockerfile, e.g. https://github.com/azatsarynnyy/simple-nodejs-weather-app
Open project folder as a Theia workspace, File > Open Workspace...
Login to the image registry. buildah login docker.io in the plugin sidecar.
Run Kubernetes: Run command.

Expected behavior

The built image should be pushed to the remote registry.

Runtime

kubernetes (include output of kubectl version)
Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minishift (include output of minishift version and oc version)
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

Screenshots

Installation method

chectl
che-operator
minishift-addon
I don't know

Environment

Additional context

The text was updated successfully, but these errors were encountered:

azatsarynnyy · 2019-11-19T09:00:36Z

> buildah bud with ignore_chown_errors=false in storage.conf

`

"docker.io/library/node:8": Error committing the finished image: error adding layer with blob "sha256:9a0b0ce99936ce4861d44ce1f193e881e5b40b5bf1847627061205b092fa7f1d": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
`

> buildah bud with ignore_chown_errors=true in storage.conf

The build is finished successfully.

> buildah push

Getting image source signatures error copying layers and metadata from "containers-storage:[overlay@/home/theia/.local/share/containers/storage+/var/tmp/theia:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.ignore_chown_errors=true]docker.io/azatsarynnyy/test:che" to "docker://azatsarynnyy/test:che": Error trying to reuse blob sha256:97041f29baff775daa9b4f608c11db724fbe406c6478f69721a85df4006ef7c0 at destination: Error checking whether a blob sha256:97041f29baff775daa9b4f608c11db724fbe406c6478f69721a85df4006ef7c0 exists in docker.io/azatsarynnyy/test: errors: denied: requested access to the resource is denied error parsing HTTP 401 response body: unexpected end of JSON input: "" ERRO exit status 1

> buildah login docker.io doesn't help

> buildah push --creds

Error: Error copying image to the remote destination: Error writing blob: Patch https://registry-1.docker.io/v2/azatsarynnyy/test/blobs/uploads/c62ae769-ebe1-4108-aed6-cf87143d6e9c?_state=lD2Y1PFrcgp4Odp93l-Y4Dib2b94Vwv3Uu0Vy5Vf2PR7Ik5hbWUiOiJhemF0c2FyeW5ueXkvdGVzdCIsIlVVSUQiOiJjNjJhZTc2OS1lYmUxLTQxMDgtYWVkNi1jZjg3MTQzZDZlOWMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMTEtMThUMTU6NTg6MTguMDg0MTY3NzQxWiJ9: open /home/user/.local/share/containers/storage/overlay/97041f29baff775daa9b4f608c11db724fbe406c6478f69721a85df4006ef7c0/merged/bin/bash: no such file or directory

ericwill · 2020-01-17T14:53:23Z

Just to clarify my understanding: this building + pushing is happening inside the Kubernetes plugin's sidecar container, correct?

azatsarynnyy · 2020-01-17T15:30:32Z

@ericwill correct

ericwill · 2020-01-20T16:54:35Z

I can't get the image to even build. When following the steps to reproduce I get this error after running "Kubernetes Run":

Copying blob sha256:9935d0c62ace92b388be202275e222007d6cac10b9c1f2c1ea63af38c09ea7ab Copying blob sha256:645c20ec82147f2bdc182280ac02b60687cfac33c4231059051bbdcf60e31035 Copying blob sha256:e705a4c4fd310b96bfb3d7928428e65f0d3f5bad0cd0bda1434aee1d89418468 Copying blob sha256:c877b722db6f4d4297c21f07162d21fa6ac6f51d03360f75e24cfa3bcaefe57d Copying blob sha256:146bd6a886182fde06fbf747470b1c89814bc8ab1c96fdf1aef6107171959fe6 Copying blob sha256:db0efb86e80601b5bbdbb7c406426982c4202d339687c14c3941b364527e2249 Copying blob sha256:db8fbd9db2fe6382804432552a60f85b0e292cdb34b8a5c751c4a3061f68df29 Copying blob sha256:1c151cd1b3eabced84742529489b3c8c344181b4cbcdcfccdb1f4a7c7d0239ec Copying blob sha256:fbd993995f407f3681fc8399ae67c04bba8e15c7f754b1af8b37f50b35f4a11e Copying config sha256:8eeadf3757f41ffbde6fbb4bbbe14f9f5b9ce33cc1aa88bb5c0b4d6e9467509d Writing manifest to image destination Storing signatures error clearing supplemental groups list: operation not permittedsubprocess exited with status 1 error building at STEP "RUN npm install": exit status 1 level=error msg="exit status 1"

If I try to build an image inside the sidecar container (outside of Che, just on my machine), I get the following error:

STEP 1: FROM node:8 Getting image source signatures Copying blob 645c20ec8214 done Copying blob db0efb86e806 done Copying blob 146bd6a88618 done Copying blob c877b722db6f done Copying blob e705a4c4fd31 done Copying blob 9935d0c62ace done Copying blob 1c151cd1b3ea done Copying blob fbd993995f40 done Copying blob db8fbd9db2fe done Copying config 8eeadf3757 done Writing manifest to image destination Storing signatures ERRO error unmounting /var/lib/containers/storage/overlay/6e43ae2afd86c8e714f702272d869d2bfe76c9dc6a8904d6a6258c8cd4f4ef1f/merged: invalid argument error mounting new container: error mounting build container "03e2fac45ef08b9d73c9dadf9daf8fdbe2dd4dd21692b82d7fc5f548313b0e67": error creating overlay mount to /var/lib/containers/storage/overlay/6e43ae2afd86c8e714f702272d869d2bfe76c9dc6a8904d6a6258c8cd4f4ef1f/merged: using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first fuse-overlayfs: cannot mount: No such file or directory : exit status 1 ERRO exit status 1

I was able to workaround the building issue locally by running the sidecar container with some additional options:
podman run -it --privileged --cap-add=SYS_ADMIN --device /dev/fuse
but I'm not sure this is the correct way.

azatsarynnyy · 2020-01-20T21:01:56Z

I don't think we can use podman in privileged mode in a sidecar.

All I can advise is trying it with the latest buildah. A lot of improvements/fixes were made there recently. Or maybe something is missing in storage.conf.

vitaliy-guliy · 2020-02-19T10:18:53Z

@azatsarynnyy could you attach the devfile which you were using when you faced the problem?

azatsarynnyy · 2020-02-19T10:37:38Z

@vitaliy-guliy I had no particular devfile as I remember. Just added a k8s plugin from the Dashboard

vitaliy-guliy · 2020-02-19T12:52:33Z

Kubernetes plugin does not work on latest Che-Theia running on local minikube.
The devfile used to create a workspsce

metadata:
  name: theia-plugin
components:
  - id: eclipse/che-theia/latest
    memoryLimit: 1500M
    type: cheEditor
    alias: theia-editor
  - id: che-incubator/typescript/latest
    memoryLimit: 2048M
    type: chePlugin
  - id: redhat/vscode-yaml/latest
    type: chePlugin
  - id: eclipse/che-machine-exec-plugin/latest
    type: chePlugin
  - mountSources: true
    endpoints:
      - name: theia-dev-flow
        port: 3010
        attributes:
          protocol: http
          public: 'true'
    memoryLimit: 3Gi
    type: dockerimage
    image: 'quay.io/eclipse/che-theia-dev:latest'
    alias: che-dev
  - id: ms-kubernetes-tools/vscode-kubernetes-tools/latest
    type: chePlugin
apiVersion: 1.0.0
commands:
  - name: 1. List projects
    actions:
      - workdir: /projects
        type: exec
        command: |
          ls -la; \
          echo -e "\e[32mDone\e[0m"
        component: che-dev

azatsarynnyy · 2020-02-19T21:57:20Z

@vitaliy-guliy I've updated Steps to reproduce with step 4 to set the image build tool from Docker to Buildah.

I tried with the following Devfile

metadata:
  name: wksp-custom-j4xr
projects:
  - name: simple-nodejs-weather-app
    source:
      location: 'https://github.com/azatsarynnyy/simple-nodejs-weather-app'
      type: git
      branch: master
components:
  - id: ms-kubernetes-tools/vscode-kubernetes-tools/latest
    type: chePlugin
apiVersion: 1.0.0

and got

[Buildah 2020-02-19 21:50:56.754]
Getting image source signatures
Copying blob sha256:e705a4c4fd310b96bfb3d7928428e65f0d3f5bad0cd0bda1434aee1d89418468
Copying blob sha256:c877b722db6f4d4297c21f07162d21fa6ac6f51d03360f75e24cfa3bcaefe57d
Copying blob sha256:db0efb86e80601b5bbdbb7c406426982c4202d339687c14c3941b364527e2249
Copying blob sha256:146bd6a886182fde06fbf747470b1c89814bc8ab1c96fdf1aef6107171959fe6
Copying blob sha256:9935d0c62ace92b388be202275e222007d6cac10b9c1f2c1ea63af38c09ea7ab
Copying blob sha256:645c20ec82147f2bdc182280ac02b60687cfac33c4231059051bbdcf60e31035
Copying blob sha256:db8fbd9db2fe6382804432552a60f85b0e292cdb34b8a5c751c4a3061f68df29
Copying blob sha256:1c151cd1b3eabced84742529489b3c8c344181b4cbcdcfccdb1f4a7c7d0239ec
Copying blob sha256:fbd993995f407f3681fc8399ae67c04bba8e15c7f754b1af8b37f50b35f4a11e
Copying config sha256:8eeadf3757f41ffbde6fbb4bbbe14f9f5b9ce33cc1aa88bb5c0b4d6e9467509d
Writing manifest to image destination
Storing signatures
error clearing supplemental groups list: operation not permittedsubprocess exited with status 1
error building at STEP "RUN npm install": exit status 1
level=error msg="exit status 1"

* Update configuration for the buildah * Remove comments * Update version of Helm and Kubectl * Add XDG_RUNTIME_DIR environment variable * Simplify storage configuration Fixes eclipse-che/che#15035 Signed-off-by: Vitaliy Gulyy <[email protected]>

azatsarynnyy added kind/bug Outline of a bug - must adhere to the bug report template. team/ide labels Oct 30, 2019

vparfonov added this to the Backlog - IDE 1 milestone Oct 30, 2019

benoitf added the severity/P2 Has a minor but important impact to the usage or development of the system. label Oct 30, 2019

vparfonov mentioned this issue Nov 4, 2019

IDE-2019-11-26 (Sprint: 175) #15044

Closed

19 tasks

vparfonov modified the milestones: Backlog - IDE 1, 7.5.0 Nov 11, 2019

vparfonov mentioned this issue Nov 21, 2019

IDE-2019-12-17 (Sprint: 176) #15278

Closed

14 tasks

vparfonov modified the milestones: 7.5.0, Backlog - IDE 1, 7.6.0 Nov 28, 2019

ericwill mentioned this issue Dec 17, 2019

Che Plugins Sprint 177 #15513

Closed

15 tasks

ericwill mentioned this issue Jan 8, 2020

Che Plugins Sprint 178 #15633

Closed

16 tasks

ericwill mentioned this issue Jan 28, 2020

Che Plugins Sprint 179 #15862

Closed

12 tasks

ericwill mentioned this issue Feb 19, 2020

Che Plugins Sprint 180 #16083

Closed

21 tasks

azatsarynnyy mentioned this issue Feb 21, 2020

Editors Sprint 180 #16086

Closed

16 tasks

ericwill mentioned this issue Feb 24, 2020

How can i deploy docker image (Python) in Eclipse Che Workspace #16111

Closed

ericwill modified the milestones: 7.6.0, 7.10.0 Mar 3, 2020

vitaliy-guliy mentioned this issue Mar 3, 2020

Update configuration for the buildah che-dockerfiles/che-sidecar-kubernetes-tooling#8

Merged

This was referenced Mar 10, 2020

Update image for kubernetes plugin eclipse-che/che-plugin-registry#422

Merged

Update buildah configuration che-dockerfiles/che-sidecar-kubernetes-tooling#9

Merged

Add new version of Kubernetes plugin eclipse-che/che-plugin-registry#423

Merged

vitaliy-guliy closed this as completed Mar 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[k8s plugin] buildah push command doesn't work #15035

[k8s plugin] buildah push command doesn't work #15035

azatsarynnyy commented Oct 30, 2019

azatsarynnyy commented Nov 19, 2019

ericwill commented Jan 17, 2020

azatsarynnyy commented Jan 17, 2020

ericwill commented Jan 20, 2020

azatsarynnyy commented Jan 20, 2020

vitaliy-guliy commented Feb 19, 2020

azatsarynnyy commented Feb 19, 2020

vitaliy-guliy commented Feb 19, 2020

azatsarynnyy commented Feb 19, 2020

[k8s plugin] buildah push command doesn't work #15035

[k8s plugin] buildah push command doesn't work #15035

Comments

azatsarynnyy commented Oct 30, 2019

Describe the bug

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

Additional context

azatsarynnyy commented Nov 19, 2019

ericwill commented Jan 17, 2020

azatsarynnyy commented Jan 17, 2020

ericwill commented Jan 20, 2020

azatsarynnyy commented Jan 20, 2020

vitaliy-guliy commented Feb 19, 2020

azatsarynnyy commented Feb 19, 2020

vitaliy-guliy commented Feb 19, 2020

azatsarynnyy commented Feb 19, 2020