Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading repositories with submodules is repeated. Failed to clone submodule from googlesource #300

Closed
MarshalX opened this issue Feb 17, 2021 · 8 comments · Fixed by moby/buildkit#1987 or moby/moby#42056
Labels
kind/upstream Changes need to be made on upstream project

Comments

@MarshalX
Copy link

My workflow:

jobs:
  build_push_to_registry:
    name: Push Docker image to GitHub Packages
    runs-on: ubuntu-latest
    steps:
      - name: Check out the repo
        uses: actions/checkout@v2
        with:
          submodules: recursive
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.CR_PAT }}
      - name: Push to GitHub Packages
        uses: docker/build-push-action@v2
        with:
          file: build/manylinux/Dockerfile
          push: true
          tags: tgcalls/manylinux:latest

The second way:

- name: Check out the repo
    uses: actions/checkout@v2
- name: Checkout submodules
    run: git submodule update --init --recursive

My repository is private. It has several submodules. They are all public. Some on github, others on google. The checkout proceeds normally in two ways (see above). When trying to use build-push-action (with and without Buildx step), it cannot clone google submodules. Probably something is being mixed into the request. For example, some kind of token that Google cannot validate. Requests do not go through. Do not clone sambmodules...

Logs:

#1 [internal] load git source https://github.com/MarshalX/tgcalls.git#heads/master
#1 sha256:7f00bd87a6bdc53569455e81eceb45e3c91d9a6363df8e8c6c31f4e3a2fa9a08
#1 0.014 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
#1 0.236 6eb146a71f45e268d96b6631e31482ddd610eb78	refs/heads/master
#1 1.612 From https://github.com/MarshalX/tgcalls
#1 1.612  * [new branch]      master     -> heads/master
#1 1.612  * [new branch]      master     -> origin/master
#1 2.289 Submodule 'cmake' (https://github.com/desktop-app/cmake_helpers) registered for path 'cmake'
#1 2.290 Submodule 'tgcalls/third_party/pybind11' (https://github.com/pybind/pybind11) registered for path 'tgcalls/third_party/pybind11'
#1 2.290 Submodule 'tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx' (https://chromium.googlesource.com/webm/libvpx) registered for path 'tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx'
#1 2.291 Submodule 'tgcalls/third_party/webrtc/src/third_party/libyuv' (https://chromium.googlesource.com/libyuv/libyuv) registered for path 'tgcalls/third_party/webrtc/src/third_party/libyuv'
#1 2.297 Cloning into '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/cmake'...
#1 2.552 Cloning into '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/tgcalls/third_party/pybind11'...
#1 2.953 Cloning into '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx'...
#1 3.043 remote: INVALID_ARGUMENT: Request contains an invalid argument
#1 3.043 remote: [type.googleapis.com/google.rpc.LocalizedMessage]
#1 3.043 remote: locale: "en-US"
#1 3.043 remote: message: "Invalid authentication credentials. Please generate a new identifier: https://chromium.googlesource.com/new-password"
#1 3.043 remote: 
#1 3.043 remote: [type.googleapis.com/google.rpc.RequestInfo]
#1 3.043 remote: request_id: "1953dfe24529482399a08c19095952ca"
#1 3.043 fatal: unable to access 'https://chromium.googlesource.com/webm/libvpx/': The requested URL returned error: 400
#1 3.045 fatal: clone of 'https://chromium.googlesource.com/webm/libvpx' into submodule path '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx' failed
#1 3.046 Failed to clone 'tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx'. Retry scheduled
#1 3.049 Cloning into '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/tgcalls/third_party/webrtc/src/third_party/libyuv'...
#1 3.172 remote: INVALID_ARGUMENT: Request contains an invalid argument
#1 3.172 remote: [type.googleapis.com/google.rpc.LocalizedMessage]
#1 3.172 remote: locale: "en-US"
#1 3.172 remote: message: "Invalid authentication credentials. Please generate a new identifier: https://chromium.googlesource.com/new-password"
#1 3.172 remote: 
#1 3.172 remote: [type.googleapis.com/google.rpc.RequestInfo]
#1 3.172 remote: request_id: "0eb89bc4773e4066960db6e5454bfe3b"
#1 3.172 fatal: unable to access 'https://chromium.googlesource.com/libyuv/libyuv/': The requested URL returned error: 400
#1 3.174 fatal: clone of 'https://chromium.googlesource.com/libyuv/libyuv' into submodule path '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/tgcalls/third_party/webrtc/src/third_party/libyuv' failed
#1 3.174 Failed to clone 'tgcalls/third_party/webrtc/src/third_party/libyuv'. Retry scheduled
#1 3.177 Cloning into '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx'...
#1 3.275 remote: INVALID_ARGUMENT: Request contains an invalid argument
#1 3.275 remote: [type.googleapis.com/google.rpc.LocalizedMessage]
#1 3.275 remote: locale: "en-US"
#1 3.275 remote: message: "Invalid authentication credentials. Please generate a new identifier: https://chromium.googlesource.com/new-password"
#1 3.275 remote: 
#1 3.275 remote: [type.googleapis.com/google.rpc.RequestInfo]
#1 3.275 remote: request_id: "43c0584098af4e24a79f0a120968ed82"
#1 3.275 fatal: unable to access 'https://chromium.googlesource.com/webm/libvpx/': The requested URL returned error: 400
#1 3.277 fatal: clone of 'https://chromium.googlesource.com/webm/libvpx' into submodule path '/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/fs/tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx' failed
#1 3.277 Failed to clone 'tgcalls/third_party/webrtc/src/third_party/libvpx/source/libvpx' a second time, aborting
#1 ERROR: failed to update submodules for https://github.com/MarshalX/tgcalls.git: exit status 1
------
 > [internal] load git source https://github.com/MarshalX/tgcalls.git#heads/master:
------
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to update submodules for https://github.com/MarshalX/tgcalls.git: exit status 1

As you can see the repository https://chromium.googlesource.com/webm/libvpx/ and https://chromium.googlesource.com/libyuv/libyuv/ is public.

  1. How can I disable sending unnecessary headers when cloning submodules?
  2. How can I disable cloning when I have already cloned the repository and all submodules?

Thanks!

@crazy-max
Copy link
Member

@MarshalX Yes maybe authentication header is sent to submodules as well, that's why you have this kind of issue. Looks like a bug on buildkit. WDYT @tonistiigi?

In the meantime you can use the Path context:

jobs:
  build_push_to_registry:
    name: Push Docker image to GitHub Packages
    runs-on: ubuntu-latest
    steps:
      - name: Check out the repo
        uses: actions/checkout@v2
        with:
          submodules: recursive
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.CR_PAT }}
      - name: Push to GitHub Packages
        uses: docker/build-push-action@v2
        with:
          context: .
          file: build/manylinux/Dockerfile
          push: true
          tags: tgcalls/manylinux:latest

@crazy-max crazy-max added the kind/upstream Changes need to be made on upstream project label Feb 17, 2021
MarshalX added a commit to MarshalX/tgcalls that referenced this issue Feb 17, 2021
@MarshalX
Copy link
Author

@crazy-max Thank you very much for such a quick response! Thanks for solving my problem! I am sorry that I slightly missed the repository for creating Issue. Got it after seeing the command that runs build-push-action

@crazy-max
Copy link
Member

crazy-max commented Feb 19, 2021

@MarshalX Can you make a test with:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          driver-opts: image=crazymax/buildkit:git-token-scope

@tonistiigi
Copy link
Member

https://github.com/moby/buildkit/releases/tag/v0.8.2n is out and this should be fixed

deadprogram added a commit to tinygo-org/tinygo that referenced this issue Jun 2, 2021
clly added a commit to clly/minic that referenced this issue Mar 4, 2022
Use context option as described here: docker/build-push-action#300
@roymiloh
Copy link

roymiloh commented Oct 5, 2022

@crazy-max could you please explain why having context: . solves the issue? it actually does (I use docker/build-push-action@v3), but I'd be glad to understand why. Does having a context imply that no submodules should be cloned?

@crazy-max
Copy link
Member

@roymiloh

could you please explain why having context: . solves the issue?

If context input is not specified it will use the default Git context instead and clone the repository through BuildKit. But having context: . will use the current dir as context. You have to check out the repo first using actions/checkout and that's why it works as workaround because BuildKit will not clone the repository. This has been fixed since BuildKit 0.8.2 (see moby/buildkit#1987), so you should be able to use a Git context.

@roymiloh
Copy link

roymiloh commented Oct 5, 2022

I see, thanks. Well, I actually came across this issue because things didn't work:

#1 2.547 Submodule '***' (https://github.com/***.git) registered for path '***'
#1 2.551 Cloning into '/var/lib/docker/overlay2/***'...
#1 2.645 remote: Repository not found.
#1 2.645 fatal: repository 'https://github.com/***.git/' not found
#1 2.647 fatal: clone of 'https://github.com/***.git' into submodule path '/var/lib/docker/overlay2/***' failed
#1 2.647 Failed to clone '***'. Retry scheduled

... and now I understand why having a context is a workaround (I use actions/checkout along the workflow indeed). Anyway - it could be that there's another issue, as (IIUC) actions/checkout doesn't clone submodules by default, so maybe it doesn't work in general for some reason (because something is misconfigured on my end).

@crazy-max
Copy link
Member

@roymiloh It doesn't checkout submodules by default: https://github.com/actions/checkout/#usage

    # Whether to checkout submodules: `true` to checkout submodules or `recursive` to
    # recursively checkout submodules.
    #
    # When the `ssh-key` input is not provided, SSH URLs beginning with
    # `[email protected]:` are converted to HTTPS.
    #
    # Default: false
    submodules: ''

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/upstream Changes need to be made on upstream project
Projects
None yet
4 participants