Unexpected stop of container build [CML, AWS, github]

[sergeychuvakin]

My scenario:
I'm trying to create appropriate pipeline for my ML project. I'm using the following CML yaml file:

on:
  # Trigger the workflow on push or pull request
  push:
    branches:
      - mybranch

jobs:
  deploy-runner:
    runs-on: [ubuntu-latest]
    steps:
      - uses: iterative/setup-cml@v1
      - uses: actions/checkout@v2
      - name: Deploy runner on EC2
        env:
          PERSONAL_ACCESS_TOKEN: ${{ secrets.REPO_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-west-1
        run: |
          cml-runner \
              --repo https://github.com/MyCompany/myrepo \
              --token=$PERSONAL_ACCESS_TOKEN \
              --cloud aws \
              --cloud-region us-west-1 \
              --cloud-type=g3.4xlarge \
              --labels=cml-runner \
              --idle-timeout 30
    
  model-training:
    needs: [deploy-runner]
    runs-on: [self-hosted, cml-runner]
    container:
      image: docker://dvcorg/cml:0-dvc1-base1-gpu
      options: --gpus all
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: '3.6'
      - name: Train model
        env:
          repo_token: ${{ secrets.REPO_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.DVC_ACCESS_KEY }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.DVC_SECRET_KEY }}
          REQUIREMENTS_FILE: 'training/training_req.txt'
        run: |
          export AWS_DEFAULT_REGION=us-east-1
          echo "Install reqs"
          sudo apt update
          sudo apt-get install default-jre scala
          pip install py4j
          pip install --no-cache-dir -e .
          export PYSPARK_PYTHON=python3
          echo "Start CML"
          python3 -m spacy download en_core_web_sm
          echo "Pull data"
          dvc repro
          echo "## Model metrics" > report.md
          cat prepare_data/metrics.txt >> report.md
          cml-send-comment report.md
          

As you can notice I used the following image docker://dvcorg/cml:0-dvc1-base1-gpu, but I started receive the following error message:

I can see that container started to build but unexpectedly stopped, and I do not see the reason of this behavior. Actually i did not change anything in my script, and it just stopped work, but earlier I run it successfully.

Thanks!

[sergeychuvakin]

Full logs archive.
logs_261.zip

[DavidGOrtega]

@sergeychuvakin thanks for reporting. Im checking.

[DavidGOrtega]

From logs:

Re-evaluate condition on job cancellation for step: 'Initialize containers'.
2021-07-27T17:34:49.8283399Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

[DavidGOrtega]

With self hosted runners this is happening too. Log says:

2021-07-28 07:15:35Z: Job run completed with result: Failed

An error occurred: Access denied. System:ServiceIdentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD needs View permissions to perform the action.

An error occurred: Access denied. System:ServiceIdentity;

Seems to be this

However why is failing downloading the image is not clear

[DavidGOrtega]

Ok, GH runner seems to be restarting due to updates. This is a new behaviour

Runner will exit shortly for update, should back online within 10 seconds.

[sergeychuvakin]

Ok, GH runner seems to be restarting due to updates. This is a new behaviour

Runner will exit shortly for update, should back online within 10 seconds.

Ok, thank you for investigation! Appreciate your efforts, looking forward to updates

[casperdcl]

doesn't seem fixed (yet)

[DavidGOrtega]

fixed! Seems that the error that appeared later on was a hicup. I have tried multiple times successfully

[sergeychuvakin]

Thanks! I was able to run it as well.