The self-hosted runner: cml-xxx lost communication with the server

[TorRient]

Hi everyone, i have a problem when i using CML to create host runner.

Log issue:
The self-hosted runner: cml-xxx lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

My cml.yaml:

name: CML Report
on:
  push:
    branches:
    - master
jobs:
  deploy-runner:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-cml@v1
      - uses: actions/checkout@v2
      - name: Deploy runner on EC2
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          GOOGLE_APPLICATION_CREDENTIALS_DATA: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_DATA }}
        run: |
          cml runner \
              --cloud=gcp \
              --cloud-region=us-west \
              --cloud-type=e2-highmem-4 \
              --labels=cml-cpu \
  evaluate:
    needs: [deploy-runner]
    runs-on: [self-hosted, cml-cpu]
    container:
      image: docker://ghcr.io/iterative/cml:0-dvc2-base1
    steps:
      - uses: actions/checkout@v2
      - name: Generate metrics report
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          GOOGLE_APPLICATION_CREDENTIALS_DATA: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_DATA }}
        run: |
          pip install -r requirements.txt
          dvc repro
          echo "## Metrics" >> report.md
          dvc metrics show --show-md >> report.md
          cml send-comment report.md

Any solution?
Thanks all!

[dacbd]

This is a known issue, if you are able to provide more logs we can try and give you a recommendation on what to change.

You can monitor this issue for updates: #1016
And this issue for how to get some additional logs that may help us if possible: #1006

The GHA agent tends to fail in rough ways when the instance becomes starved, Increasing the instance size should solve the problem but we are working to make a solution that allows the instance to fail safely and still be able to clean up the instance.

[DavidGOrtega]

duplicated of #1014

[TorRient]

The best solution is to not use cml runner, create runner on GitHub client and use VM instance on GCP / AWS ... connect to it 👍

[DavidGOrtega]

Just to clarify... cml runner uses GH runner internally 😁 So probably you have not yet solved your issue

[TorRient]

It is a temporary alternative to CML RUNNER, it's better to be faulty :D. I'm kidding.
I still appreciate cml runner, I am trying to fix that.