Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to use containers One or more containers failed to start. #2173

Closed
jlouros opened this issue Oct 4, 2022 · 6 comments
Closed

Unable to use containers One or more containers failed to start. #2173

jlouros opened this issue Oct 4, 2022 · 6 comments
Labels
bug Something isn't working

Comments

@jlouros
Copy link

jlouros commented Oct 4, 2022 •
edited
Loading

Describe the bug
This issue happened after version v2.298.0 was released.
Currently, we are unable to use containers in our Workflows.
Error message is a vague One or more containers failed to start.

To Reproduce
Steps to reproduce the behavior:

  1. configure a workflow using services
name: Validate consul-template

on:
  workflow_call:

jobs:
  validate-consul-template:
    if: needs.read-workflow-settings.outputs.spa_only == 'false'
    needs:
      - read-workflow-settings
    runs-on: [self-hosted, linux, shs]
    steps:
      - name: checkout source code
        uses: actions/checkout@v3

      - name: run consul-template validation
        env:
          VAULT_ADDR: http://127.0.0.1:8200
          VAULT_TOKEN: myroot
        run: |
          echo "we run consul-template validation here!"
        shell: pwsh

    services:
      consul:
        image: public.ecr.aws/hashicorp/consul:1.11.2
        ports:
          - 8500:8500
      vault:
        image: public.ecr.aws/hashicorp/vault:1.9.3
        options: --name vault
        ports:
          - 8200:8200
        env:
          VAULT_DEV_ROOT_TOKEN_ID: myroot
          VAULT_ADDR: http://127.0.0.1:8200
          VAULT_TOKEN: myroot
  1. GitHub Workflows throws a vague error stating it cannot start the containers

Expected behavior

Should not error initializing Docker containers

Runner Version and Platform

Version of your runner? v2.298.0

OS of the machine running the runner? Linux

What's not working?

Please include error messages and screenshots.

image

Job Log Output

2022-10-04T15:09:14.1955842Z Requested labels: self-hosted, linux, shs
2022-10-04T15:09:14.1955920Z Job defined at: ***/github-actions/.github/workflows/shared-validate-code.yml@refs/heads/master
2022-10-04T15:09:14.1955946Z Waiting for a runner to pick up this job...
2022-10-04T15:09:16.6345183Z Job is about to start running on the runner: i-0176fa65802bbb2e7 (organization)
2022-10-04T15:09:20.6523258Z Current runner version: '2.298.0'
2022-10-04T15:09:20.6531924Z Runner name: 'i-0176fa65802bbb2e7'
2022-10-04T15:09:20.6532509Z Runner group name: 'Default'
2022-10-04T15:09:20.6533294Z Machine name: 'ip-10-198-144-169'
2022-10-04T15:09:20.6535980Z ##[group]GITHUB_TOKEN Permissions
2022-10-04T15:09:20.6536764Z Actions: write
2022-10-04T15:09:20.6537222Z Checks: write
2022-10-04T15:09:20.6537576Z Contents: write
2022-10-04T15:09:20.6537976Z Deployments: write
2022-10-04T15:09:20.6538392Z Discussions: write
2022-10-04T15:09:20.6538829Z Issues: write
2022-10-04T15:09:20.6539250Z Metadata: read
2022-10-04T15:09:20.6539638Z Packages: write
2022-10-04T15:09:20.6540027Z Pages: write
2022-10-04T15:09:20.6540397Z PullRequests: write
2022-10-04T15:09:20.6540858Z RepositoryProjects: write
2022-10-04T15:09:20.6541339Z SecurityEvents: write
2022-10-04T15:09:20.6541749Z Statuses: write
2022-10-04T15:09:20.6542107Z ##[endgroup]
2022-10-04T15:09:20.6546261Z Secret source: Actions
2022-10-04T15:09:20.6547048Z Prepare workflow directory
2022-10-04T15:09:20.8318322Z Prepare all required actions
2022-10-04T15:09:20.8581515Z Getting action download info
2022-10-04T15:09:21.5171297Z Download action repository 'actions/checkout@v3' (SHA:93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8)
2022-10-04T15:09:22.2194101Z Download action repository '***/github-actions@master' (SHA:cc465980866bb42a96e6267d8667bfa4a0dd14f4)
2022-10-04T15:09:23.4924297Z Uses: ***/github-actions/.github/workflows/shared-validate-code.yml
2022-10-04T15:09:23.4926558Z Complete job name: validate-code / validate-consul-template
2022-10-04T15:09:23.5760108Z ##[group]Checking docker version
2022-10-04T15:09:23.5788945Z ##[command]/home/runners/bin/docker version --format '{{.Server.APIVersion}}'
2022-10-04T15:09:23.6043244Z '1.41'
2022-10-04T15:09:23.6063645Z Docker daemon API version: '1.41'
2022-10-04T15:09:23.6064226Z ##[command]/home/runners/bin/docker version --format '{{.Client.APIVersion}}'
2022-10-04T15:09:23.6336521Z '1.41'
2022-10-04T15:09:23.6353780Z Docker client API version: '1.41'
2022-10-04T15:09:23.6369406Z ##[endgroup]
2022-10-04T15:09:23.6374951Z ##[group]Clean up resources from previous jobs
2022-10-04T15:09:23.6382805Z ##[command]/home/runners/bin/docker ps --all --quiet --no-trunc --filter "label=765292"
2022-10-04T15:09:23.6775317Z ##[command]/home/runners/bin/docker network prune --force --filter "label=765292"
2022-10-04T15:09:23.6986724Z ##[endgroup]
2022-10-04T15:09:23.6987108Z ##[group]Create local container network
2022-10-04T15:09:23.7000535Z ##[command]/home/runners/bin/docker network create --label 765292 github_network_fdb043dc3db245deb7bc0960653ffe70
2022-10-04T15:09:23.7573742Z e39243ad3bab404286ec1706405d91507c7e26e656680f4166d9f76f4ae557c6
2022-10-04T15:09:23.7592754Z ##[endgroup]
2022-10-04T15:09:23.7713831Z ##[group]Starting consul service container
2022-10-04T15:09:23.7740795Z ##[command]/home/runners/bin/docker pull public.ecr.aws/hashicorp/consul:1.11.2
2022-10-04T15:09:25.0705795Z 1.11.2: Pulling from hashicorp/consul
2022-10-04T15:09:25.4080643Z 5758d4e389a3: Pulling fs layer
2022-10-04T15:09:25.4081857Z 893bc6280f09: Pulling fs layer
2022-10-04T15:09:25.4082792Z a2874d28cf40: Pulling fs layer
2022-10-04T15:09:25.4084147Z 9f9456e01a30: Pulling fs layer
2022-10-04T15:09:25.4085437Z 763d944e188d: Pulling fs layer
2022-10-04T15:09:25.4086254Z bca03003813c: Pulling fs layer
2022-10-04T15:09:25.4086809Z 9f9456e01a30: Waiting
2022-10-04T15:09:25.4091917Z 763d944e188d: Waiting
2022-10-04T15:09:25.4092666Z bca03003813c: Waiting
2022-10-04T15:09:25.7584470Z 893bc6280f09: Verifying Checksum
2022-10-04T15:09:25.7585262Z 893bc6280f09: Download complete
2022-10-04T15:09:26.0386102Z 5758d4e389a3: Verifying Checksum
2022-10-04T15:09:26.0386754Z 5758d4e389a3: Download complete
2022-10-04T15:09:26.1851588Z 9f9456e01a30: Verifying Checksum
2022-10-04T15:09:26.1852201Z 9f9456e01a30: Download complete
2022-10-04T15:09:26.2799952Z 5758d4e389a3: Pull complete
2022-10-04T15:09:26.4577417Z 763d944e188d: Verifying Checksum
2022-10-04T15:09:26.4578005Z 763d944e188d: Download complete
2022-10-04T15:09:26.4721999Z 893bc6280f09: Pull complete
2022-10-04T15:09:26.6347271Z bca03003813c: Download complete
2022-10-04T15:09:27.6316162Z a2874d28cf40: Verifying Checksum
2022-10-04T15:09:27.6317753Z a2874d28cf40: Download complete
2022-10-04T15:09:29.4995634Z a2874d28cf40: Pull complete
2022-10-04T15:09:29.5644975Z 9f9456e01a30: Pull complete
2022-10-04T15:09:29.6268148Z 763d944e188d: Pull complete
2022-10-04T15:09:29.6873317Z bca03003813c: Pull complete
2022-10-04T15:09:29.6994954Z Digest: sha256:c3f65c5ab2261f12cf37dd229597990cc96c53bc869f90e740a59729bf0e1b7a
2022-10-04T15:09:29.7038114Z Status: Downloaded newer image for public.ecr.aws/hashicorp/consul:1.11.2
2022-10-04T15:09:29.7073523Z public.ecr.aws/hashicorp/consul:1.11.2
2022-10-04T15:09:29.7166147Z ##[command]/home/runners/bin/docker create --name d5f00fc5db7d4dbeb188a2cb20d1743a_publicecrawshashicorpconsul1112_48cd11 --label 765292 --network github_network_fdb043dc3db245deb7bc0960653ffe70 --network-alias consul -p 8500:8500  -e GITHUB_ACTIONS=true -e CI=true public.ecr.aws/hashicorp/consul:1.11.2
2022-10-04T15:09:29.7619946Z 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:29.7645703Z ##[command]/home/runners/bin/docker start 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:30.1999512Z 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:30.2024648Z ##[command]/home/runners/bin/docker ps --all --filter id=5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
2022-10-04T15:09:30.2285663Z 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6 Up Less than a second
2022-10-04T15:09:30.2314878Z ##[command]/home/runners/bin/docker port 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:30.2600723Z 8500/tcp -> 0.0.0.0:8500
2022-10-04T15:09:30.2601507Z 8500/tcp -> :::8500
2022-10-04T15:09:30.2734030Z ##[endgroup]
2022-10-04T15:09:30.2735530Z ##[group]Starting vault service container
2022-10-04T15:09:30.2736685Z ##[command]/home/runners/bin/docker pull public.ecr.aws/hashicorp/vault:1.9.3
2022-10-04T15:09:31.5376222Z 1.9.3: Pulling from hashicorp/vault
2022-10-04T15:09:31.8797392Z 97518928ae5f: Pulling fs layer
2022-10-04T15:09:31.8798412Z 532c9aba9710: Pulling fs layer
2022-10-04T15:09:31.8799304Z b112c367dc25: Pulling fs layer
2022-10-04T15:09:31.8799794Z 473974dcf0bf: Pulling fs layer
2022-10-04T15:09:31.8800256Z 9a87bfa9e28f: Pulling fs layer
2022-10-04T15:09:31.8800761Z 7e648fb1bea9: Pulling fs layer
2022-10-04T15:09:31.8829875Z 9a87bfa9e28f: Waiting
2022-10-04T15:09:31.8830726Z 7e648fb1bea9: Waiting
2022-10-04T15:09:31.8831300Z 473974dcf0bf: Waiting
2022-10-04T15:09:32.2341390Z 532c9aba9710: Download complete
2022-10-04T15:09:32.2903679Z b112c367dc25: Verifying Checksum
2022-10-04T15:09:32.2910028Z b112c367dc25: Download complete
2022-10-04T15:09:32.4383231Z 97518928ae5f: Download complete
2022-10-04T15:09:32.6201549Z 97518928ae5f: Pull complete
2022-10-04T15:09:32.6654249Z 9a87bfa9e28f: Verifying Checksum
2022-10-04T15:09:32.6654833Z 9a87bfa9e28f: Download complete
2022-10-04T15:09:32.7483919Z 532c9aba9710: Pull complete
2022-10-04T15:09:32.8800629Z 7e648fb1bea9: Verifying Checksum
2022-10-04T15:09:32.8802198Z 7e648fb1bea9: Download complete
2022-10-04T15:09:33.0258207Z b112c367dc25: Pull complete
2022-10-04T15:09:35.4098570Z 473974dcf0bf: Verifying Checksum
2022-10-04T15:09:35.4099849Z 473974dcf0bf: Download complete
2022-10-04T15:09:38.4116152Z 473974dcf0bf: Pull complete
2022-10-04T15:09:38.5185007Z 9a87bfa9e28f: Pull complete
2022-10-04T15:09:38.5822858Z 7e648fb1bea9: Pull complete
2022-10-04T15:09:38.5952987Z Digest: sha256:c019349bcaeeb42cff890110212f571b6745b969c01e734f401998b6fcb3a75a
2022-10-04T15:09:38.5998870Z Status: Downloaded newer image for public.ecr.aws/hashicorp/vault:1.9.3
2022-10-04T15:09:38.6019862Z public.ecr.aws/hashicorp/vault:1.9.3
2022-10-04T15:09:38.6074550Z ##[command]/home/runners/bin/docker create --name d3d52b1873b44ce483fd5183225552e7_publicecrawshashicorpvault193_3caf42 --label 765292 --network github_network_fdb043dc3db245deb7bc0960653ffe70 --network-alias vault -p 8200:8200 --name vault -e "VAULT_DEV_ROOT_TOKEN_ID=myroot" -e "VAULT_ADDR=http://127.0.0.1:8200" -e "VAULT_TOKEN=myroot" -e GITHUB_ACTIONS=true -e CI=true public.ecr.aws/hashicorp/vault:1.9.3
2022-10-04T15:09:38.6578944Z d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:38.6590879Z ##[command]/home/runners/bin/docker start d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:39.0210698Z d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:39.0227532Z ##[command]/home/runners/bin/docker ps --all --filter id=d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
2022-10-04T15:09:39.0429153Z d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879 Up Less than a second
2022-10-04T15:09:39.0445947Z ##[command]/home/runners/bin/docker port d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:39.0660341Z 8200/tcp -> 0.0.0.0:8200
2022-10-04T15:09:39.0661250Z 8200/tcp -> :::8200
2022-10-04T15:09:39.0677740Z ##[endgroup]
2022-10-04T15:09:39.0720662Z ##[group]Waiting for all services to be ready
2022-10-04T15:09:39.0788969Z ##[command]/home/runners/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:39.1015701Z ##[command]/home/runners/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:39.1243017Z ##[endgroup]
2022-10-04T15:09:39.1243403Z ##[group]Service container consul failed.
2022-10-04T15:09:39.1253035Z ##[command]/home/runners/bin/docker logs --details 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:39.1461110Z  ==> Starting Consul agent...
2022-10-04T15:09:39.1462189Z             Version: '1.11.2'
2022-10-04T15:09:39.1462901Z             Node ID: '8a0a0d76-85d2-c427-db46-1690935cdc9e'
2022-10-04T15:09:39.1463594Z           Node name: '5bdd1a7bc18b'
2022-10-04T15:09:39.1464342Z          Datacenter: 'dc1' (Segment: '<all>')
2022-10-04T15:09:39.1465052Z              Server: true (Bootstrap: false)
2022-10-04T15:09:39.1465801Z         Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
2022-10-04T15:09:39.1466578Z        Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
2022-10-04T15:09:39.1467451Z             Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2022-10-04T15:09:39.1468229Z  
2022-10-04T15:09:39.1468805Z  ==> Log data will now stream in as it occurs:
2022-10-04T15:09:39.1469423Z  
2022-10-04T15:09:39.1470307Z  2022-10-04T15:09:30.283Z [INFO]  agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8a0a0d76-85d2-c427-db46-1690935cdc9e Address:127.0.0.1:8300}]"
2022-10-04T15:09:39.1476440Z  2022-10-04T15:09:30.283Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: 5bdd1a7bc18b.dc1 127.0.0.1
2022-10-04T15:09:39.1477351Z  2022-10-04T15:09:30.284Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: 5bdd1a7bc18b 127.0.0.1
2022-10-04T15:09:39.1478132Z  2022-10-04T15:09:30.284Z [INFO]  agent.router: Initializing LAN area manager
2022-10-04T15:09:39.1478871Z  2022-10-04T15:09:30.284Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
2022-10-04T15:09:39.1479704Z  2022-10-04T15:09:30.284Z [INFO]  agent.server.raft: entering follower state: follower="Node at 127.0.0.1:8300 [Follower]" leader=
2022-10-04T15:09:39.1480592Z  2022-10-04T15:09:30.284Z [INFO]  agent.server: Adding LAN server: server="5bdd1a7bc18b (Addr: tcp/127.0.0.1:8300) (DC: dc1)"
2022-10-04T15:09:39.1481794Z  2022-10-04T15:09:30.286Z [WARN]  agent: grpc: addrConn.createTransport failed to connect to {dc1-127.0.0.1:8300 0 5bdd1a7bc18b <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:0->127.0.0.1:8300: operation was canceled". Reconnecting...
2022-10-04T15:09:39.1483119Z  2022-10-04T15:09:30.286Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2022-10-04T15:09:39.1483988Z  2022-10-04T15:09:30.286Z [INFO]  agent.server: Handled event for server in area: event=member-join server=5bdd1a7bc18b.dc1 area=wan
2022-10-04T15:09:39.1485319Z  2022-10-04T15:09:30.286Z [WARN]  agent: grpc: addrConn.createTransport failed to connect to {dc1-127.0.0.1:8300 0 5bdd1a7bc18b <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:0->127.0.0.1:8300: operation was canceled". Reconnecting...
2022-10-04T15:09:39.1486642Z  2022-10-04T15:09:30.287Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
2022-10-04T15:09:39.1487775Z  2022-10-04T15:09:30.287Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
2022-10-04T15:09:39.1488819Z  2022-10-04T15:09:30.287Z [INFO]  agent: started state syncer
2022-10-04T15:09:39.1489461Z  2022-10-04T15:09:30.287Z [INFO]  agent: Consul agent running!
2022-10-04T15:09:39.1490420Z  2022-10-04T15:09:30.287Z [INFO]  agent: Started gRPC server: address=[::]:8502 network=tcp
2022-10-04T15:09:39.1491359Z  2022-10-04T15:09:30.344Z [WARN]  agent.server.raft: heartbeat timeout reached, starting election: last-leader=
2022-10-04T15:09:39.1492267Z  2022-10-04T15:09:30.344Z [INFO]  agent.server.raft: entering candidate state: node="Node at 127.0.0.1:8300 [Candidate]" term=2
2022-10-04T15:09:39.1493048Z  2022-10-04T15:09:30.344Z [DEBUG] agent.server.raft: votes: needed=1
2022-10-04T15:09:39.1493834Z  2022-10-04T15:09:30.344Z [DEBUG] agent.server.raft: vote granted: from=8a0a0d76-85d2-c427-db46-1690935cdc9e term=2 tally=1
2022-10-04T15:09:39.1494618Z  2022-10-04T15:09:30.344Z [INFO]  agent.server.raft: election won: tally=1
2022-10-04T15:09:39.1495406Z  2022-10-04T15:09:30.344Z [INFO]  agent.server.raft: entering leader state: leader="Node at 127.0.0.1:8300 [Leader]"
2022-10-04T15:09:39.1496172Z  2022-10-04T15:09:30.344Z [INFO]  agent.server: cluster leadership acquired
2022-10-04T15:09:39.1496904Z  2022-10-04T15:09:30.344Z [INFO]  agent.server: New leader elected: payload=5bdd1a7bc18b
2022-10-04T15:09:39.1497694Z  2022-10-04T15:09:30.345Z [INFO]  agent.leader: started routine: routine="federation state anti-entropy"
2022-10-04T15:09:39.1498516Z  2022-10-04T15:09:30.345Z [DEBUG] agent.server.autopilot: autopilot is now running
2022-10-04T15:09:39.1499304Z  2022-10-04T15:09:30.345Z [DEBUG] agent.server.autopilot: state update routine is now running
2022-10-04T15:09:39.1500123Z  2022-10-04T15:09:30.345Z [INFO]  agent.leader: started routine: routine="federation state pruning"
2022-10-04T15:09:39.1501077Z  2022-10-04T15:09:30.346Z [DEBUG] connect.ca.consul: consul CA provider configured: id=fb:50:9b:45:1a:65:15:c1:68:57:73:5f:da:cd:b8:0d:0f:e2:26:eb:68:66:43:11:85:9d:67:a9:7a:56:9c:b9 is_primary=true
2022-10-04T15:09:39.1502077Z  2022-10-04T15:09:30.347Z [INFO]  connect.ca: initialized primary datacenter CA with provider: provider=consul
2022-10-04T15:09:39.1502966Z  2022-10-04T15:09:30.347Z [INFO]  agent.leader: started routine: routine="intermediate cert renew watch"
2022-10-04T15:09:39.1503771Z  2022-10-04T15:09:30.347Z [INFO]  agent.leader: started routine: routine="CA root pruning"
2022-10-04T15:09:39.1504581Z  2022-10-04T15:09:30.347Z [INFO]  agent.leader: started routine: routine="CA root expiration metric"
2022-10-04T15:09:39.1505417Z  2022-10-04T15:09:30.347Z [INFO]  agent.leader: started routine: routine="CA signing expiration metric"
2022-10-04T15:09:39.1506251Z  2022-10-04T15:09:30.347Z [INFO]  agent.leader: started routine: routine="virtual IP version check"
2022-10-04T15:09:39.1507096Z  2022-10-04T15:09:30.347Z [DEBUG] agent.server: successfully established leadership: duration=2.714056ms
2022-10-04T15:09:39.1508092Z  2022-10-04T15:09:30.347Z [INFO]  agent.server: member joined, marking health alive: member=5bdd1a7bc18b partition=default
2022-10-04T15:09:39.1508969Z  2022-10-04T15:09:30.347Z [DEBUG] agent.leader: stopping routine: routine="virtual IP version check"
2022-10-04T15:09:39.1509789Z  2022-10-04T15:09:30.347Z [DEBUG] agent.leader: stopped routine: routine="virtual IP version check"
2022-10-04T15:09:39.1510578Z  2022-10-04T15:09:30.571Z [INFO]  agent.server: federation state anti-entropy synced
2022-10-04T15:09:39.1511403Z  2022-10-04T15:09:30.709Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
2022-10-04T15:09:39.1512146Z  2022-10-04T15:09:30.709Z [INFO]  agent: Synced node info
2022-10-04T15:09:39.1512731Z  2022-10-04T15:09:30.709Z [DEBUG] agent: Node info in sync
2022-10-04T15:09:39.1513475Z  2022-10-04T15:09:33.666Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
2022-10-04T15:09:39.1514219Z  2022-10-04T15:09:33.666Z [DEBUG] agent: Node info in sync
2022-10-04T15:09:39.1539720Z ##[error]Failed to initialize container public.ecr.aws/hashicorp/consul:1.11.2
2022-10-04T15:09:39.1553075Z ##[endgroup]
2022-10-04T15:09:39.1553571Z ##[group]Service container vault failed.
2022-10-04T15:09:39.1554495Z ##[command]/home/runners/bin/docker logs --details d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:39.1762490Z ##[error]Failed to initialize container public.ecr.aws/hashicorp/vault:1.9.3
2022-10-04T15:09:39.1763625Z ##[endgroup]
2022-10-04T15:09:39.1871632Z ##[error]One or more containers failed to start.
2022-10-04T15:09:39.1978592Z Stop and remove container: d5f00fc5db7d4dbeb188a2cb20d1743a_publicecrawshashicorpconsul1112_48cd11
2022-10-04T15:09:39.1985364Z ##[command]/home/runners/bin/docker rm --force 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:39.8824814Z 5bdd1a7bc18b5ec75692456eab1b14db016ecf51faabcf4473c89b95e9a0c6c6
2022-10-04T15:09:39.8856228Z Stop and remove container: d3d52b1873b44ce483fd5183225552e7_publicecrawshashicorpvault193_3caf42
2022-10-04T15:09:39.8857397Z ##[command]/home/runners/bin/docker rm --force d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:40.1124814Z d43e6ef14e44d30dabd62f648291c089078121bea02f71ecffd773d0396e8879
2022-10-04T15:09:40.1149729Z Remove container network: github_network_fdb043dc3db245deb7bc0960653ffe70
2022-10-04T15:09:40.1156223Z ##[command]/home/runners/bin/docker network rm github_network_fdb043dc3db245deb7bc0960653ffe70
2022-10-04T15:09:40.2285746Z github_network_fdb043dc3db245deb7bc0960653ffe70
2022-10-04T15:09:40.2470280Z Cleaning up orphan processes

Runner and Worker's Diagnostic Logs

If applicable, add relevant diagnostic log information. Logs are located in the runner's _diag folder. The runner logs are prefixed with Runner_ and the worker logs are prefixed with Worker_. Each job run correlates to a worker log. All sensitive information should already be masked out, but please double-check before pasting here.
@jlouros jlouros added the bug Something isn't working label Oct 4, 2022
@masterful
Copy link

I won't add our configuration to this comment, but we also encountered the same issue where workflows/jobs reliant on a service (in our case mysql:5.7) were no longer initializing properly due to the service(s) being identified as unhealthy/failing to start.

I believe it's likely related to #2110 since reverting our runtime to 2.297.0 (and disabling updates when configuring the runner) caused our workflows to begin working again.

@cktaylor
Copy link

cktaylor commented Oct 4, 2022 •
edited
Loading

Ditto - we have seen the issue as well (with mysql, memcache, and redis)

The containers are hitting the // Container has no HEALTHCHECK logic, and the changes to RunContainersHealthcheck now will compare the resulting empty string against "healthy" - add it as a unhealthy container and then ultimately - throw new InvalidOperationException

Thereby failing the job and our workflows.

This seems to go against the release notes where it suggests it would use the exit code when a healthcheck isn't configured.

@sblack4
Copy link

sblack4 commented Oct 4, 2022

Double ditto for runner version 2.298.0. We have service containers for redis and MySQL. When the containers get to the Initialize containers step they try to evaluate the health check (which doesn't exist) with commands like the below

Waiting for all services to be ready
  /usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" 53dff748f2051ae29cb684a724dadbac4d7fb04b09d543ad9d8b7d4949f88f8c
  /usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" dcf699c9a5e7047740ce068d06cc808d5ad2372ee40a53f82cb2b05d0b1c4f

And the workflow fails with Error: One or more containers failed to start.. The fix was to add health checks like those included below

    services:
      redis:
        image: redis:6.2
        options: >-
          --health-cmd "redis-cli ping"

      mysql:
        image: mysql:8
        options: >-
          --health-cmd "mysqladmin ping -h localhost"

@fhammerl
Copy link
Contributor

fhammerl commented Oct 4, 2022

Hi All, thanks for reporting this issue so fast. We'll be releasing 2.298.2 soon to revert the changes causing this issue.

@fhammerl
Copy link
Contributor

fhammerl commented Oct 4, 2022

Thanks for your patience on this one. If you keep experiencing these service failures on 2.298.2 or higher, I will reopen this ticket.

@fhammerl fhammerl closed this as completed Oct 4, 2022
@trixobird
Copy link

I facing this issue today out of the blue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@masterful @cktaylor @jlouros @trixobird @sblack4 @fhammerl