docker pull failed with context canceled error.

[rpramodd]

Originally posted by @cgbaker in #4934 (comment)

Find the requested details below.

Error Message:

• Docker version on client node ==> 18.09.5
• Client logs ==>

Recent Events:
Time Type Description
2019-05-09T10:39:44Z Restarting Task restarting in 15.732300553s
2019-05-09T10:39:44Z Driver Failure Failed to pull gcr.io/project_name/image: context canceled
2019-05-09T10:38:14Z Driver Docker image pull progress: Pulled 17/17 (1.776 GiB/1.776 GiB) layers: 0 waiting/0 pulling
2019-05-09T10:36:04Z Driver Docker image pull progress: Pulled 17/17 (1.776 GiB/1.776 GiB) layers: 0 waiting/0 pulling
2019-05-09T10:33:54Z Driver Docker image pull progress: Pulled 17/17 (1.776 GiB/1.776 GiB) layers: 0 waiting/0 pulling
2019-05-09T10:31:54Z Driver Downloading image
2019-05-09T10:31:53Z Task Setup Building Task Directory
2019-05-09T10:31:53Z Received Task received by client

• Job spec (if possible)
• Docker image, or some general statistics (size, number of layers) if they seem relevant
  image size ==> 10.8 GB( base windows image is:
  microsoft/windowsservercore:1709 )

Dockerfile:

FROM microsoft/windowsservercore:1709
FROM openjdk:10.0.2-jdk-windowsservercore-1709
COPY runner.exe hostfile c:/temp/
ENV JAVA_OPTS "--add-modules java.xml.bind"
ENTRYPOINT c:/temp/runner.exe

[lmayorga1980]

I am having the same issue with the following docker version:

Client: Docker Engine - Enterprise
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        cde21d3829
 Built:             10/08/2019 17:59:26
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Enterprise
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.10
  Git commit:       cde21d3829
  Built:            10/08/2019 17:57:54
  OS/Arch:          windows/amd64
  Experimental:     false

Also seems like nomad is always trying to clean/pull the image even though the image is already in the docker host. Maybe https://www.nomadproject.io/docs/drivers/docker.html#force_pull is not enforced?

[jdebbink]

I am having the same issue with Windows Server 2019 pulling the ltsc2019 containers. Same versions as @lmayorga1980 on Nomad 0.10. I am running in AWS and this is repeatable using t3.medium and c5.xlarge instances for the Windows clients.

The only work workaround I've found to this issue is to use the larger instances with NVME drives like c5d.xlarge but would prefer not to use such large instances due to cost.

@nickethier @cgbaker I see in this issue
that a 2 minute timeout was introduced and is likely the culprit for this issue.

When I manually pull a docker image I see it pause at the end of extracting as shown below, sometimes for more than 2 minutes.

Are there any plans to configure this timeout? This seems somewhat detrimental for Windows.

[Flaniga3]

Is there any news on this? This is a huge problem for my company, and may prevent us from using Nomad. We need the ability to either use a locally cached Docker image, or configure the timeout for communication from Docker as we're using large Windows containers.

[hilayc]

We are having this issue too.
There should be an option to customize docker-image download timeout for each task.
Please get to this soon.

Our workaround is to manually downloading (docker pull ...) the docker-image on each server before running it via Nomad.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.