Job submission problem - private docker registry job image

[JianfeiHu]

When cluster is built, the registry on the configured azure is submitted to job. Does image URL fill in the server name?

[fanyangCS]

here is an example: "image": "xxx.azurecr.io/demo4pai:tag", please replace xxx with your private docker registry and fill in the ":tag" if you have.

[JianfeiHu]

It always implies authentication.
Error response from daemon: Get https://XXXX.azurecr.io/v2/aiplatform/pai.run.cntk.test/manifests/latest: unauthorized: authentication required

[YanjieGao]

please check these docker account configuration at the cluster is configured correct or not.
https://github.com/Microsoft/pai/blob/f69396fa20590b66df33f755a6d19f338796d21e/cluster-configuration/services-configuration.yaml#L35

[JianfeiHu]

There is no problem with username and password in the configuration file.

[YanjieGao]

Please debug at the occur error's server without PAI.
SSH to that server.
At that server pull and run your image locally to check it is registry problem or pai's problem.

[JianfeiHu]

It can be pull on other machines.

[YanjieGao]

need your hands on login?

[JianfeiHu]

@ydye

[YanjieGao]

Hi @ydye, jianfei's private azure acr repo image. If he config at service config of pai account and password, seems also meet error.
When he locally use that docker after hands on login could pull and run it.

[fanyangCS]

Please do not expose docker registry name as it is sensitive.

[fanyangCS]

can you provide more details (e.g., log). I do not understand the problem. In which step are you blocked?

[JianfeiHu]

http://10.150.144.225:8188/applicationhistory/logs/10.150.144.221:8041/container_1535419823654_0259_01_000002/container_1535419823654_0259_01_000002/hu

[YanjieGao]

error is ++[07:06:48] echo 'Can not pull Docker image'

Please refer: job_tutorial.md to have a try config the auth file:
Reference: https://github.com/Microsoft/pai/blob/master/docs/job_tutorial.md
'''
If you're using a private Docker registry which needs authentication for image pull and is different from the registry used during deployment, please create an authentication file in the following format, upload it to HDFS and specify the path in authFile parameter in config file.

docker_registry_server
username
password
NOTE: If you're using a private registry at Docker Hub, you should use docker.io for docker_registry_server field in the authentication file.
'''