testing: Flakey Windows TestOOMContainer

[sharanyad]

Summary

Windows functional test TestOOMContainer has been failing intermittently and frequently in the PRs.

Description

Expected Behavior

Test passes

Observed Behavior

Test fails

Environment Details

this is on last few PRs against gpu-support/dev:
#1757
#1761
#1762

Supporting Log Snippets

=== RUN   TestOOMContainer
--- FAIL: TestOOMContainer (10.51s)
	utils_windows.go:87: Registry location test_path_1545897391560631000
	utils_windows.go:97: datadir C:\Users\ADMINI~1\AppData\Local\Temp\amazon-ecs-agent\ftest_temp\ecs_integ_testdata298549938\data
	utils_windows.go:105: Created directory C:\Users\ADMINI~1\AppData\Local\Temp\amazon-ecs-agent\ftest_temp\ecs_integ_testdata298549938 to store test data in
	utils.go:164: Found agent metadata: {Cluster:ecs-functional-tests ContainerInstanceArn:0xc042348360 Version:Amazon ECS Agent - v1.23.0 (*UNKNOWN)}
	utils.go:185: Task definition: ecsftest-oom-container-windows-601ba04ca0a9e1d8536dadd3f48ea215:1
	utils.go:205: Started task: arn:aws:ecs:us-west-2:<redacted>:task/0233f825-b97b-4399-a711-577c6118fddd
	functionaltests_windows_test.go:214: 
			Error Trace:	functionaltests_windows_test.go:214
			Error:      	Received unexpected error:
			            	Task terminal; will never reach RUNNING
			            	github.com/aws/amazon-ecs-agent/agent/functional_tests/util.(*TestTask).waitStatus.func1
			            		C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/functional_tests/util/utils.go:527
			            	runtime.goexit
			            		C:/Go/src/runtime/asm_amd64.s:2337
			Test:       	TestOOMContainer
			Messages:   	Expect task to be running
	utils.go:172: Preserving test dir for failed test C:\Users\ADMINI~1\AppData\Local\Temp\amazon-ecs-agent\ftest_temp\ecs_integ_testdata298549938

Agent logs:

2018-12-27T07:56:40Z [INFO] Task engine [arn:aws:ecs:us-west-2:<redacted>:task/0233f825-b97b-4399-a711-577c6118fddd]: error transitioning container [memory-overcommit] to [CREATED]: Error: No such image: amazon/amazon-ecs-windows-python:make

[fenxiong]

Looks like we are having some trouble building the image this task uses (amazon/amazon-ecs-windows-python:make):

3-windowsservercore: Pulling from library/python
5847a47b8593: Pulling fs layer
10167262a014: Pulling fs layer
7c1b47cffe80: Pulling fs layer
bd2a3b91f57d: Pulling fs layer
02c93bd7124b: Pulling fs layer
1dfc9e9df202: Pulling fs layer
ac9dee776488: Pulling fs layer
8a421a1e3176: Pulling fs layer
3bb61653227a: Pulling fs layer
bd2a3b91f57d: Waiting
1dfc9e9df202: Waiting
02c93bd7124b: Waiting
ac9dee776488: Waiting
3bb61653227a: Waiting
8a421a1e3176: Waiting
7c1b47cffe80: Verifying Checksum
7c1b47cffe80: Download complete
bd2a3b91f57d: Verifying Checksum
bd2a3b91f57d: Download complete
02c93bd7124b: Verifying Checksum
02c93bd7124b: Download complete
1dfc9e9df202: Verifying Checksum
1dfc9e9df202: Download complete
ac9dee776488: Verifying Checksum
ac9dee776488: Download complete
8a421a1e3176: Verifying Checksum
8a421a1e3176: Download complete
3bb61653227a: Verifying Checksum
3bb61653227a: Download complete
5847a47b8593: Verifying Checksum
5847a47b8593: Download complete
10167262a014: Verifying Checksum
10167262a014: Download complete
5847a47b8593: Pull complete
10167262a014: Pull complete
7c1b47cffe80: Pull complete
bd2a3b91f57d: Pull complete
02c93bd7124b: Pull complete
docker : failed to register layer: re-exec error: exit status 1: output: time="2018-12-27T07:26:33Z" level=error 
msg="hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\\\?\\C:\\ProgramD
ata\\docker\\windowsfilter\\0f02b4a3330424dc9a231b3481d7ce755777f3f1c858bf8578d8f82f5e33edd3 flavour=1 
folder=C:\\Windows\\TEMP\\hcs038920135" 
At C:\Users\Administrator\AppData\Local\Temp\amazon-ecs-agent\go\src\github.com\aws\amazon-ecs-agent\misc\windows-pytho
n\build.ps1:14 char:1
+ docker pull python:3-windowsservercore
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (failed to regis...\hcs038920135" :String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) 
layerId=\\?\C:\ProgramData\docker\windowsfilter\0f02b4a3330424dc9a231b3481d7ce755777f3f1c858bf8578d8f82f5e33edd3 
flavour=1 folder=C:\Windows\TEMP\hcs038920135

docker : Error response from daemon: No such image: python:3-windowsservercore
At C:\Users\Administrator\AppData\Local\Temp\amazon-ecs-agent\go\src\github.com\aws\amazon-ecs-agent\misc\windows-pytho
n\build.ps1:15 char:1
+ docker tag python:3-windowsservercore amazon/amazon-ecs-windows-pytho ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...ndowsservercore:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

[fenxiong]

#1772 fixes this. closing