PutItem fails on all but one (or two) instances

[Quantumplation]

Version of AWS SDK for Go?

go get github.com/aws/aws-sdk-go/aws

From the logs below, looks like 1.25.16

Version of Go (go version)?

go version go1.13.1 linux/amd64

What issue did you see?

I am running a small one node kubernetes cluster, and deploying 5 instances of a small web server which saves statistics to dynamodb. Occasionally, my cluster will get into a state where a very small subset of containers ccan save to dynamodb. Redeploying / Restarting the containers doesn't fix it. I was only able to fix it by tearing down the VM itself and redeploying that.

I've already checked my AWS write capacity, and I'm waaay below it. I'm set to 2 writes per second, I'm writing once every 30 seconds, the containers have staggered start times. All charts for my throughput / throttling show that this is healthy.

This may be an EKS / AWS issue, rather than an issue with this module, but I don't know enough about any of this to differentiate, so if that's the case, some help debugging / redirecting to the right destination would be appreciated :)

Here's some logs with debugging turned on, though I've redacted things that look authorization related just in case...

2019/10/22 02:24:21 Saving...
2019/10/22 02:24:21 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
Host: ${IP}
User-Agent: aws-sdk-go/1.25.16 (go1.13.3; linux; amd64)
Accept-Encoding: gzip


-----------------------------------------------------
2019/10/22 02:24:21 DEBUG: Response ec2metadata/GetMetadata Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 63
Accept-Ranges: none
Content-Type: text/plain
Date: Tue, 22 Oct 2019 02:24:21 GMT
Last-Modified: Tue, 22 Oct 2019 02:23:32 GMT
Server: EC2ws


-----------------------------------------------------
2019/10/22 02:24:21 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials/${node-instance-role} HTTP/1.1
Host: ${IP}
User-Agent: aws-sdk-go/1.25.16 (go1.13.3; linux; amd64)
Accept-Encoding: gzip


-----------------------------------------------------
2019/10/22 02:24:22 DEBUG: Response ec2metadata/GetMetadata Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 1298
Accept-Ranges: none
Content-Type: text/plain
Date: Tue, 22 Oct 2019 02:24:22 GMT
Last-Modified: Tue, 22 Oct 2019 02:23:32 GMT
Server: EC2ws


-----------------------------------------------------
2019/10/22 02:24:22 DEBUG: Request dynamodb/PutItem Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: dynamodb.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.25.16 (go1.13.3; linux; amd64)
Content-Length: 681
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=${credential}, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=${signature}
Content-Type: application/x-amz-json-1.0
X-Amz-Date: 20191022T022421Z
X-Amz-Security-Token: ${token}
X-Amz-Target: DynamoDB_20120810.PutItem


-----------------------------------------------------                                                                                                                                                              
2019/10/22 02:24:52 DEBUG: Send Request dynamodb/PutItem failed, attempt 0/10, error RequestError: send request failed                                                                                                          
caused by: Post https://dynamodb.us-east-1.amazonaws.com/: dial tcp: i/o timeout                                                                                                                                                
2019/10/22 02:24:52 DEBUG: Request dynamodb/PutItem Details:                                                                                                                                                                    
---[ REQUEST POST-SIGN ]-----------------------------                                                                                                                                                                           
POST / HTTP/1.1                                                                                                                                                                                                                 
Host: dynamodb.us-east-1.amazonaws.com                                                                                                                                                                                          
User-Agent: aws-sdk-go/1.25.16 (go1.13.3; linux; amd64)                                                                                                                                                                         
Content-Length: 681                                                                                                                                                                                                             
Accept-Encoding: identity                                                                                                                                                                                                       
Authorization: AWS4-HMAC-SHA256 Credential=${credential}, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature ${signature}
Content-Type: application/x-amz-json-1.0                                                                                                                                                                                        
X-Amz-Date: 20191022T022452Z                                                                                                                                                                                                    
X-Amz-Security-Token: ${token}                              
X-Amz-Target: DynamoDB_20120810.PutItem                                                                                                                                                                                         

...

This then retries 10 times, and fails.

Steps to reproduce

The code being executed is here:

https://github.com/Quantumplation/eks-play/blob/c1367591f1ec72a23bb1067ea33243b3dab86b19/go-server.go

[ruudk]

Duplicate of #2972 and #2980