AWS SDK init error slow down the start time

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

deploy disagg tiflash on AWS S3

2. What did you expect to see? (Required)

3. What did you see instead (Required)

there are some error log when AWS SDK init, takes more than 40 seconds to starts a TiFlash instance

[2023/03/24 09:00:19.072 +00:00] [INFO] [Server.cpp:1082] ["ServerInfo: CPU: \n     logical cores: 16\n     physical cores: 16\n     frequency: 0MHz\n     l1 cache size: 65536\n     l1 cache line size: 64\n     l2 cache size: 1048576\n     l2 cache line size: 64\n     l3 cache size: 33554432\n     l3 cache line size: 64\n     arch: arm64\nDisk: \n     name: /dev/nvme0n1p1\n     type: SSD\n     total space: 21452795904\n     free space: 15912763392\n     mount point: /dev/termination-log\n     fstype: xfs\nDisk: \n     name: /dev/nvme0n1p1\n     type: SSD\n     total space: 21452795904\n     free space: 15912763392\n     mount point: /etc/resolv.conf\n     fstype: xfs\nDisk: \n     name: /dev/nvme0n1p1\n     type: SSD\n     total space: 21452795904\n     free space: 15912763392\n     mount point: /etc/hostname\n     fstype: xfs\nDisk: \n     name: /dev/nvme0n1p1\n     type: SSD\n     total space: 21452795904\n     free space: 15912763392\n     mount point: /etc/hosts\n     fstype: xfs\nDisk: \n     name: /dev/nvme1n1\n     type: SSD\n     total space: 211243999232\n     free space: 211151294464\n     mount point: /data0\n     fstype: ext4\nDisk: \n     name: overlay\n     type: UNKNOWN\n     total space: 21452795904\n     free space: 15912763392\n     mount point: /\n     fstype: overlay\nMemory: \n     capacity: 65949151232\n"] [thread_id=1]
[2023/03/24 09:00:19.085 +00:00] [INFO] [Server.cpp:1133] ["disaggregated_mode=Storage use_autoscaler=false enable_s3=true"] [thread_id=1]
[2023/03/24 09:00:19.090 +00:00] [WARN] [S3Common.cpp:126] ["tag=ClientConfiguration message=Retry Strategy will use the default max attempts."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:19.092 +00:00] [WARN] [S3Common.cpp:126] ["tag=ClientConfiguration message=Retry Strategy will use the default max attempts."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:24.101 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:24.101 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:24.101 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Request failed, now waiting 0 ms before attempting again."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:29.109 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:29.109 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:29.109 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Request failed, now waiting 50 ms before attempting again."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:34.168 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:34.168 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:34.168 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Request failed, now waiting 100 ms before attempting again."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:39.271 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:39.271 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:39.271 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Can not retrieve resource from https://sts.us-east-1.amazonaws.com"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:39.271 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Get an empty credential from sts"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:40.273 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:40.273 +00:00] [ERROR] [S3Common.cpp:126] ["tag=EC2MetadataClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:40.273 +00:00] [WARN] [S3Common.cpp:126] ["tag=EC2MetadataClient message=Request failed, now waiting 0 ms before attempting again."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:41.274 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:41.274 +00:00] [ERROR] [S3Common.cpp:126] ["tag=EC2MetadataClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:41.274 +00:00] [ERROR] [S3Common.cpp:126] ["tag=EC2MetadataClient message=Can not retrieve resource from http://169.254.169.254/latest/api/token"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:46.284 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:46.284 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:46.285 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Request failed, now waiting 0 ms before attempting again."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:51.296 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:51.296 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:51.296 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Request failed, now waiting 50 ms before attempting again."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:56.355 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:56.355 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:00:56.355 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Request failed, now waiting 100 ms before attempting again."] [source=AWSClient] [thread_id=1]
[2023/03/24 09:01:01.465 +00:00] [ERROR] [S3Common.cpp:126] ["tag=CurlHttpClient message=Curl returned error code 28 - Timeout was reached"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:01:01.465 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Http request to retrieve credentials failed"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:01:01.465 +00:00] [ERROR] [S3Common.cpp:126] ["tag=STSResourceClient message=Can not retrieve resource from https://sts.us-east-1.amazonaws.com"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:01:01.465 +00:00] [WARN] [S3Common.cpp:126] ["tag=STSResourceClient message=Get an empty credential from sts"] [source=AWSClient] [thread_id=1]
[2023/03/24 09:01:01.465 +00:00] [INFO] [PathCapacityMetrics.cpp:91] ["Init capacity [path=/data0/tiflash/] [capacity=0.00 B]"] [thread_id=1]

4. What is your TiFlash version? (Required)

master

[JaySon-Huang]

ref #6882

[JaySon-Huang]

similar to #7176. Fixed after using the Poco http client