too many open files!

[youjiahe]

Query:

level=warn ts=2020-08-18T08:24:38.513114583Z caller=proxy.go:456 err="receive series from Addr: 10.92.206.68:10101 LabelSets: [name:"monitor" value:"CK_TEST" ][name:"monitor" value:"prometheus_thanos" ] Mint: 1595928792342 Maxt: 1597730400000: rpc error: code = Aborted desc = fetch series for block 01EFQ5FQET70JRRTDS42784NZ3: preload chunks: read range for 0: get range reader: Get "http://thanos-ecs-oss-sz.oss-cn-shenzhen.aliyuncs.com/01EFQ5FQET70JRRTDS42784NZ3%2Fchunks%2F000001\": dial tcp 120.77.167.27:80: socket: too many open files" msg="returning partial response"

Store:

Aug 18 16:26:31 sz-thanos-test thanos[18809]: level=warn ts=2020-08-18T08:26:31.762308091Z caller=store.go:332 msg="syncing blocks failed" err="BaseFetcher: iter bucket: listing aliyun oss bucket failed: Get "http://ecs-pms-test.oss-cn-shenzhen.aliyuncs.com/?delimiter=%2F&encoding-type=url&marker=01ECWTPYD7J49MXGY827MYJ489%2F&prefix=\": dial tcp: lookup ecs-pms-test.oss-cn-shenzhen.aliyuncs.com on 100.100.2.136:53: dial udp 100.100.2.136:53: socket: too many open files"

[GiedriusS]

Hello! Sorry for the problems. What kind of ulimits do you have on the Thanos process? (: How could we reproduce this?

[youjiahe]

root soft nofile 409600
root hard nofile 409600
thanos soft nofile 409600
thanos hard nofile 409600

• soft nofile 409600
• hard nofile 409600

hanos-test thanos]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30631
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 409600
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 30631
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited