request help: APIs too many cause 503 + proxy delay too much

[azura27]

Issue description

Our product environment has too many APIs, and the process of get info from etcd will be slow,
I wanna know if the prefix get method could be improve. OR is there similar issue I could refer

Environment

CentOS7
Request help without environment information will be ignored or closed.

• apisix version (cmd: apisix version):
• OS (cmd: uname -a):
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
• apisix-dashboard version, if have:
• luarocks version, if the issue is about installation (cmd: luarocks --version):

[azura27]

监控get /routes信息太多 好像和之前的类似

[spacewander]

How many routes do you have?
How many APISIX/etcd nodes do you have?
What about their sizes?

[azura27]

How many routes do you have? 8906
How many APISIX/etcd nodes do you have? three node construct a cluster, config in file two
What about their sizes? a former record is to fetch 8M and cause error "received message larger than max "

[azura27]

using etcdctl tools I could pull all routes(size 7M) which means the etcd server allow to fetch such size message, but with APISIX cause error--get_format(): body contains : grpc received message larger than max (705561 vs. 4194304)

[spacewander]

How many routes do you have? 8906
How many APISIX/etcd nodes do you have? three node construct a cluster, config in file two
What about their sizes? a former record is to fetch 8M and cause error "received message larger than max "

Maybe you can upgrade the hardware of etcd cluster, see https://etcd.io/docs/v3.4/op-guide/hardware/

As for the "received message larger than max ", there is an issue in etcd: etcd-io/etcd#12576

[azura27]

How many routes do you have? 8906
How many APISIX/etcd nodes do you have? three node construct a cluster, config in file two
What about their sizes? a former record is to fetch 8M and cause error "received message larger than max "

Maybe you can upgrade the hardware of etcd cluster, see https://etcd.io/docs/v3.4/op-guide/hardware/

As for the "received message larger than max ", there is an issue in etcd: etcd-io/etcd#12576

Thanks, we have a look. Is there a schedule for optimization of apisix, pull less data or compress? Because the resource cost is limited in Company sometimes

[spacewander]

We have plan to optimize the etcd operation, but currently we don't have enough people to work on it, so it can't be done in a few months.

[azura27]

We have plan to optimize the etcd operation, but currently we don't have enough people to work on it, so it can't be done in a few months.

Is there any direction for reference, first time in making research on etcd

[spacewander]

Fixed by etcd-io/etcd#13077 which is included in the https://github.com/etcd-io/etcd/releases/tag/v3.5.0