Major slowdowns writing to tile38

[TheCycoONE]

Describe the bug
We are experiencing significant slow downs in our write operations. For the past 3 days we've been seeing write speeds for points of approximately 10-30/s, down from 10,000 or more per second typically. We've experienced these slow downs a few times in the past. We expect but cannot yet confirm that it may be related to replication.

We are running 6 instances of tile38, split between two geographically separated datacenters with ~94ms round trip ping time between the two datacenters, with one instance as the leader and all others as replicas. The coordination is managed by redis sentinal.

To Reproduce
This is a behaviour we've observed in production several times, but we do not have steps to reproduce outside of production.

Expected behavior
We expect consistent write speed that can keep up with our load of thousands of points per second.

Logs
Not Applicable

Operating System (please complete the following information):

• OS: [Rocky Linux 8.6]
• CPU: [Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz x4]
• Version: [1.31.0]
• Container: [VMWare ESX]

Additional context
See attached
tile38_slowness_server_cmds.txt

[tidwall]

Sorry, but unable to help without some way to reproduce the issue.

[Kilowhisky]

Are there a large number of points at the same GPS location? I've found that if you reuse the same static point or have tons of points at the same location it will slow things down substantially as the index's get huge.