Robustness test kubernetes traffic sometimes doesn't issue any deletes

[serathius]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Number of keys keeps growing to absurd numbers, due to no delete requests.

What did you expect to happen?

Average of keys to be kept around 10 keys, the intention of the current logic was:

• Issue only creates if number of keys drops below 5
• Issue only deletes if number of keys is over 15

Expect it might a bug with nonUniqueWriteLimiter.

How can we reproduce it (as minimally and precisely as possible)?

n/a

Anything else we need to know?

No response

Etcd version (please run commands below)

main branch

Etcd configuration (command line flags or environment variables)

TestRobustnessExploratory_Kubernetes_HighTraffic_ClusterOfSize3 test

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[henrybear327]

@serathius I can try to tackle this issue!

/assign @henrybear327

[fuweid]

I try to fix #17780 with more deletion.
I think the root cause is that we only return token only for deletion.

etcd/tests/robustness/traffic/etcd.go

Lines 129 to 131 in ce2dbbb

	if requestType == Delete || requestType == LeaseRevoke {
	nonUniqueWriteLimiter.Return()
	}

When non-deletion or non-lease-revoke has more weight, they won't return the token from limiter.
We should return all if we take it from limiter.

cc @henrybear327 @serathius

[fuweid]

ping @henrybear327 if you don't have time, may I take this one? since it's related to #18099 issue. Thanks

[henrybear327]

ping @henrybear327 if you don't have time, may I take this one? since it's related to #18099 issue. Thanks

Hello @fuweid, I have unassigned myself. There was nothing specific to handover from my side.

/assign @fuweid

[fuweid]

closed by #18135

[serathius]

Awesome work @fuweid !