feat: fork tendermint client creator to NOT use mutex

[kjessec]

• Default tendermint client uses RWMutex
• This disallows concurrently executing sync & query
• Disable RWMutex to see if:
  • Performance is better
  • Is safe

[jeffwoooo]

구동 중 panic 되어 확인 결과

• app client 가 service.Service interface 도 구현해야 함
• abcicli.Client interface를 composite 만 하고 구현이 없음

두 이슈 해결 중에 local_client.go 를 통째로 가져오게 되었습니다. (localClient가 private 하여 구현을 가져올 수가 없음ㅜㅜ)

해결 후 쿼리 실행까지는 잘되고 performance와 sync 중 안정성을 테스트해보겠습니다.

[jeffwoooo]

did some apache benchmark test between main branch(with mutex) and this branch(without mutex),
and it seems there is no performance improvement.

main branch with mutex - a2d6343
this branch without mutex - edc40db

Document Path: /staking/validators/terravaloper1259cmu5zyklsdkmgstxhwqpe0utfe5hhyty0at/delegations?height=5354666
Document Length: 40452 bytes

1. Concurrency Level: 1 & Total requests: 10

with mutex
Time taken for tests:   48.678 seconds
Requests per second:    0.21 [#/sec] (mean)
Time per request:       4867.779 [ms] (mean)
Time per request:       4867.779 [ms] (mean, across all concurrent requests)
Transfer rate:          8.18 [Kbytes/sec] received

without mutex
Time taken for tests:   48.794 seconds
Requests per second:    0.20 [#/sec] (mean)
Time per request:       4879.447 [ms] (mean)
Time per request:       4879.447 [ms] (mean, across all concurrent requests)
Transfer rate:          8.16 [Kbytes/sec] received

2. Concurrency Level: 10 & Total requests: 50

with mutex
Time taken for tests:   57.891 seconds
Requests per second:    0.86 [#/sec] (mean)
Time per request:       11578.123 [ms] (mean)
Time per request:       1157.812 [ms] (mean, across all concurrent requests)
Transfer rate:          34.40 [Kbytes/sec] received

without mutex
Time taken for tests:   55.611 seconds
Requests per second:    0.90 [#/sec] (mean)
Time per request:       22244.443 [ms] (mean)
Time per request:       1112.222 [ms] (mean, across all concurrent requests)
Transfer rate:          35.81 [Kbytes/sec] received

3. Concurrency Level: 20 & Total requests: 50

with mutex
Time taken for tests:   55.539 seconds
Requests per second:    0.90 [#/sec] (mean)
Time per request:       22215.505 [ms] (mean)
Time per request:       1110.775 [ms] (mean, across all concurrent requests)
Transfer rate:          35.85 [Kbytes/sec] received

without mutex
Time taken for tests:   55.611 seconds
Requests per second:    0.90 [#/sec] (mean)
Time per request:       22244.443 [ms] (mean)
Time per request:       1112.222 [ms] (mean, across all concurrent requests)
Transfer rate:          35.81 [Kbytes/sec] received

[kjessec]

@jeffwoooo the PR was more about allowing block injection while a query takes longer than a block interval

[jeffwoooo]

thank you for the reply. that is right.
so now, I want to check if a long query slow down block time.

I am testing with a request to /staking/validators/terravaloper1p54hc4yy2ajg67j645dn73w3378j6k05vmx9r9/delegations\?height=5354666
but the request does not seem to slow block time even in main branch since it took only about 7 seconds.

is there a query that take much longer than 7 seconds?

[kjessec]

@jeffwoooo let's try terravaloper15zcjduavxc5mkp8qcqs9eyhwlqwdlrzy6jln3m instead, with differing heights.

[jeffwoooo]

@kjessec Hi!

While testing with queries, I wondered why long query doesn't slow down syncing and finally figured out that block excutor(=> sync) is not using same mutex with query client.

Multi app conn create 4 abci clients who have each one’s mutex, and assign them for each Mempool, Consensus, Query and Snapshot connection.

So, long query would not slow down syncing by locking at least.

Would you please check following and confirm that?

https://github.com/tendermint/tendermint/blob/849461aab254014f6b7976710400ea59f83265ce/proxy/multi_app_conn.go#L85-L121

[kjessec]

@jeffwoooo oh this may be the case for sure. Hmm weird -- I thought we observed a case where the entire sync process is blocked during long queries.

We should close this PR maybe?

[jeffwoooo]

Yes, I agree to close it.

I thought that I observed that case too.
But I'm sure it was caused by other issue that we don't know yet.

Let's open another issue again if we watch that case again.

[jeffwoooo]

turned out the 4 local clients use same mutex given from local client creator.
reopening it.

[jeffwoooo]

i want to test running sync under heavy load.

[jeffwoooo]

it still slowing down syncing. i suspect db-side locking.