I suppose this happens due to max_in_flinght=1 per connection:

https://github.com/list-family/ansq/blob/71a8a73a385613532607707a78727ff1cab09582/ansq/tcp/reader.py#L128-L135C7

Feel free to send a PR with set_max_in_flight() enhancement. Relevant code is here:

https://github.com/list-family/ansq/blob/71a8a73a385613532607707a78727ff1cab09582/ansq/tcp/reader.py#L137C35-L144

As a naive implementation we could call conn.set_max_in_flight(N) for each connections on reader.set_max_in_flight(N). What do you think?

Actually, I'm not very familiar with asynchronous programming. I feel that ansq is already the easiest to use asynchronous nsq package in python so far. But if the consumption efficiency of ansq is so weak, basically, I can't use it in production environment.
By the way, I noticed that you updated ansq to v0.2.1 yesterday, what's new in the current version?

Temporary workaround is to increase manually RDY via:

for conn in reader.connnections:
    await conn.rdy(2500)

By the way, I noticed that you updated ansq to v0.2.1 yesterday, what's new in the current version?

See the releases page for details.

Actually, I'm not very familiar with asynchronous programming. I feel that ansq is already the easiest to use asynchronous nsq package in python so far. But if the consumption efficiency of ansq is so weak, basically, I can't use it in production environment.

Thanks anyway for the issue. Much appreciated. I'll try to come up with a naive solution later on my spare time.

Thanks for your work. Good luck!

Here's what i've found in some of my tests:

The higher RDY does indeed imply higher throughput, though, mind it got saturated at RDY ~ 64-128. Of course the actual point of saturation will depend on the scenario.

@magniff nice demonstration! Yeah, RDY is a key concept that tunes the performance and throughput of a client, see docs.

For the data protocol, we made a key design decision that maximizes performance and throughput by pushing data to the client instead of waiting for it to pull. This concept, which we call RDY state, is essentially a form of client-side flow control.

@rxy1212 could you check #101? Does it work for you?

pip install 'git+https://github.com/atugushev/ansq@set-max-in-flight#egg=ansq'

@atugushev I don't know what's wrong, but I can't install ansq with "pip install 'git+https://github.com/atugushev/ansq@set-max-in-flight#egg=ansq'"

@rxy1212 what's the error?

pip install 'git+https://github.com/atugushev/ansq@set-max-in-flight#egg=ansq'
Collecting ansq
  Cloning https://github.com/atugushev/ansq (to revision set-max-in-flight) to /tmp/pip-install-2nnxqqju/ansq_55a7f5e47b9549909853031f14db2476
  Running command git clone --filter=blob:none --quiet https://github.com/atugushev/ansq /tmp/pip-install-2nnxqqju/ansq_55a7f5e47b9549909853031f14db2476
  fatal: unable to access 'https://github.com/atugushev/ansq/': Encountered end of file
  error: subprocess-exited-with-error

  × git clone --filter=blob:none --quiet https://github.com/atugushev/ansq /tmp/pip-install-2nnxqqju/ansq_55a7f5e47b9549909853031f14db2476 did not run successfully.
  │ exit code: 128
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://github.com/atugushev/ansq /tmp/pip-install-2nnxqqju/ansq_55a7f5e47b9549909853031f14db2476 did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

But general packages installed with pip work well

@rxy1212 does git clone https://github.com/atugushev/ansq work for you?

@atugushev I got this output

Cloning into 'ansq'...
fatal: unable to access 'https://github.com/atugushev/ansq/': Empty reply from server

@rxy1212 seems like you need a VPN.