-
Notifications
You must be signed in to change notification settings - Fork 227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pubsub-emulator throw error when publish 50k messages. #992
Comments
@mrdulin Thanks for the detailed write-up! Just based on the Java error trace at the top, I'm guessing this is something with the Pub/Sub emulator, but I'll try out your repro steps. |
@feywind ok. Thanks. But I also hit this issue in the production environment. If I publish 50K messages without using const n = 50 * 1000;
for (let i = 0; i < n; i++) {
const data = `message payload ${i}`;
const dataBuffer = Buffer.from(data);
await topic.publish(dataBuffer)
}
// some logic ...
res.json(data); Any suggestions about how to publish a huge amount of messages without blocking the execution of subsequent logic? Do I need to use |
Realistically you should be able to just fire and forget (unless the Node process is terminated or something) so it does kind of seem like there's some library issue there. We're going to be pulling in some much newer versions of the pieces under the Pub/Sub library soon with the 2.0 release of the Pub/Sub library. I'm hoping that will help with some of these issues. I think the plan is to release that this week. |
|
@feywind ok. thanks. I will try it. |
@mrdulin Just checking in to see if you're still experiencing these issues? |
There hasn't been any movement on this for a while, so I'm going to go ahead and close it. (Please feel free to re-open if you need any other help with it.) |
@stephenplusplus @feywind The issue is gone when I upgraded the client library to |
Added a test case for the new close function
Environment details
uname -a
=>Darwin US_C02WG0GXHV2V 17.7.0 Darwin Kernel Version 17.7.0: Thu Jan 23 07:05:23 PST 2020; root:xnu-4570.71.69~1/RELEASE_X86_64 x86_64
node -v
=>v10.16.2
npm -v
=>6.14.4
@google-cloud/pubsub
version:"@google-cloud/pubsub": "^1.7.3"
Description
When I tried to publish 50k messages, the pubsub-emulator throw below error infinitely:
Error stack
My publisher also throws many same errors:
My subscriber throw below errors:
Actually, I don't know it's a bug or I am using pubsub incorrectly. So I decide to post this as a github issue. I found there are some questions on stackoverflow and issues on github:
But didn't find a solution.
Steps to reproduce
I made a minimal code example for reproducing it. Please tell what's going on? Thanks!
Repo: https://github.com/mrdulin/nodejs-gcp/tree/master/src/pubsub/pubsub-emulator
gcloud beta emulators pubsub start --project=$PROJECT_ID
Additional information
I also got the error:
Retry total timeout exceeded before any response was received
in production environments. So it may not be a problem of pubsub-emulator@google/pubsub
version:"@google-cloud/pubsub": "^1.6.0"
That's why I am trying to make an example to reproduce it. But can't figure out what's going on.
The only way I can reproduce this error now is to post 50k messages.
Test cases: https://gist.github.com/mrdulin/79f1689a9baaafaef90fcad42646bf6d
The text was updated successfully, but these errors were encountered: