Google Cloud Worker

[martijnvdbrug]

This discussion started on slack, but moved to GH discussion to preserve this info :)

Martijn 3 days ago
Hi all. I'm trying to implement the worker using Google Cloud TaskQueue. It works like this: Put a message in the queue, TaskQueue will push the message to a given endpoint. How would I accept tasks/jobs coming in from an endpoint? It looks like the current JobQueueStrategey only supports polling (by starting/listening on a queue)

Michael Bromley 3 days ago
Did you already take a look at this: https://github.com/vendure-ecommerce/vendure/blob/master/packages/job-queue-plugin/src/pub-sub/pub-sub-job-queue-strategy.ts ?
The JobQueueStrategy is not restricted to polling - the above example is push (subscription) based.

Martijn 1 day ago
Ibthink this example uses pull subscriptions, not entirely sure. I will look into it 👍

Gabriele 1 day ago
Hi @martijn, I have spent a lot of time trying to deploy Vendure to Google Cloud, using Cloud Run for Server & Worker. It doesn’t work: jobs are hanging indefinitely and the same for assets upload (which I have adapted for Google Cloud Storage through a plugin). Even disabling Cloud Run’s downscale to zero (i.e. setting minimum container scale to 1) doesn’t change this. Maybe you have had similar experience? I was thinking of switching to Kubernetes, but deploying the server to Cloud Run and implementing an alternative job queue implementation seems interesting. Though I am afraid server implementation has problems too, because asset upload is performed by the server, as Michael confirmed, and it doesn’t work either.
If you can share more on your experience with Google Cloud it could be beneficial for me! Thanks!!

Gabriele 1 day ago
I also tag @s3m3n as I found some of his previous messages regarding Google Cloud, maybe he has some more experience to share on this. Thanks!

Martijn 3 hours ago
@gabriele Thanks for your response! I did get the asset upload to work with Cloud Run, I can share a code sample with you if needed. I am having the same issues with the worker on Cloud Run: everything in Cloud Run should be handled within a request context, everything outside of the request context is not guaranteed to finish, even with a min instance of 1.
I have the feeling that pull subscriptions don't entirely work because of this, so I am looking into Task Queue.

Michael Bromley 2 hours ago
Guys this knowledge is very valuable. It would be amazing if someone could write up their findings somewhere public e.g a gist GH discussion thread and I can later incorporate it into the docs. Does not have to be really polished, just the key points that will save other devs lots of time and research. (edited)

discodancer 1 hour ago
You can try this tool - it's in beta still, but you might get access: https://docs.github.com/en/discussions

Gabriele 6 minutes ago
@martijn thanks, I am trying to move to Google Kubernetes Engine, but a code sample for asset management will be useful, I have implemented a plugin and it reads correctly, but upload methods never get called! @michael Bromley as soon as I have a Google cloud instance working I want to write a blog post about it and also share some code. I am also using Pulumi infrastructure as code framework, which is quite interesting

[martijnvdbrug]

@gabriele this is the Asset Storage strategy I use, and works for me https://github.com/martijnvdbrug/pinelab-vendure-plugins/blob/master/packages/vendure-plugin-google-storage-assets/src/google-storage-strategy.ts

[martijnvdbrug]

A bit outdated, but nevertheless https://martijn-brug.medium.com/vendure-ecommerce-on-google-cloud-run-225b26c8133a

[martijnvdbrug]

Investigating Task Queue in Cloud Run , as it pushes messages from the queue to an endpoint, everything works within a request context. I have to investigate a bit further on how this fits in the current jobqueuestrategy though

[s3m3n]

This might be the problem. Each task spawns from the beginning and in Vendure you must bootstrap whole project to make simple few lines logic. We were checking possibilities here. Hope you can find solution so we can turn off our VM for worker only.

[coderjib]

@s3m3n Could you please share some details on how you implemented only the worker on K8s, as it now (Vendure v. 1.1.0) comes together with the server? Like @martijnvdbrug I am trying to work on the "Cloud Run only" path, as K8s is definitely more complicated (also in terms of IAM and authentication against other GCP resources), but I am not sure that I will come to a solution. Thanks

[s3m3n]

We would love to run with Cloud Run only, but I think it's impossible, because of worker.

as it now (Vendure v. 1.1.0) comes together with the server

I don't know what you mean? We run backend on Cloud Run and worker on individual VM. Both connect to the same Cloud SQL instance, use same Docker image, they just use different init scripts to run backend / worker.

[coderjib]

@martijnvdbrug thanks a lot (!!!) for sharing your gcp-asset-storage plugin, with it now that part is working, so there was some issue in my implementation of the plugin
I have now implemented the pubsub plugin suggested by Michael, could it solve our issues with the worker?

@michaelbromley the plugin is correctly sending jobs messages to the PubSub topic, but the subscription part (i.e. receiving the message and performing the job) is not happening at all. I think I have to manually set a push endpoint on the subscription configuration (I didn't because I thought it used a pull approach). Am I right? In this case, which endpoint should I set to let Vendure actually receive messages?
Thanks!

[michaelbromley]

I'm afraid I don't actually have experience using the PubSub strategy - it was created by someone else (they are using it successfully in prod) but I don't have specific instructions on how to use. That's part of the reason it is not published as a package right now.

[coderjib]

So, after some work it seems that the pubsub plugin is kind of working (still not sure the whole jobs flow is correctly performed, though). But, also based on the brief conversation with @s3m3n above, I think I'd need some clarifications on the new worker approach.
Question 1: is still an encouraged approach for production to run the Worker in a separate machine/container than the server or both are now meant to be run together (with yarn start)?
Question 2: in case the Worker is still meant to be run separately, because it doesn't listen anymore for incoming traffic (right?), Cloud Run fails to do its health check for it. So it fails. Then, @michaelbromley , I just ask you to confirm that I can partially restore the previous behaviour with no negative side effects, i.e. still add the listening on port xxx (e.g. 3020), just for the purpose of responding to a health check request, that seems mandatory on Cloud Run (i.e., its own control system needs to always be aware of the status/readiness of each container...).

Final note for those working on Google Cloud Run: I am insisting on it because I am forcing it to have a minimum running container of 1, and so for what I understand it shouldn't ever scale down to zero and thus it should behave quite similarly to a K8s cluster (or at least I don't see why it shouldn't).
In any case I will come back to you reporting any relevant findings

[michaelbromley]

1. It does not need to be on a separate machine/container, but if not it is bound by the resource requirements on the server. For many shops this is probably totally fine. I would not start the server nor the worker via yarn/npm though, I would either run them directly with node dist/index.js or use something like pm2 which runs the same command.
2. It might be a good idea to implement an HTTP health check a standard on the worker. I would just need to see how that fits in with the standalone application method that the worker now uses, which does not include a network layer.

[coderjib]

Good morning, I have now tested my implementation, and it is actually working, at least for apply-collection-filters and search index job queues (still have to test email jobs). Thanks a lot for all your hints!
These are the key points of my implementation, in case someone wants to try it:

• Single Cloud Run instance for worker and server
• Cloud Run set to never scale down to zero (metadata.annotations["autoscaling.knative.dev/minScale"] set to 1)
• Google PubSub configured with a topic and subscription for each of the 3 job queues, all passed to Vendure (and then specifically to the PubSub plugin) as env variables
• Cloud Run service account authorized to these roles, in order to "talk" to PubSub: roles/pubsub.publisher, roles/pubsub.viewer, roles/pubsub.subscriber

The only thing that seems not to work is that jobs, while actually being executed (tested both by checking worker debug logs and actual db modifications) are not listed at all in the admin area. @michaelbromley any idea of why Jobs, while being present in the db, are not retrieved at all?

Please find here the code of my Cloud Run VendureServer service, implemented with Pulumi infrastructure as code:
link

Of course I will test this further before moving my production store to GCP.

[michaelbromley]

The jobs do not show in the Admin UI because the pubsub strategy does not implement the InspectableJobQueueStrategy interface. I think this is because of limitations of the pubsub API - it does not enable querying specific jobs.

[coderjib]

Edit: after further testing, I can say it is not working reliably, i.e. not always; depending on the status of the Cloud Run service, sometimes PubSub messages are not received. Investigating (see conversation below) the use of a fully push subscription (i.e. with Vendure exposing an endpoint dedicated to jobs to perform)

[martijnvdbrug]

A little update:

I am working on a plugin which implements Google Cloud Tasks as JobQueueStrategy for Vendure https://github.com/martijnvdbrug/pinelab-vendure-plugins/tree/feature/pubsub-push/packages/vendure-plugin-google-cloud-tasks

Cloud Tasks plugin: https://www.npmjs.com/package/vendure-plugin-google-cloud-tasks

This will run on Google Cloud Run/appengine, because jobs are pushed to an endpoint, processing jobs within a request context. Keep in mind that the request timeout for HTTP targets for Google Tasks is 30 minutes (should be enough per job, in most cases)

[coderjib]

I have double checked both Google Cloud PubSub and Cloud Tasks services, and both push models only talk about an HTTPS endpoint. I will add a comment in the issue to mention this use case, besides the health check endpoint

[coderjib]

Update: Martijn's plugin now works for me in a production environment with server+worker running in the same thread on a Cloud Run Service. Thanks again, will implement a full deployment with this approach for now, even if definitely sub-optimal, and keep an eye on issue #994

[martijnvdbrug]

Good to hear you got it to work. I was already afraid it wouldnt work with seperated worker..
Let me know if you need more info/help. Thanks for testing 🙂

[martijnvdbrug]

@michaelbromley i am not aware of any other way to invoke google cloud run besides http endpoints. As far as i am awary everything needs to happen in request context, or has to be moved to a different google cloud product.

[michaelbromley]

@martijnvdbrug thanks, I am going to look into the best way to add a light HTTP layer to the worker for the next minor version I think.

[martijnvdbrug]

ℹ️ Workaround untill the worker supports incoming HTTP calls:

This version supports Google Cloud tasks with a seperate Vendure worker instance: https://www.npmjs.com/package/vendure-plugin-google-cloud-tasks/v/1.2.0

** Please note that the current worker does not listen for incoming HTTP calls, so you will need to start the whole Vendure app, not just the worker **

1. Start the worker instance with this config:

bootstrap(config)
  .then(async (app) => {
    await app.get(JobQueueService).start();
    Logger.info(`\x1b[46mUsing database ${process.env.DATABASE_NAME} \x1b[0m`);
  })
  .catch((err) => {
    Logger.error(err);
  });

⚠️ This also starts the Graphql API's and the admin UI in your worker instance
Let's say the worker is now running on worker.yourhost.io

2. Start your user-facing Vendure instance like you would normally do and pass the Cloud Tasks plugin CloudTasksPlugin.init({ taskHandlerHost: 'https://worker.yourhost.io' ...