Skip to content
This repository has been archived by the owner on Sep 12, 2023. It is now read-only.

feature: abstract the async queue #14

Closed
peterschutt opened this issue Oct 24, 2022 · 2 comments · Fixed by #83
Closed

feature: abstract the async queue #14

peterschutt opened this issue Oct 24, 2022 · 2 comments · Fixed by #83

Comments

@peterschutt
Copy link
Member

peterschutt commented Oct 24, 2022

Thinking of having a visitor method on the service object, so when an operation is performed on the resource, the object and operation type are put on the async queue. The worker instantiates the service object instance and passes the object and operation to the visit method. This way we can keep all the business logic in the service object and keep the configuration of the queue and worker minimal.

As this stuff is all intra-application, thinking about using pickle for serializing the model instance so that we don't need to serialize to json and deserialize back to sqlalchemy model. Should really do some profiling on this before doing it though as if not a big win then prob not worth the WTF factor.

@peterschutt
Copy link
Member Author

peterschutt commented Oct 25, 2022

To Do:

  • profile pickle vs json for serialization.
  • build the worker function, it will need to import the service object, instantiate it, and call the callback method.
  • build the enqueue mechanism, it will need to retrieve and serialize all data required by worker.
  • documentation

@peterschutt
Copy link
Member Author

Profiling here: https://gist.github.com/peterschutt/7530bc939ff5c268b2b8139e7d74596e

I'm not going to use pickle, artifact is 3x size of using json and speedup per serialize/deserialize iteration was minimal.

peterschutt added a commit that referenced this issue Nov 5, 2022
peterschutt added a commit that referenced this issue Nov 5, 2022
peterschutt added a commit that referenced this issue Nov 5, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant