Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework shared memory #54

Open
4 of 7 tasks
pietroalbini opened this issue Apr 10, 2016 · 0 comments
Open
4 of 7 tasks

Rework shared memory #54

pietroalbini opened this issue Apr 10, 2016 · 0 comments

Comments

@pietroalbini
Copy link
Contributor

pietroalbini commented Apr 10, 2016

The current status of shared memory is a real mess, and I really need to solve it:

First of all, you can't easily use sub-dictionaries because they're not synchronized, and you need to do something like this, which is really ugly:

mydata = shared["mydata"]
mydata["test"] = "abc"
shared["mydata"] = mydata

Other than that, shared memory is directly available via the shared argument of any hook: this might seem convenient, but prevents adding more shared things, like a global memory (as requested in #22) and locks, which are currently implemented by adding the method at runtime.

In order to solve this problems, and allowing to create custom drivers more easily, a big rewrite of the whole thing is needed. The new API I want for it is:

# For a component's memory
shared.memory["mydata"] = shared.object("dict")
shared.memory["mydata"]["test"] = "yay"
with shared.lock("hello"):
    pass

# A custom bucket
bucket = bot.shared.bucket("test")
bucket.memory["mydata"] = "test

  • Move the functionality of shared into shared.memory
  • Use a custom proxy instead of the multiprocessing one, so it's easier to write custom drivers and the API is consistent between them
    • Add basic support for objects management
    • Add ability to export/import objects and switch drivers
    • Add ability to add memory preparers
    • Add support for the runner driver
  • Add some sort of garbage collection
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

1 participant