Request to share selective memories from main component to other components

[patternspandemic]

Hi Pietro,

Might it be possible to allow access to certain memories set directly on the bot's main component from all or some other components? Such is the case when some number of components require the same shared resource like a DB connection. I suppose I could try sub-classing a component with a memory of a DB connection, but I'm not sure the base class' shared memory will translate to its sub-classes.

I think a builtin method that doesn't require sub-classing would be more useful and easier to use anyhow. Perhaps if nothing else a simple way to allow a component to access to the main_component's shared memory. Thoughts?

Thank you,
Brad

[pietroalbini]

Subclassing won't work, because components (and bots) have unique, random IDs assigned during initialization. This was made to prevent conflicts between components, but this also kills your use case as a side effect.

The hacky way to fix this issue is manually editing the Component._component_id right after you create the bot (or in the __init__).

I'll think about an API for this, but I don't think this will be implemented before a refactor of the shared memory I plan to do in the future (converting it to a drivers-powered thing, in order to provide a standardized API and different storage engines).

[patternspandemic]

Thanks for the info. I believe I've found a solution that will work for me given the backend I plan to use.

[pietroalbini]

I thought about it a bit, and I think I found a good API for this:

class MyComponent(botogram.Component):
    component_name = "test"
    component_isolated = False

That component_isolated attribute specifies if the component's resources should be isolated from the other components (currently only the shared memory), with True as the default (as it is right now). If the argument is set to False, the component shares the same resources as the main component.

I am, as always, open for feedback :)

[patternspandemic]

I like this. I'd assume any non-isolated component could still initialize a memory (not just the bot/main component). That way everything's still handled by the responsible component.

With this, one could also easily setup something like component dependencies. A BackendComponent can initialize a shared memory holding a connection to the backend. Then any number of other components expecting such a connection in the shared memory can simply depend on the BackendComponent. Maybe a check can be made from Bot.use.

Whether this fits in or is non-limiting to your future plans for shared memories, I'm not sure.

[pietroalbini]

I like this. I'd assume any non-isolated component could still initialize a memory (not just the bot/main component). That way everything's still handled by the responsible component.

Yeah, you should be able to initialize the memory from any non-isolated components.

With this, one could also easily setup something like component dependencies. A BackendComponent can initialize a shared memory holding a connection to the backend. Then any number of other components expecting such a connection in the shared memory can simply depend on the BackendComponent. Maybe a check can be made from Bot.use.

Maybe in the future. The problem with this is, there are no way to reference another component:

• The name of the component can't be enforced to be unique
• Moving around Components doesn't work on Windows (everytime the same issue)
• Random UUIDs are random (doh!)

Another possible API, which allows to use multiple shared shared memories:

class MyComponent(botogram.Component):
    component_name = "test"
    component_memory = "backend"

[patternspandemic]

there are no way to reference another component: ...

I see. Then having multiple named shared memories seems like a simple solution that requires only a little more thought when putting dependent components together. Any components that need to work together simply use the same shared memory space.

Should components be able to utilize multiple SharedMemory spaces? I.e.

component_memories = [
    'default',  # memories initialized from the main component (bot)
    'backend'  # memories initialized from a BackendComponent
]

How would memory initialization then work? It seems component should be able to initialize memories to any space.

# Initialize a memory into the "default" space
@bot.init_shared_memory(space="default")
def default_memory(shared)
    shared["example"] = 1

# Within a component, initialize a memory to the "backend" space
self.add_shared_memory_initializer("backend", connection_memory)

When accessing shared memories, perhaps the shared argument is then a dict of named SharedMemory spaces the component subscribed to? Or maybe just an object where the named spaces are attributes...

@bot.command("example")
def example_command(shared):
    num = shared["default"]["example"]  # or
    num = shared.default["example"]

I like the solution of named SharedMemory spaces, with the ability to initialize memories to any space from any component.

[pietroalbini]

Sorry for the delayed response! Had a busy week.

I'm not fully sure about multiple shared memories for every component. The first example breaks every current code, and the second one might break existing methods (it's a dict-like object). I'll think about this later.

[patternspandemic]

Another (perhaps common) use case I've come across where inter-component memory sharing is required is with OAuth type scenarios.

When looking to use various Web APIs which require authorization, access tokens are generally short lived, or can be revoked for a number of reasons. Once you have to renew access tokens, there's no way to provide the new tokens to all components that may need them.

By the way, congrats on releasing botogram to the wider world! Nice work.

[patternspandemic]

FYI, if anyone else is manually setting component_id on components to achieve inter-component memory sharing, note that only one component may initialize (prepare) memories. Namely, the first component to register memory initializers with the used ID.

[pietroalbini]

I'll add a new shared.global_memory as part of a big refactor of the whole shared memory.

I don't think messing around with other components' memories is a good idea, because there is no dependency management and a component should be able to manage its memory without other components playing around with it.

[patternspandemic]

I don't think messing around with other components' memories is a good idea, because there is no dependency management and a component should be able to manage its memory without other components playing around with it.

Sounds reasonable. With the shared.global_memory what will be the default way to handle collisions when components' prepared memories conflict? Looks like a custom driver could be used to customize the behavior, for instance to keep a history, or bag of values tied to their components. Basically whatever makes sense for your needs.

[pietroalbini]

With the shared.global_memory what will be the default way to handle collisions when components' prepared memories conflict?

Prefixing every item with your component's name? If you want to avoid conflicts there is the component memory.

[pietroalbini]

The refactor planned in #54 should also allow you to create custom memories.

[pietroalbini]

bucket = bot.shared.get_bucket("backend")
bucket.memory["a"] = "b"

Would this be enough @patternspandemic?

[patternspandemic]

Sure. Looks to be a global memory with named access to 'buckets' for organization? Works for me, I think as long as any component can initialize memories to any bucket.

[pietroalbini]

Uh, yeah, I forgot to explain what buckets are!

I'm implementing the new shared state (uh, new name) from scratch (with a lot of nice things), and I'm designing it to be easily customizable (you will be able to easily create drivers, for example backed by a database or redis) and better organized.

Buckets are now the groups of shared things (memories or locks): each component/bot combination has its own bucket as before, there will maybe be a global bucket (I'm deciding if it's necessary now) and you will be able to create new buckets as shown in the previous comment.

This means buckets you create will have the exact functionality of the ones you get on your hooks, because they will be the same thing after all. With a bit of hackery you will also be able to access buckets of other components (those buckets are named {uuid_of_bot}:{uuid_of_component}).

Another thing is, you will be able to create shared objects detached to the main bucket.memory of a bucket:

shared.memory[chat.id] = shared.object("dict")
shared.memory[chat.id]["action"] = "I'm finally synchronized!"

EDIT: this probably won't be the final API.

[patternspandemic]

I see how a global bucket wouldn't be necessary if any component could ask for a named bucket from shared by name. I like the idea of naming it clearly for it's use.

I assume prepared memories are still part of buckets, and that a component prepares memories to its own bucket. I guess this would mean the shared argument to a memory preparer defaults to the component's bucket. What about preparing memories to a named bucket? Perhaps one just gets the bucket from the same shared argument to the hook?

@prepare_memory
def prepare_some_memories(shared):
    # Prep a memory to this component's bucket
    shared.memory["count"] = 0
    # Request a named bucket, and add a memory to it.
    my_bucket = shared.get_bucket("my_bucket")
    my_bucket.memory["other"] = "Custom bucket memory!"

If one can get a named bucket from the shared argument in any hook, this setup looks quite useful!

I think I need to see more examples of the detached shared.object functionality to grasp its significance. Is it just that one doesn't have to reassign it back to the shared memory? That is of course nice not to have to remember.

Good work Pietro

[pietroalbini]

I assume prepared memories are still part of buckets, and that a component prepares memories to its own bucket.

Yes, I don't want to take (useful) stuff away :)

I guess this would mean the shared argument to a memory preparer defaults to the component's bucket.

The shared argument you get in hooks is the component bucket, even in this case.

What about preparing memories to a named bucket? Perhaps one just gets the bucket from the same shared argument to the hook?

I need to think more about an API for this. The example you posted won't work most of the times, because preparers are called when they're needed (they must not be considered as an on_start hook), so my_bucket wouldn't be initialized in some cases. If you have any ideas for this let me know!

If one can get a named bucket from the shared argument in any hook, this setup looks quite useful!

You get a named bucket from the bot.shared.get_bucket (or maybe .bucket) method. bot.shared will be the manager of all the buckets (currently it's bot._shared_memory -- you shouldn't use that).

I think I need to see more examples of the detached shared.object functionality to grasp its significance. Is it just that one doesn't have to reassign it back to the shared memory? That is of course nice not to have to remember.

The "I don't want to reassing" was one of the most important reasons I started this rewrite: it's really ugly to type, and it leds to race conditions really easily (you should put a lock around each of this operation). One of the other nice things is, you can create any synchronized objects botogram supports: currently dict and lock, but I want to add list and maybe set (and it's relatively easy to create your own ones -- example of the proxy and driver support for the dict).