.Net: SK Agents

[SergeyMenshykh]

Motivation and Context

This ADR outlines high-level ideas/design for SK agents.

[crickman]

I am also curious if "user-proxy" agent might need to be addressed to fully support agent collaboration/chat patterns?

[crickman]

FWIW - ADR "31" is already overloaded

[crickman]

Perhaps we can consider the upcoming Azure support: Azure/azure-sdk-for-net#40817

[joslat]

I am also curious if "user-proxy" agent might need to be addressed to fully support agent collaboration/chat patterns?

Continuing on @crickman reply, adding some thoughts:

Why not implementing all the Autogen agent types? as in https://github.com/microsoft/autogen/pull/924/files#diff-9860f01d79405521f2af05fe4040ed4f50498c9108956723f4bc5f89d88ab5ad :

• ConversableAgent
• AssistantAgent
• GroupChatManager
• UserProxyAgent...

IMHO this implementation is gold, and a basis for inspiration - the only issue I see is that is bound to OpenAI API.
I like the GroupChat & more streamlined SequentialGroupChat - this is very needed IMHO

But let me get to the core, I mean Kernel ;)
The other point I want to bring is a suggestion on the design which the ADR does not mention: kernel centricity.
There is no central kernel in the agent story. A new one is created on each Agent, that's why we see the configuration and the model passed on the agent builder. And then it is hidden from the developer, it is internal. IMHO this is not good.

Why not pass the "central" kernel on its builder, and then use it? in case that we need parallelized execution - the main reason I believe that triggered this design decision - each agent has its own kernel (heart) - why not create a pool of kernels that are a clone from the main, central kernel (no configuration needed) and then each agent can take the next available, free, kernel/heart? ;)

This way they can execute in parallel and this approach would simplify the architecture (read: reduce code) and increase performance by reusing ready-to-use kernels.

This pool of kernels, could be easily coordinated by the kernel itself, keeping the kernel centricity. Maybe with a thread safe collection of kernels that have a flag if it is used and by who, each with a semaphore to avoid race conditions, et voila... we have a better solution than creating all the kernels beforehand. It also would scale better and be more performant.

And also solve the multi-model compatability (sort of) - as the Kernel is supporting already different AI Models. So we could tell the agent what serviceId to use (as https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/KernelSyntaxExamples/Example62_CustomAIServiceSelector.cs)

Thoughts?

[SergeyMenshykh]

I am also curious if "user-proxy" agent might need to be addressed to fully support agent collaboration/chat patterns?

Yes, it may be necessary. Once the agents' abstraction is in place, we will begin adding concrete implementations for agents, and the "user-proxy" agent could be one of those implementations. There will be a user story to track the "user-proxy" agent for SK.

[SergeyMenshykh]

FWIW - ADR "31" is already overloaded

Yes, thank you. I will change the number before submitting the PR to the main branch.

[SergeyMenshykh]

Perhaps we can consider the upcoming Azure support: Azure/azure-sdk-for-net#40817

Yep, definitely, we should migrate our agent to the new API when it's available. I'll create an issue to track this.

[SergeyMenshykh]

Why not implementing all the Autogen agent types? as in https://github.com/microsoft/autogen/pull/924/files#diff-9860f01d79405521f2af05fe4040ed4f50498c9108956723f4bc5f89d88ab5ad :

• ConversableAgent
• AssistantAgent
• GroupChatManager
• UserProxyAgent...

Once the agent's abstraction is in place, it will be a matter of prioritizing the design and implementation for the agents in a way that aligns with SK's overall approach, including DI, filters, tools, and so on.

[SergeyMenshykh]

But let me get to the core, I mean Kernel ;) The other point I want to bring is a suggestion on the design which the ADR does not mention: kernel centricity. There is no central kernel in the agent story. A new one is created on each Agent, that's why we see the configuration and the model passed on the agent builder. And then it is hidden from the developer, it is internal. IMHO this is not good.

Why not pass the "central" kernel on its builder, and then use it? in case that we need parallelized execution - the main reason I believe that triggered this design decision - each agent has its own kernel (heart) - why not create a pool of agents and then each agent can take the next available, free, kernel? ;)

And that could be easily coordinated by the kernel, with a semaphore, thread-safe et voila... we have a better solution than creating all the kernels beforehand. It also would scale better and be more performant.

And also solve the multi-model compatability (sort of) - as the Kernel is supporting already different AI Models.

Thoughts?

Even though it is not stated in the ADR, and it should have been and will be, every feature we design or add to SK is supposed to be built around the kernel and leverage as much as possible of all the benefits SK provides or supports, such as DI, plugins, function calling, abstractions, etc. Therefore, agents are no exception in this regard.

Regarding passing the kernel to agents, it is a good idea and aligns with the way we plan to implement agents. SK agents should be flexible and accept an instance of a kernel provided by the SK consumer code. Whether it is one instance of the kernel for all agents, or each agent gets its own instance, it is up to the SK consumer code to decide, depending on the scenario it wants to implement.

Kernel/Agents Pool: It might make sense to arrange agents or kernels into a pool if the cost of creating or destroying them is high. For example, if the classes use expensive system resources that can be shared and reused, it would be logical to organize them in a pool to minimize system resource usage. Regardless, the SK consumer code has a few mechanisms to optimize the cost if it is high. A kernel/agent can be registered as a singleton and reused during the hosting app's lifetime to minimize the cost. If the singleton solution is not sufficient, nothing prevents it from pooling kernels or agents if that is what the hosting app requires.

[AshD]

What is the rationale for the Agent class requiring the Assistant APIs?

AutoGen uses the Chat Completion API which is implemented by multiple providers like vLLM, Llama.cpp, etc.

Thanks,
Ash

[joslat]

What is the rationale for the Agent class requiring the Assistant APIs?

AutoGen uses the Chat Completion API which is implemented by multiple providers like vLLM, Llama.cpp, etc.

Thanks, Ash

Hi Ash,
Initially The agents was developed on top of the OpenAI Assistants API, which offers some features like thread management, tools, function calling and files (not supported just yet but in short will - there are two PRs ongoing)

But as the team said, they want to decouple from the OpenAI implementation, and after this "base work is done" - which is what this ADR is about, to provide a base architecture/structure from where to build the agent story for all models (and also OpenAI).
@SergeyMenshykh correct me if I am wrong or not precise enough...

[AshD]

What is the rationale for the Agent class requiring the Assistant APIs?

AutoGen uses the Chat Completion API which is implemented by multiple providers like vLLM, Llama.cpp, etc.

Thanks, Ash

Hi Ash,

Initially The agents was developed on top of the OpenAI Assistants API, which offers some features like thread management, tools, function calling and files (not supported just yet but in short will - there are two PRs ongoing)

But as the team said, they want to decouple from the OpenAI implementation, and after this "base work is done" - which is what this ADR is about, to provide a base architecture/structure from where to build the agent story for all models (and also OpenAI).

@SergeyMenshykh correct me if I am wrong or not precise enough...

Thanks @joslat Looking forward to test this