Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for plugin manifest #1666

Closed
wants to merge 8 commits into from
Closed

Support for plugin manifest #1666

wants to merge 8 commits into from

Conversation

lemillermicrosoft
Copy link
Member

@lemillermicrosoft lemillermicrosoft commented Jun 22, 2023 •
edited
Loading

Motivation and Context

Developers do not always have access to the kernel code or access to change it. Plugin developers will need to be able to create their plugins and deploy those. After they do that, they will use the manifest file with the description in it to store into a centralized location. This location will need to be fast to query and retrieve plugins that will help solve the ask.

In support of #1728

Description

This pull request adds a new feature to the sequential planner that allows users to specify a custom function to get the available functions for planning. This can be useful for scenarios where the default function is not sufficient or where users want to use a different set of functions for different queries. The pull request also adds support for importing a plugin manifest from a URL, which is a JSON file that lists the names and URLs of chat GPT plugins. This can make it easier for users to discover and use multiple plugins without having to import them individually.

Details:

  • Add a new property to SequentialPlannerConfig called GetAvailableFunctionsAsync, which is a delegate that takes a config and a semantic query and returns an ordered enumerable of FunctionView objects. If this property is null, the default function is used.
  • Modify SKContextSequentialPlannerExtensions.GetManualAsync to use the custom function if provided, or the default function otherwise.
  • Add a TODO comment in SequentialPlanner to note that the kernel needs to be able to load functions on-demand if a custom function is used.
  • Add a new extension method to IKernel called ImportPluginManifestFromUrlAsync, which takes a URL of a plugin manifest and imports all the plugins listed in it. The plugin manifest is a JSON file that follows the PluginManifest model, which has a list of RemotePlugin objects, each with a name and a URL.
  • Add a new model class called PluginManifest, which represents the plugin manifest JSON file, and a nested class called RemotePlugin, which represents a single plugin entry.
  • Add a private helper method to KernelChatGptPluginExtensions called ParsePluginJson, which deserializes the plugin manifest JSON file and validates it.

Contribution Checklist

@github-actions github-actions bot added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel labels Jun 22, 2023
@lemillermicrosoft lemillermicrosoft self-assigned this Jun 22, 2023
@lemillermicrosoft lemillermicrosoft force-pushed the 621_plugin_manifest branch 2 times, most recently from 5e3fb13 to 2129485 Compare June 22, 2023 21:46
@lemillermicrosoft lemillermicrosoft force-pushed the 621_plugin_manifest branch 2 times, most recently from dff0cce to 2765b70 Compare June 26, 2023 19:29
@lemillermicrosoft lemillermicrosoft changed the title Add custom function option for sequential planner and support for plugin manifest Support for plugin manifest Jun 27, 2023
@lemillermicrosoft
🛠️ Refactor plan parsing and add GetFunction option
9fce6e8 
This commit refactors the SequentialPlanParser class to simplify the
logic and avoid catching and rethrowing exceptions. It also adds a
GetFunction option to the SequentialPlannerConfig class, which allows
the caller to provide a custom function to get a function by name. This
can be useful for scenarios where the available functions are not
statically defined or need to be resolved dynamically. Additionally,
this commit changes the stop sequence for the plan parsing to be
"<!-- END -->" instead of just "<!--", to avoid ambiguity with other
comments.

The commit also refactors the plan parsing logic to use a function
delegate instead of a SKContext parameter. This allows for more
flexibility and customization in how the skill functions are retrieved.
The function delegate can be provided by the SequentialPlannerConfig or
by a default implementation that uses the SKContext.

The commit also updates the unit tests and the planner extensions to
use the new function delegate parameter. The skprompt.txt file was
updated to clarify some points about the function syntax and the
available functions. The examples were also reorganized and simplified.
@lemillermicrosoft
📝 Remove text and comment nodes from plan steps
9879f80 
This commit changes the SequentialPlanParser to ignore text and comment
nodes when parsing a plan from XML. These nodes are not valid steps and
should not be added to the plan. This also updates the unit tests to
reflect the expected behavior. Additionally, this commit adds some
comments to the skprompt.txt file to clarify the expected format and
validity of the plans.

Squashed commit of the following:

commit eeae9b0
Author: Lee Miller <[email protected]>
Date:   Thu Jun 22 15:26:51 2023 -0700

    🐛 Fix sequential plan parser to ignore non-function tags

    The sequential plan parser was incorrectly adding non-function tags
    as steps in the plan, resulting in extra and invalid steps. This
    commit fixes the parser to skip any tags that are not <function>
    and only add the function steps to the plan. This also updates the
    unit tests to reflect the expected behavior and output of the parser.

commit 0e2e9c1
Author: Lee Miller <[email protected]>
Date:   Thu Jun 22 15:23:14 2023 -0700

    🐛 Fix plan parsing to ignore text and comments

    The plan parser was incorrectly adding text and comment nodes as steps
    in the plan, which could cause errors or unexpected behavior. This
    commit fixes the parser to skip over these nodes and only add function
    nodes as steps. It also adds some TODO comments for possible future
    enhancements to use text or comments as reasoning or desired functions
    for a plan.

commit 5bb555c
Author: Lee Miller <[email protected]>
Date:   Thu Jun 22 14:44:29 2023 -0700

    🛠️ Fix stop sequence for sequential planner

    The stop sequence for the sequential planner was missing the "END" part,
    which could cause problems when parsing the XML plan. This commit fixes
    the stop sequence to include the "END" comment.

    It also simplifies the instructions for creating a plan, removing some
    redundant or outdated information. It adds an example of how to call a
    function with multiple inputs, and how to use XML comments in the input
    values.
@lemillermicrosoft
🔄 Swap parameter order in ToPlanFromXml method
efc062b 
The parameter order in the ToPlanFromXml method was inconsistent with
the rest of the codebase, and could cause confusion for callers. This
commit swaps the order of the allowMissingFunctions and GetFunction
parameters, and updates the XML documentation accordingly. No changes
were made to the method logic or functionality.
@lemillermicrosoft
📝 Update SequentialPlanner to use AzureChatCompletionService and lowe…
cd89d95 
…r relevancy threshold

This commit updates the SequentialPlanner to use the
AzureChatCompletionService instead of the AzureTextCompletionService,
which is more suitable for generating natural language responses. It
also lowers the relevancy threshold for the planner to 0.5, which may
improve the quality of the plans.

Additionally, this commit updates the skprompt.txt file to include the
available functions in the prompt, and to clarify the steps for creating
a plan and calling a function. It also adds some examples of valid and
invalid function calls.

Finally, this commit removes a redundant EmailSkillFake import in the
SequentialPlannerTests class, and adds a fake EmailSkill to simulate the
use of a custom skill in the planning process. No functional changes
were made to the tests or the planner.
@lemillermicrosoft
👌 Simplify and rename function-related code
7ea8479 
- Use ternary operator to simplify GetAvailableFunctionsAsync logic
- Rename GetFunction to GetSkillFunction for clarity and consistency
- Update references to GetFunction in SequentialPlanParser and SequentialPlanner
- Rename GetFunction property in SequentialPlannerConfig to GetSkillFunction
@lemillermicrosoft
🛠️ Rename GetFunction to GetSkillFunction in SequentialPlanParser
7a847f6 
This commit renames the GetFunction method in the SequentialPlanParser
class to GetSkillFunction, to better reflect its purpose and avoid
confusion with other methods that also deal with functions. The method
is used to create a delegate that takes a skill name and returns a
corresponding function object. All the references and tests that use
this method are also updated accordingly. This change improves the
readability and clarity of the code.
@lemillermicrosoft
🔌 Add extension methods for importing plugin manifest
aec848d 
This commit adds two extension methods for the IKernel interface that
allow importing a plugin manifest document from a URL or a string. The
plugin manifest is a JSON document that lists the names and URLs of
remote plugins that provide chat GPT skills. The extension methods use
the existing ImportChatGptPluginSkillFromUrlAsync method to import each
plugin and its functions, and return a dictionary of the imported
functions. The commit also adds a model class for the plugin manifest
and its contents.
@lemillermicrosoft
WIP 🧪 Add integration tests for sequential planner config and plugin …
def7b3f 
…manifest

This commit adds two new integration test classes for the sequential
planner configuration and the plugin manifest functionality. The tests
use the Azure OpenAI services for text completion and embedding
generation, and require the corresponding configuration values to be
set in the appsettings.json file. The tests also use a custom logger
and a redirect output helper to capture the test output. The test
classes implement the IDisposable interface to properly dispose of the
logger and the output helper.

The commit also adds the ability to supply custom functions for getting
available functions and getting a specific function in the sequential
planner config. This allows the planner to use functions from a plugin
manifest instead of relying on the default context.Skills. The commit
includes a test case for creating a plan with custom functions, using a
fake email skill and a chat plugin skill from Klarna. The test case
demonstrates how to load a plugin manifest, populate a vector store
with the functions, and use the custom functions to create and invoke
a plan.
@lemillermicrosoft lemillermicrosoft added this to the Sprint 37 milestone Jun 27, 2023
@lemillermicrosoft
Copy link
Member Author

Thanks for your PR and contribution. For now, we are closing old drafts. Please re-open as appropriate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SergeyMenshykh SergeyMenshykh SergeyMenshykh left review comments

Assignees

@lemillermicrosoft lemillermicrosoft

Labels
kernel Issues or pull requests impacting the core kernel .NET Issue or Pull requests regarding .NET code
Projects
None yet
Milestone
R3: Cycle 2
Development

Successfully merging this pull request may close these issues.
2 participants
@lemillermicrosoft @SergeyMenshykh