Python: FunctionCallingStepwisePlanner ignoring passed excluded_functions

[ColtonBehannon]

I know there are changes with the planner happening, but I currently still use the function calling stepwise planner and am experiencing an issue where my passed list of excluded functions are being ignored. If I include

logger.info(f"excluded_functions list: {excluded_functions}")
logger.info(f"planner.excluded_functions: {planner.options.excluded_functions}")

after instantiating the planner I get the correct list of excluded functions both times. However, when I generate the plan with planner._generate_plan() it is only excluding the sequential_planner-create_plan function.

The ramifications of this are that if a user selects only a subset of the functions to use that are available in the kernel, then the planner will create a plan that ignores this and pretends that all functions will be available leading to a broken or incorrect execution.

I think this is happening as a result of this line of the _generate_plan method but it could also just be that I am using it incorrectly.

Please let me know if this is intended and I am just using it incorrectly. The only workaround I can think of would be a separate kernel with only the configured plugins/functions available for the planner.

Here is the relevant code.

options = FunctionCallingStepwisePlannerOptions(
        max_iterations=10,
        max_tokens=400,
        get_initial_plan=get_initial_plan,
        excluded_functions=excluded_functions,
)
planner = FunctionCallingStepwisePlanner(service_id=service_id, options=options)
llm_generated_plan = await planner._generate_plan(
        question=user_input, kernel=kernel, arguments=arguments
)

[ColtonBehannon]

@moonbox3 any ideas?

[moonbox3]

Hi @ColtonBehannon, that line you pointed to in _generate_plan does look indeed to be an issue. We should be concatenating the hard-coded excluded plugin with whatever else is included in self.options.excluded_plugins.

The only thing I can think of to immediately unblock you, unless you want to raise a PR to fix the issue, is to override the generate plan method, and tailor it to work as you expect. Something like this:

    async def _new_generate_plan(
        self,
        question: str,
        kernel: Kernel,
        arguments: KernelArguments,
    ) -> str:
        """Generate the plan for the given question using the kernel."""
        generate_plan_function = self._create_config_from_yaml(kernel)
        # build the proper list of excluded functions
        functions_manual = [
            kernel_function_metadata_to_function_call_format(f)
            for f in kernel.get_list_of_function_metadata({
                "excluded_functions": <list of excluded functions>
            })
        ]
        generated_plan_args = KernelArguments(
            name_delimiter="-",
            available_functions=functions_manual,
            goal=question,
        )
        generated_plan_args.update(arguments)
        generate_plan_result = await kernel.invoke(generate_plan_function, generated_plan_args)
        return str(generate_plan_result)

    for question in questions:
        planner._generate_plan = _new_generate_plan.__get__(planner, FunctionCallingStepwisePlanner)
        await planner._generate_plan(question=question, kernel=kernel, arguments=KernelArguments())
        result = await planner.invoke(kernel, question)
        print(f"Q: {question}\nA: {result.final_answer}\n")

[ColtonBehannon]

I just wanted to confirm I wasn't missing something. Thanks for the reply and workaround!