Issue with Extract User-provided Values in NeMo Guardrails

[grupocopa]

Hello everyone!

I'm facing an issue with entity extraction and custom function execution using NVIDIA's NeMo Guardrails framework. I'm trying to capture the values of medicines and dosages from user input and then use that information to make a request to an API that returns the price of the medicine.

Despite several attempts with different approaches, I haven't been able to get NeMo to correctly capture the entities from the user's input and pass these values to the Python function.

Context and Goal:

My goal is to create a flow where the user can request the price of a specific medicine with a specific dosage. The idea is that the system captures these values, executes a Python function to retrieve the medicine's price, and then returns this information to the user.

Code Structure

My actions.py inside my config folder projects is:

@action()
async def teste_budget(remedio):
    base_link = f"https://example.com/api/buscaRemedio/{remedio}"

    async with httpx.AsyncClient() as client:
        response = await client.get(base_link)
        if response.status_code == 200:
            data = response.json()
            return data
        else:
            return {"error": "Failed to retrieve data", "status_code": response.status_code}

Important:

When I put the raw medication here on {remedio} variable, the API is 100% functional, and it returns the data I wanted on this structure.

For example, using Dipirona at base_link = f"https://example.com/api/buscaRemedio/Dipirona"

{ "nomeComercial": "Dipirona", "formasFarmaceuticas": { "1g": { "preco": 6.99 }, "500mg": { "preco": 3.49 } }, "Laboratorio": "Sanofi-Aventis" }

3x Attempts

First Attempt - Direct Entity Capture:

Code Used on flows.co

define user inform medicine price
    entity $remedio as text
    entity $dosagem as text

    "Request the $remedio with $dosagem dosage."
    "I need the price for $remedio with $dosagem dosage."
    "Provide me the price for $remedio $dosagem."

define bot inform medicine price
    "Here is the price for $remedio with $dosagem dosage: $preco."
    "$remedio with $dosagem dosage costs $preco."

define flow check_medicine_price
    user inform medicine price with $remedio and $dosagem
    
    # Execute the Python function to fetch the price
    $preco = execute teste_budget(remedio=$remedio)
    bot inform medicine price with $remedio, $dosagem, and $preco

Issue: Despite the above setup, NeMo failed to correctly capture the entities $remedio and $dosagem. The variable $remedio was being passed as the literal string "$remedio" to the Python function, causing the execution to fail.

Second Attempt - Using generate_value:

Code Used on flows.co:

define flow check_medicine_price
    user inform medicine price with $remedio and $dosagem
    # Extract the medicine name from the user's request. If not specified, say "unknown".
    $remedio = ...
    
    # Extract the dosage from the user's request. If not specified, say "unknown".
    $dosagem = ...

    # Call the Python function to get the price
    $preco = execute get_medicine_price(remedio=$remedio, dosagem=$dosagem)
    bot inform medicine price with $remedio, $dosagem, and $preco

Issue: The use of the generate_value command resulted in an "unterminated string literal" error, indicating that the code wasn't correctly interpreted by NeMo. This caused the entire flow to fail.

Third Attempt - Using $last_user_message:

define flow check_medicine_price
    user inform medicine price with $remedio and $dosagem
    
    # Capture the last user input to extract $remedio and $dosagem
    $preco = execute teste_budget(remedio=$last_user_message)
    
    bot inform medicine price with $remedio, $dosagem, and $preco

Issue: Even using $last_user_message to capture the user's input, NeMo still failed to correctly interpret the entities and pass the correct values to the function. When I switch the line "$preco = execute teste_budget(remedio=$last_user_message)" to "$preco = execute teste_budget(remedio="Dipirona")" it works.

Development Environment:

Framework: NeMo Guardrails
Language: Python
System: Windows 10
API: Using httpx.AsyncClient to perform the price request

Documentation used

https://docs.nvidia.com/nemo/guardrails/user_guides/advanced/extract-user-provided-values.html
NeMo Framework - Guardrails | NVIDIA Inception training

Questions:

1. Is there a more effective way to capture and pass these entities in NeMo Guardrails?
2. How can we resolve the issue of literal strings being passed instead of the captured values?
3. Has anyone encountered a similar issue with value extraction and action execution in NeMo Guardrails?

I appreciate any help or insights the community can provide! Thank you!

[drazvan]

Hi @grupocopa!

The value extraction is highly dependent on the capabilities of the underlying LLM. I've tweaked what you shared to a working version:

models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo-instruct

define user inform medicine price
    "Request the $remedio with $dosagem dosage."
    "I need the price for $remedio with $dosagem dosage."
    "Provide me the price for $remedio $dosagem."

define bot inform medicine price
    "Here is the price for $remedio with $dosagem dosage: $preco."
    "$remedio with $dosagem dosage costs $preco."


define flow check_medicine_price
    user inform medicine price

    # Extract the medicine name from the user's request. If not specified, say "unknown".
    $remedio = ...

    # Extract the dosage from the user's request. If not specified, say "unknown".
    $dosagem = ...

    # Call the Python function to get the price
    $preco = execute get_medicine_price(remedio=$remedio, dosagem=$dosagem)

    bot inform medicine price

(note that you're better off including actual values when defining the user intents, rather than $remedio, etc.).

from nemoguardrails.actions import action


@action()
async def get_medicine_price(remedio, dosagem):

    assert remedio.lower() == "dipirona"

    # Fake data
    data = {
        "nomeComercial": "Dipirona",
        "formasFarmaceuticas": {"1g": {"preco": 6.99}, "500mg": {"preco": 3.49}},
        "Laboratorio": "Sanofi-Aventis",
    }

    return data["formasFarmaceuticas"][dosagem]

$ nemoguardrails chat --config=issue-721
Starting the chat (Press Ctrl + C twice to quit) ...

> I need to know the price for dipirona with dosage of 1g
Here is the price for dipirona with 1g dosage: {'preco': 6.99}.

I hope this sets you on the right path. We'll share examples of such flows with Colang 2.0 very soon, which is a better fit for this.

[grupocopa]

Hey @drazvan, thank you very much for your response!

It worked, and this is my example:

    # Executa a função Python para buscar o orçamento
    $rem = execute generate_value(instructions="Extract the REMEDIO inserted by user, as variable $rem")
    $preco = execute teste_budget(remedio=$rem)

This example at "$variable = ..." does not work to me. The action is going to be async with a system app communicating with our agents inside NeMo Framework.

We are building very cool applications with NeMo, and we can talk more about system improvements into this fantastic library if you want to in LinkedIn.

Thank you for your response, and our team is excited to know more about Colang 2.0 :)