Which is the best approach for large FAQ

[valeValentini]

Hello everyone!

I'm developing a chatbot using Colang 2 for a project involving around 300 FAQ entries. I’d like to know the best practice for structuring a chatbot with a large number of predefined flows.
Is a conditional approach (using if/elif statements) the most efficient solution, or is there a more optimal way to handle many intents without introducing latency?

Initially, I structured the main flow using multiple or when statements to detect intents, with each intent having up to three sample expressions. The code was similar to the following:

flow main
    bot express greeting
    activate automating intent detection
    activate generating user intent for unhandled user utterance
    
    while True
        when user greeted
            bot say "Hello! Do you have any questions about..?"
        or when user said goodbye
            bot say "Goodbye!"
        or when unhandled user intent
            bot say "Sorry, I didn't understand. Could you rephrase?"
        or when user asked to contact assisteance
            bot say "If you've updated your IBAN, any payments..."

where the user flows are:

flow user said contact assistance 
     user said "How..." 
     or user said "..."
     or user said "..."

And the bot flows are

flow bot express assistance response 
   bot say "The Service..."

When the number of intents is >40, this approach started introducing significant latency (up to 10 seconds). The delay is not in the bot's response itself, but in closing the unhandled events of each flow and allowing new user input. It seems that activating multiple or when statements concurrently causes a slowdown as i aspected.

So to address this, I considered switching to a conditional branching structure, where each intent is routed using if/elif statements within the while loop, rather than activating multiple flows simultaneously. But i can't find a way to get the user intent in a string to than pass it to the condition check.

import core
import guardrails
import llm

flow main
    bot express greeting  
    activate unhandled user intent  
    
    while True
        await generating user intent for unhandled user utterance as $intent   
        bot say "{$intent}" 
        if $intent == "greeted"
            bot say "Hello! ....." 
        elif $intent == "said goodbye"
            bot say "Goodbye!"
            break
        elif $intent == "contact assistance"
            bot say "For assistance, please contact...."

So :( I can’t seem to correctly extract the intent from generating user intent for unhandled user utterance.
From debugging, I could see that the value is saved in the GenerateUserIntentActionFinished event’s return_value property, but I’m unable to retrieve and use it within the conditional branching.

Is this if/elif approach the correct one, or are there alternative methods for structuring this?
Can anyone advise on how to extract the intent effectively?

Thank you so much!!

[schuellc-nvidia]

Hi,
You are right, using the when construct is not very efficient for the number of cases your are looking for since all the flows will need to be started/aborted each time. You can use the activation pattern instead like this:

flow main
    bot express greeting
    activate automating intent detection
    activate generating user intent for unhandled user utterance
    
    activate handling user greeted
    activate handling user said goodbye
    activate handling user asked to contact assistance
    activate handling unhandled user intent
    
    wait indefinitely

flow handling user greeted
    user greeted
    bot say "Hello! ....."

flow handling user said goodbye
    user said goodbye
    bot say "Goodbye!"

flow handling user asked to contact assistance
    user asked to contact assistance
    bot say "For assistance, please contact...."

flow handling unhandled user intent
    unhandled user intent
    bot say "Sorry, I didn't understand. Could you rephrase?"

# user intent flows...

With this approach only one of the handling flows will match and advance while all the others keep waiting for their corresponding events.

Also note, that with smarted models like GTP4 you can use docstring like descriptions to define user intent flows to reduce the number of needed examples:

flow user greeted
    """User expressed greeting."""
    user said "Hi"

flow user said goodbye
    """User expressed goodbye or that they are leaving."""
    user said "Bye"

flow user asked to contact assistance
    """User asked to contact a human assistance."""
    user said "Connect me to someone real."

[schuellc-nvidia]

But if you would rather want to use the if construct, you could do it like this:

import core
import guardrails
import llm

flow main
    activate automating intent detection
    
    bot express greeting  
    
    while True:
        generating user intent for unhandled user utterance as $ref
        $intent = $ref.intent
        if $intent == "user greeted":
            bot say "Hello! ....."
        elif $intent == "user said goodbye":
            bot say "Goodbye!"
            break
        elif $intent == "user asked to contact assistance"
            bot say "For assistance, please contact...."
        else:
            bot say "Sorry, I didn't understand. Could you rephrase?"

# Definition of all the user intent flows

Or alternatively:

flow main
    activate automating intent detection
    activate generating user intent for unhandled user utterance
    
    bot express greeting  
    
    while True:
        match (generating user intent for unhandled user utterance).Finished() as $ref
        $intent = $ref.flow.intent
        if $intent == "user greeted":
...            

[valeValentini]

Hi!! Thank you so much for your reply, it was really helpful. I applied the first version you suggested, and it worked well. But I'v also run into a few issues and have some questions..

Since I have a large number of intents, including all of them in the model prompt causes it to grow (up to ~3k tokens per call) so I had to increase the max_length parameter for the generate_user_intent_from_user_action task since I ran in the _Error while execution 'GenerateUserIntentAction' with parameters '{'user_action': 'user said "ciao"', 'max_example_flows': 20}': Prompt exceeds max length of 16000 characters even without history_

Here's my config.yml file:

colang_version: "2.x"

models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo

instructions:
  - type: general
    content: |
        .................

sample_conversation: |
  user action: user said "..."

rails:
  dialog:
    single_call:
      enabled: True
      fallback_to_multiple_calls: True
      user_messages:
        embeddings_only: True
        embeddings_only_similarity_threshold: 0.4
        embeddings_only_fallback_intent: unhandled user intent

knowledge_base:
  embedding_search_provider:
    name: default
    parameters:
      embedding_engine: openai
      embedding_model: text-embedding-ada-002
    cache:
      enabled: True
      key_generator: md5
      store: filesystem
      store_config: {}

prompts:
  - task: generate_user_intent_from_user_action
    models:
      - openai/gpt-3.5-turbo
    max_length: 18000
    messages:
      - type: system
        content: "{{ general_instructions }}"
      - type: system
        content: "This is how a conversation between a user and the bot can go:"
      - "{{ sample_conversation | to_messages_v2 }}"
      - type: system
        content: |-
          "These are the most likely user intents:"
          {{ examples }}
      - type: system
        content: "This is the current conversation between the user and the bot:"
      - "{{ history | colang | to_messages_v2}}"
      - type: user
        content: "user action: {{ user_action }}"
      - type: system
        content: "Derive `user intent:` from user action considering the intents from section 'These are the most likely user intents':"

So my questions are:
Is there a way to dynamically reduce the number of intents sent to the model? Like retrieve only the most relevant intents based on embeddings rather than including all intents in the prompt? I don't get what’s the point of having activate automating intent detection if all intents still need to be part of the prompt?

With the implementation you suggested it seems that modifying the embeddings_only_similarity_threshold doesn't effect the responses at all. Is because they're not in a while loop?

Also why if I call the server i get this error _ERROR:nemoguardrails.server.api:Providing assistant messages as input is not supported for Colang 2.0 configurations._
Is it related to how assistant messages are structured in my configuration file, or is there a specific limitation with Colang 2.0?

Again thank you so much for your time and help!

[schuellc-nvidia]

How many user intents do you have? How many examples to do you provide for each of them in average?

[valeValentini]

I have around 220 intents, and for each intent i’ve provided 3-4 user expressions and a single bot response.

[schuellc-nvidia]

Ok, try to us the if-construct for now which should make that better. I see that we currently don't have a good solution to limit the number of included intents in the LLM prompt. I will create a feature request for that.