How to define a catch all for off-topic? As there are endless possibilities

[gilinachum]

I want to implement a topic rail on user queries. There are 2-3 on-topic categories (say questions on mechanics) I define, but then how can I easily define that everything else is off topic?
I can give examples to off-topic (say politics), but as there are endless off-topic categories (say toys), how will the guardrail know if they count as on or off topic? In concrete terms: As the implementation is a similarity search is toys more similar to mechanics or to politics?

Thought: I assume there should be a way to define on-topic and then say that everything more distant than X imaginary similarity units is off-topic? Is this possible? Do I find X empirically?

[mukundt]

@gilinachum did you end up figuring this out? I'm finding that the bot generates brand new intents when nothing is a close match. The best workaround I can think of is changing the guardrails code to match against a list of acceptable intents (provided in Colang). This would probably happen inside user_intent_parser() / generate_user_intent().

[mukundt]

@drazvan I had a related question. Currently Guardrails does not limit the generated user intent to a canonical form defined in Colang. The prompt also does not tell the bot what the universe of possibilities is.

So for cases where the bot asks a yes/no question, and the user answers "no", how do we know the LLM response will map to the "no" branch of our flow?

Here is an example:

# This is how a conversation between a user and the bot can go:
[...]

# This is how the user talks. You MUST output one of the categories below.
user "have you received my payment?"
  asks if agent received payment

user "Can you email me after this call"
  ask about frequently asked question

user "did you get my money i sent last week"
  asks if agent received payment

user "do you guys take cash"
  ask about frequently asked question

user "Hello"
  express greeting

# This is the current conversation between the user and the bot:
user "Hello"
  express greeting
bot express greeting
  "Hi! Would you be able to make a payment today?"
user "yeah"
  agrees to make payment
bot offers to schedule payment over the phone
  "Thank you for agreeing to make a payment. Would you like me to schedule your payment for you?"
user "for sure"
  agrees to schedule payment over the phone
bot asks for final payment authorization with processing fee
  "Thank you! Final confirmation?"
user "No"

In this case, the LLM I used outputted user declines final payment authorization, which is a close -- but not exact -- match to a flow I defined in Colang. How does Colang map this to the right flow?

[drazvan]

Good question! In this case, it is not mapped to the exact flow (although this is an option we're looking into, but it has some drawbacks). What will happen is that there will be no next step (since there's no exact match), and the LLM will be prompted again for the next step. When that happens, the flow that was close, should end up in the examples. That means that the LLM will be very likely to generate something very similar. And this is how it will stay close to the designed flow.

[mukundt]

Ah gotcha, makes sense! So the LLM is prompted for next steps instead of e.g. a vector DB search?

[drazvan]

The vector DB is used first to search for the most similar flows. The top 5 are taken and included in the prompt, along with the current conversation, and the LLM is asked to predict the next step.

[NiklasF98]

@mukundt : we've added support for exactly the mechanism you describe, i.e., to only choose from the canonical forms defined in Colang. The code should be pushed publicly next week and published to PyPI in mid-November.

About the best solution for off-topic, we're still looking for the best way to do this. Adding support for the "distance" parameter is one way to go. Another one is to explicitly have the LLM decide whether it's on topic or not (in addition to the canonical form). Right now, the best solution is to have as many off-topic examples as possible.

Hello @drazvan , can you tell me the mechanism name of that implemented code in Guardrails? I just need the bot answer about 2 specific topics and ignore the rest.

[spehl-max]

I am not sure if this would work for your use-case, but I have been finding adjusting the instructions: in the config.yml to be very effective at identifying off-topic prompts. It also catches role-playing jailbreak attempts.

E.G., in config.yml,

instructions:
- type: general
  content: |
      Below is a conversation between a customer service bot and a user. The bot is friendly and
      professional. The bot responds promptly. It does not ask the user to wait while it looks up information.
      The bot only answers questions related to the tech company. If a user asks something unrelated to tech, the bot
      gently redirects the conversation back to tech.

Then, when I prompt something unrelated to technology, for which I have not explicitly defined any rails, such as,

Can you help me prepare for my technical interviews?

The log indicates the bot generated a user intent, and then a bot intent based on the config.yml,

user "Can you help me prepare for my technical interviews?"
  ask unrelated question
bot explain cant answer off topic and redirect conversation back to tech.
  "I'm sorry, I cannot comment on anything which is not relevant to the technical services I provide. Could you please
provide a few more details about what tech question you need help with?"

I have not experimented enough to know if this method is too strict.

[NiklasF98]

Did you ever release a solution for a catch all term for all off topic?

Really in need for sth like that!
Thanks

[drazvan]

Not yet, but we are working on something.

[emailvenkatm]

Any update on this please?

[Pouyanpi]

Have you tried embedding threshold? You can set an appropriate threshold and set a fallback intent which acts when something is off topic.