Merge pull request #105 from ks6088ts-labs/feature/issue-69_promptflo…

…w-demo

fork chat-math-variant sample

.pre-commit-config.yaml

@@ -13,4 +13,4 @@ repos:
     rev: 24.2.0
     hooks:
     -   id: black
-exclude: 'generated/.*|artifacts/.*|.jsonl|.csv'
+exclude: 'generated/.*|artifacts/.*|.jsonl|.csv|.json'

apps/11_promptflow/README.md

@@ -203,6 +203,32 @@ $ pf run create \
 $ pf run show-details --name $RUN_NAME
 ```
 
+### chat-math-variant
+
+Tuning prompts using `variants` is a powerful feature in Prompt flow. It allows you to test different prompts and see which one works best for your use case.
+
+Prompt flow repository provides an example of a chat flow with math variants at [examples/flows/chat/chat-math-variant](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat/chat-math-variant).
+
+To understand how to use variants, you can refer to the [How-to Guides > Tune prompts using variants](https://microsoft.github.io/promptflow/how-to-guides/tune-prompts-with-variants.html) document.
+
+```shell
+cd apps/11_promptflow/chat-math-variant
+
+# Create run with multiple lines data with variant
+$ RUN_NAME=chat-math-variant-$(date +%s)
+$ VARIANT='${chat.variant_0}'
+$ pf run create \
+    --name $RUN_NAME \
+    --flow . \
+    --data ./data.jsonl \
+    --column-mapping question='${data.question}' \
+    --variant $VARIANT \
+    --stream
+
+# Show run details
+$ pf run show-details --name $RUN_NAME
+```
+
 ## References
 
 - [Prompt flow > repos](https://github.com/microsoft/promptflow)

apps/11_promptflow/chat-math-variant/.gitignore

@@ -0,0 +1,5 @@
+.env
+__pycache__/
+.promptflow/*
+!.promptflow/flow.tools.json
+.runs/

apps/11_promptflow/chat-math-variant/.promptflow/flow.tools.json

@@ -0,0 +1,65 @@
+{
+  "package": {},
+  "code": {
+    "chat.jinja2": {
+      "type": "llm",
+      "inputs": {
+        "chat_history": {
+          "type": [
+            "string"
+          ]
+        },
+        "question": {
+          "type": [
+            "string"
+          ]
+        }
+      },
+      "source": "chat.jinja2"
+    },
+    "chat_variant_1.jinja2": {
+      "type": "llm",
+      "inputs": {
+        "chat_history": {
+          "type": [
+            "string"
+          ]
+        },
+        "question": {
+          "type": [
+            "string"
+          ]
+        }
+      },
+      "source": "chat_variant_1.jinja2"
+    },
+    "chat_variant_2.jinja2": {
+      "type": "llm",
+      "inputs": {
+        "chat_history": {
+          "type": [
+            "string"
+          ]
+        },
+        "question": {
+          "type": [
+            "string"
+          ]
+        }
+      },
+      "source": "chat_variant_2.jinja2"
+    },
+    "extract_result.py": {
+      "type": "python",
+      "inputs": {
+        "input1": {
+          "type": [
+            "string"
+          ]
+        }
+      },
+      "source": "extract_result.py",
+      "function": "my_python_tool"
+    }
+  }
+}

apps/11_promptflow/chat-math-variant/README.md

@@ -0,0 +1,48 @@
+# Test your prompt variants for chat with math
+This is a prompt tuning case with 3 prompt variants for math question answering.
+
+By utilizing this flow, in conjunction with the `evaluation/eval-chat-math` flow, you can quickly grasp the advantages of prompt tuning and experimentation with prompt flow. Here we provide a [video](https://www.youtube.com/watch?v=gcIe6nk2gA4) and a [tutorial]((../../../tutorials/flow-fine-tuning-evaluation/promptflow-quality-improvement.md)) for you to get started.
+
+Tools used in this flow：
+- `llm` tool
+- custom `python` Tool
+
+## Prerequisites
+
+Install promptflow sdk and other dependencies in this folder:
+```bash
+pip install -r requirements.txt
+```
+
+## Getting started
+
+### 1 Create connection for LLM tool to use
+Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
+
+Currently, there are two connection types supported by LLM tool: "AzureOpenAI" and "OpenAI". If you want to use "AzureOpenAI" connection type, you need to create an Azure OpenAI service first. Please refer to [Azure OpenAI Service](https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/) for more details. If you want to use "OpenAI" connection type, you need to create an OpenAI account first. Please refer to [OpenAI](https://platform.openai.com/) for more details.
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Note in [flow.dag.yaml](flow.dag.yaml) we are using connection named `open_ai_connection`.
+```bash
+# show registered connection
+pf connection show --name open_ai_connection
+```
+
+### 2 Start chatting
+
+```bash
+# run chat flow with default question in flow.dag.yaml
+pf flow test --flow .
+
+# run chat flow with new question
+pf flow test --flow . --inputs question="2+5=?"
+
+# start a interactive chat session in CLI
+pf flow test --flow . --interactive
+
+# start a interactive chat session in CLI with verbose info
+pf flow test --flow . --interactive --verbose

apps/11_promptflow/chat-math-variant/chat.jinja2

@@ -0,0 +1,13 @@
+# system:
+You are an assistant to calculate the answer to the provided math problems.
+Please return the final numerical answer only, without any accompanying reasoning or explanation.
+
+{% for item in chat_history %}
+# user:
+{{item.inputs.question}}
+# assistant:
+{{item.outputs.answer}}
+{% endfor %}
+
+# user:
+{{question}}

apps/11_promptflow/chat-math-variant/chat_variant_1.jinja2

@@ -0,0 +1,23 @@
+# system:
+You are an assistant to calculate the answer to the provided math problems.
+Please think step by step.
+Return the final numerical answer only and any accompanying reasoning or explanation seperately as json format.
+
+# user:
+A jar contains two red marbles, three green marbles, ten white marbles and no other marbles. Two marbles are randomly drawn from this jar without replacement. What is the probability that these two marbles drawn will both be red? Express your answer as a common fraction.
+# assistant:
+{Chain of thought: "The total number of marbles is $2+3+10=15$.  The probability that the first marble drawn will be red is $2/15$.  Then, there will be one red left, out of 14.  Therefore, the probability of drawing out two red marbles will be: $$\\frac{2}{15}\\cdot\\frac{1}{14}=\\boxed{\\frac{1}{105}}$$.", "answer": "1/105"}
+# user:
+Find the greatest common divisor of $7!$ and $(5!)^2.$
+# assistant:
+{"Chain of thought": "$$ \\begin{array} 7! &=& 7 \\cdot 6 \\cdot 5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1 &=& 2^4 \\cdot 3^2 \\cdot 5^1 \\cdot 7^1 \\\\ (5!)^2 &=& (5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1)^2 &=& 2^6 \\cdot 3^2 \\cdot 5^2 \\\\ \\text{gcd}(7!, (5!)^2) &=& 2^4 \\cdot 3^2 \\cdot 5^1 &=& \\boxed{720} \\end{array} $$.", "answer": "720"}
+{% for item in chat_history %}
+
+# user:
+{{item.inputs.question}}
+# assistant:
+{{item.outputs.answer}}
+{% endfor %}
+
+# user:
+{{question}}

apps/11_promptflow/chat-math-variant/chat_variant_2.jinja2

@@ -0,0 +1,39 @@
+# system:
+You are an assistant to calculate the answer to the provided math problems.
+Please think step by step.
+Return the final numerical answer only and any accompanying reasoning or explanation seperately as json format.
+
+# user:
+A jar contains two red marbles, three green marbles, ten white marbles and no other marbles. Two marbles are randomly drawn from this jar without replacement. What is the probability that these two marbles drawn will both be red? Express your answer as a common fraction.
+# assistant:
+{Chain of thought: "The total number of marbles is $2+3+10=15$.  The probability that the first marble drawn will be red is $2/15$.  Then, there will be one red left, out of 14.  Therefore, the probability of drawing out two red marbles will be: $$\\frac{2}{15}\\cdot\\frac{1}{14}=\\boxed{\\frac{1}{105}}$$.", "answer": "1/105"}
+# user:
+Find the greatest common divisor of $7!$ and $(5!)^2.$
+# assistant:
+{"Chain of thought": "$$ \\begin{array} 7! &=& 7 \\cdot 6 \\cdot 5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1 &=& 2^4 \\cdot 3^2 \\cdot 5^1 \\cdot 7^1 \\\\ (5!)^2 &=& (5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1)^2 &=& 2^6 \\cdot 3^2 \\cdot 5^2 \\\\ \\text{gcd}(7!, (5!)^2) &=& 2^4 \\cdot 3^2 \\cdot 5^1 &=& \\boxed{720} \\end{array} $$.", "answer": "720"}
+# user:
+A club has 10 members, 5 boys and 5 girls.  Two of the members are chosen at random.  What is the probability that they are both girls?
+# assistant:
+{"Chain of thought": "There are $\\binomial{10}{2} = 45$ ways to choose two members of the group, and there are $\\binomial{5}{2} = 10$ ways to choose two girls.  Therefore, the probability that two members chosen at random are girls is $\\dfrac{10}{45} = \\boxed{\\dfrac{2}{9}}$.", "answer": "2/9"}
+# user:
+Allison, Brian and Noah each have a 6-sided cube. All of the faces on Allison's cube have a 5. The faces on Brian's cube are numbered 1, 2, 3, 4, 5 and 6. Three of the faces on Noah's cube have a 2 and three of the faces have a 6. All three cubes are rolled. What is the probability that Allison's roll is greater than each of Brian's and Noah's? Express your answer as a common fraction.
+# assistant:
+{"Chain of thought": "Since Allison will always roll a 5, we must calculate the probability that both Brian and Noah roll a 4 or lower. The probability of Brian rolling a 4 or lower is $\\frac{4}{6} = \\frac{2}{3}$ since Brian has a standard die. Noah, however, has a $\\frac{3}{6} = \\frac{1}{2}$ probability of rolling a 4 or lower, since the only way he can do so is by rolling one of his 3 sides that have a 2. So, the probability of both of these independent events occurring is $\\frac{2}{3} \\cdot \\frac{1}{2} = \\boxed{\\frac{1}{3}}$.", "answer": "1/3"}
+# user:
+Compute $\\density binomial{50}{2}$.
+# assistant:
+{"Chain of thought": "$\\density binomial{50}{2} = \\dfrac{50!}{2!48!}=\\dfrac{50\\times 49}{2\\times 1}=\\boxed{1225}.$", "answer": "1225"}
+# user:
+The set $S = \\{1, 2, 3, \\ldots , 49, 50\\}$ contains the first $50$ positive integers.  After the multiples of 2 and the multiples of 3 are removed, how many integers remain in the set $S$?
+# assistant:
+{"Chain of thought": "The set $S$ contains $25$ multiples of 2 (that is, even numbers).  When these are removed, the set $S$ is left with only the odd integers from 1 to 49. At this point, there are $50-25=25$ integers in $S$. We still need to remove the multiples of 3 from $S$.\n\nSince $S$ only contains odd integers after the multiples of 2 are removed,  we must remove the odd multiples of 3 between 1 and 49.  These are 3, 9, 15, 21, 27, 33, 39, 45, of which there are 8.  Therefore, the number of integers remaining in the set $S$ is $25 - 8 = \\boxed{17}$.", "answer": "17"}
+{% for item in chat_history %}
+
+# user:
+{{item.inputs.question}}
+# assistant:
+{{item.outputs.answer}}
+{% endfor %}
+
+# user:
+{{question}}