Skip to content

Commit

Permalink
add AnswerCorrectness, ContextPrecision, ContextRecall and ContextRel…
Browse files Browse the repository at this point in the history 
…evance to example notebook
  • Loading branch information
b.nativi committed Aug 22, 2024
1 parent a3937a0 commit 7770c33
Show file tree
Hide file tree
Showing 2 changed files with 89 additions and 42 deletions.
2 changes: 1 addition & 1 deletion api/valor_api/backend/metrics/text_generation.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -431,7 +431,7 @@ def _compute_text_generation_metrics(
parameters={
"dataset": dataset_name,
"datum_uid": datum_uid,
"prediction": predictions,
"prediction": prediction,
},
)
]
Expand Down
129 changes: 88 additions & 41 deletions examples/text-generation/text_generation.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@
"outputs": [],
"source": [
"# For the purposes of this example, let's get metrics for just the first 5 datums.\n",
"MAX_DATUMS = 10\n",
"MAX_DATUMS = 5\n",
"\n",
"# Create, build and finalize the dataset and model.\n",
"dataset = Dataset.create(\n",
Expand Down Expand Up @@ -301,10 +301,13 @@
"eval_job = model.evaluate_text_generation(\n",
" dataset,\n",
" metrics_to_return=[\n",
" \"AnswerCorrectness\",\n",
" \"AnswerRelevance\", \n",
" \"Bias\",\n",
" \"BLEU\", \n",
" \"Coherence\", \n",
" \"ContextPrecision\",\n",
" \"ContextRecall\",\n",
" \"ContextRelevance\",\n",
" \"Faithfulness\",\n",
" \"Hallucination\",\n",
" \"ROUGE\",\n",
Expand Down Expand Up @@ -333,6 +336,15 @@
"# Here are some example metrics. These are all for query0 and were evaluated by GPT-4o.\n",
"example_expected_metrics = [\n",
" {\n",
" 'type': 'AnswerCorrectness',\n",
" 'value': 0.5,\n",
" 'parameters': {\n",
" 'dataset': 'rag_dataset', \n",
" 'datum_uid': 'query0', \n",
" 'prediction': \"Cleveland's opponents in 1884 criticized his alleged involvement in a scandal regarding an illegitimate child, which was used to counter his innocent image during the presidential campaign.\",\n",
" },\n",
" },\n",
" {\n",
" 'type': 'AnswerRelevance',\n",
" 'value': 1.0,\n",
" 'parameters': {\n",
Expand Down Expand Up @@ -361,38 +373,74 @@
" },\n",
" },\n",
" {\n",
" 'type': 'Coherence',\n",
" 'value': 4.0,\n",
" 'type': 'ContextPrecision', \n",
" 'value': 0.0,\n",
" 'parameters': {\n",
" 'dataset': 'rag_dataset',\n",
" 'datum_uid': 'query0',\n",
" 'prediction': \"Cleveland's opponents in 1884 criticized his alleged involvement in a scandal regarding an illegitimate child, which was used to counter his innocent image during the presidential campaign.\"\n",
" 'dataset': 'rag_dataset', \n",
" 'datum_uid': 'query0', \n",
" 'context_list': [\n",
" \"Cleveland was defeated in the 1888 presidential election, in part due to fraud (See Blocks of Five). He actually led in the popular vote over Benjamin Harrison (48.6% to 47.8%), but Harrison won the Electoral College by a 233-168 margin, largely by squeaking out a barely-over-1% win in Cleveland's home state of New York; in fact, had Cleveland won his home state, he would have won the electoral vote by a count of 204-197 (201 votes then needed for victory). Note, though, that Cleveland earned 24 of his electoral votes in states that he won by less than 1% (Connecticut, Virginia, and West Virginia).\", \n",
" \"Some of Cleveland's actions were controversial with political factions. Such criticisms include but are not limited to: his intervention in the Pullman Strike of 1894 in order to keep the railroads moving (a move which angered labor unions), his support of the gold standard, and opposition to free silver which alienated the agrarian wing of the Democrats. Furthermore, critics complained that he had little imagination and seemed overwhelmed by the nation's economic disasters depressions and strikes in his second term. He lost control of his party to the agrarians and silverites in 1896.\"\n",
" ]\n",
" },\n",
" },\n",
" {\n",
" 'type': 'ContextRecall',\n",
" 'value': 0.0, \n",
" 'parameters': {\n",
" 'dataset': 'rag_dataset', \n",
" 'datum_uid': 'query0', \n",
" 'context_list': [\n",
" \"Cleveland was defeated in the 1888 presidential election, in part due to fraud (See Blocks of Five). He actually led in the popular vote over Benjamin Harrison (48.6% to 47.8%), but Harrison won the Electoral College by a 233-168 margin, largely by squeaking out a barely-over-1% win in Cleveland's home state of New York; in fact, had Cleveland won his home state, he would have won the electoral vote by a count of 204-197 (201 votes then needed for victory). Note, though, that Cleveland earned 24 of his electoral votes in states that he won by less than 1% (Connecticut, Virginia, and West Virginia).\", \n",
" \"Some of Cleveland's actions were controversial with political factions. Such criticisms include but are not limited to: his intervention in the Pullman Strike of 1894 in order to keep the railroads moving (a move which angered labor unions), his support of the gold standard, and opposition to free silver which alienated the agrarian wing of the Democrats. Furthermore, critics complained that he had little imagination and seemed overwhelmed by the nation's economic disasters depressions and strikes in his second term. He lost control of his party to the agrarians and silverites in 1896.\"\n",
" ]\n",
" },\n",
" },\n",
" {\n",
" 'type': 'ContextRelevance', \n",
" 'value': 0.0,\n",
" 'parameters': {\n",
" 'dataset': 'rag_dataset', \n",
" 'datum_uid': 'query0', \n",
" 'context_list': [\n",
" \"Cleveland was defeated in the 1888 presidential election, in part due to fraud (See Blocks of Five). He actually led in the popular vote over Benjamin Harrison (48.6% to 47.8%), but Harrison won the Electoral College by a 233-168 margin, largely by squeaking out a barely-over-1% win in Cleveland's home state of New York; in fact, had Cleveland won his home state, he would have won the electoral vote by a count of 204-197 (201 votes then needed for victory). Note, though, that Cleveland earned 24 of his electoral votes in states that he won by less than 1% (Connecticut, Virginia, and West Virginia).\", \n",
" \"Some of Cleveland's actions were controversial with political factions. Such criticisms include but are not limited to: his intervention in the Pullman Strike of 1894 in order to keep the railroads moving (a move which angered labor unions), his support of the gold standard, and opposition to free silver which alienated the agrarian wing of the Democrats. Furthermore, critics complained that he had little imagination and seemed overwhelmed by the nation's economic disasters depressions and strikes in his second term. He lost control of his party to the agrarians and silverites in 1896.\"\n",
" ]\n",
" },\n",
" },\n",
" # {\n",
" # 'type': 'Coherence',\n",
" # 'value': 4.0,\n",
" # 'parameters': {\n",
" # 'dataset': 'rag_dataset',\n",
" # 'datum_uid': 'query0',\n",
" # 'prediction': \"Cleveland's opponents in 1884 criticized his alleged involvement in a scandal regarding an illegitimate child, which was used to counter his innocent image during the presidential campaign.\"\n",
" # },\n",
" # },\n",
" {\n",
" 'type': 'Faithfulness',\n",
" 'value': 0.0,\n",
" 'parameters': {\n",
" 'dataset': 'rag_dataset',\n",
" 'datum_uid': 'query0',\n",
" 'prediction': \"Cleveland's opponents in 1884 criticized his alleged involvement in a scandal regarding an illegitimate child, which was used to counter his innocent image during the presidential campaign.\",\n",
" 'context_list': [\n",
" \"Cleveland was defeated in the 1888 presidential election, in part due to fraud (See Blocks of Five). He actually led in the popular vote over Benjamin Harrison (48.6% to 47.8%), but Harrison won the Electoral College by a 233-168 margin, largely by squeaking out a barely-over-1% win in Cleveland's home state of New York; in fact, had Cleveland won his home state, he would have won the electoral vote by a count of 204-197 (201 votes then needed for victory). Note, though, that Cleveland earned 24 of his electoral votes in states that he won by less than 1% (Connecticut, Virginia, and West Virginia).\",\n",
" \"Some of Cleveland's actions were controversial with political factions. Such criticisms include but are not limited to: his intervention in the Pullman Strike of 1894 in order to keep the railroads moving (a move which angered labor unions), his support of the gold standard, and opposition to free silver which alienated the agrarian wing of the Democrats. Furthermore, critics complained that he had little imagination and seemed overwhelmed by the nation's economic disasters depressions and strikes in his second term. He lost control of his party to the agrarians and silverites in 1896.\"\n",
" ],\n",
" 'dataset': 'rag_dataset',\n",
" 'datum_uid': 'query0',\n",
" 'prediction': \"Cleveland's opponents in 1884 criticized his alleged involvement in a scandal regarding an illegitimate child, which was used to counter his innocent image during the presidential campaign.\"\n",
" },\n",
" },\n",
" {\n",
" 'type': 'Hallucination',\n",
" 'value': 1.0,\n",
" 'parameters': {\n",
" 'dataset': 'rag_dataset',\n",
" 'datum_uid': 'query0',\n",
" 'prediction': \"Cleveland's opponents in 1884 criticized his alleged involvement in a scandal regarding an illegitimate child, which was used to counter his innocent image during the presidential campaign.\",\n",
" 'context_list': [\n",
" \"Cleveland was defeated in the 1888 presidential election, in part due to fraud (See Blocks of Five). He actually led in the popular vote over Benjamin Harrison (48.6% to 47.8%), but Harrison won the Electoral College by a 233-168 margin, largely by squeaking out a barely-over-1% win in Cleveland's home state of New York; in fact, had Cleveland won his home state, he would have won the electoral vote by a count of 204-197 (201 votes then needed for victory). Note, though, that Cleveland earned 24 of his electoral votes in states that he won by less than 1% (Connecticut, Virginia, and West Virginia).\",\n",
" \"Some of Cleveland's actions were controversial with political factions. Such criticisms include but are not limited to: his intervention in the Pullman Strike of 1894 in order to keep the railroads moving (a move which angered labor unions), his support of the gold standard, and opposition to free silver which alienated the agrarian wing of the Democrats. Furthermore, critics complained that he had little imagination and seemed overwhelmed by the nation's economic disasters depressions and strikes in his second term. He lost control of his party to the agrarians and silverites in 1896.\"\n",
" ],\n",
" 'dataset': 'rag_dataset',\n",
" 'datum_uid': 'query0',\n",
" 'prediction': \"Cleveland's opponents in 1884 criticized his alleged involvement in a scandal regarding an illegitimate child, which was used to counter his innocent image during the presidential campaign.\"\n",
" },\n",
" },\n",
" {\n",
Expand Down Expand Up @@ -641,15 +689,15 @@
" 'prediction': 'British taekwondo fighter Aaron Cook plans to compete for Moldova at the 2016 Olympics in Rio after being overlooked for the Great Britain squad in London 2012. Cook received funding from a Moldovan billionaire and has now obtained Moldovan citizenship. He has decided to no longer compete for Great Britain due to feeling overlooked and unsupported, and hopes to represent Moldova at international competitions, including the Olympics. The British Olympic Association could potentially block this move, as discussions are ongoing.'\n",
" },\n",
" },\n",
" {\n",
" 'type': 'Coherence',\n",
" 'value': 5.0,\n",
" 'parameters': {\n",
" 'dataset': 'summarization_dataset',\n",
" 'datum_uid': 'article4',\n",
" 'prediction': 'British taekwondo fighter Aaron Cook plans to compete for Moldova at the 2016 Olympics in Rio after being overlooked for the Great Britain squad in London 2012. Cook received funding from a Moldovan billionaire and has now obtained Moldovan citizenship. He has decided to no longer compete for Great Britain due to feeling overlooked and unsupported, and hopes to represent Moldova at international competitions, including the Olympics. The British Olympic Association could potentially block this move, as discussions are ongoing.'\n",
" },\n",
" },\n",
" # {\n",
" # 'type': 'Coherence',\n",
" # 'value': 5.0,\n",
" # 'parameters': {\n",
" # 'dataset': 'summarization_dataset',\n",
" # 'datum_uid': 'article4',\n",
" # 'prediction': 'British taekwondo fighter Aaron Cook plans to compete for Moldova at the 2016 Olympics in Rio after being overlooked for the Great Britain squad in London 2012. Cook received funding from a Moldovan billionaire and has now obtained Moldovan citizenship. He has decided to no longer compete for Great Britain due to feeling overlooked and unsupported, and hopes to represent Moldova at international competitions, including the Olympics. The British Olympic Association could potentially block this move, as discussions are ongoing.'\n",
" # },\n",
" # },\n",
" {\n",
" 'type': 'ROUGE',\n",
" 'value': {\n",
Expand Down Expand Up @@ -837,7 +885,6 @@
" dataset,\n",
" metrics_to_return=[\n",
" \"Bias\",\n",
" \"Coherence\",\n",
" \"Toxicity\",\n",
" ],\n",
" llm_api_params = {\n",
Expand Down Expand Up @@ -882,33 +929,33 @@
"[Your Name]\"\"\"\n",
" },\n",
" },\n",
" {\n",
" \"value\": 5.0,\n",
" \"type\": \"Coherence\",\n",
" \"parameters\": {\n",
" \"dataset\": \"content_generation_dataset\",\n",
" \"datum_uid\": \"query2\",\n",
" \"prediction\": \"\"\"Subject: Project Delay Due to Funding Cuts\n",
"# {\n",
"# \"value\": 5.0,\n",
"# \"type\": \"Coherence\",\n",
"# \"parameters\": {\n",
"# \"dataset\": \"content_generation_dataset\",\n",
"# \"datum_uid\": \"query2\",\n",
"# \"prediction\": \"\"\"Subject: Project Delay Due to Funding Cuts\n",
"\n",
"Dear [Coworker's Name],\n",
"# Dear [Coworker's Name],\n",
"\n",
"I hope this message finds you well. I am writing to update you on the status of our project and unfortunately, convey some disappointing news regarding a delay in its completion.\n",
"# I hope this message finds you well. I am writing to update you on the status of our project and unfortunately, convey some disappointing news regarding a delay in its completion.\n",
"\n",
"Due to recent funding cuts within our department, our project team has been significantly affected. Several team members, including myself, have been relocated to work on other projects to address the shifting priorities resulting from the budget constraints.\n",
"# Due to recent funding cuts within our department, our project team has been significantly affected. Several team members, including myself, have been relocated to work on other projects to address the shifting priorities resulting from the budget constraints.\n",
"\n",
"As a consequence of these unexpected changes, it is with regret that I must inform you that the original deadline for our project will need to be extended. I understand the inconvenience that this may cause, and I sincerely apologize for any inconvenience this delay may bring to you and your plans.\n",
"# As a consequence of these unexpected changes, it is with regret that I must inform you that the original deadline for our project will need to be extended. I understand the inconvenience that this may cause, and I sincerely apologize for any inconvenience this delay may bring to you and your plans.\n",
"\n",
"Rest assured that despite this setback, I am fully committed to ensuring that we still deliver the project with utmost efficiency and quality. I am exploring all possible avenues to mitigate the delay and work towards completing our project in a timely manner.\n",
"# Rest assured that despite this setback, I am fully committed to ensuring that we still deliver the project with utmost efficiency and quality. I am exploring all possible avenues to mitigate the delay and work towards completing our project in a timely manner.\n",
"\n",
"I appreciate your understanding and patience during this challenging time. Your ongoing support and collaboration are invaluable as we navigate through this situation together. If you have any concerns or questions, please do not hesitate to reach out to me.\n",
"# I appreciate your understanding and patience during this challenging time. Your ongoing support and collaboration are invaluable as we navigate through this situation together. If you have any concerns or questions, please do not hesitate to reach out to me.\n",
"\n",
"Thank you for your understanding, and I look forward to working with you to successfully finalize our project.\n",
"# Thank you for your understanding, and I look forward to working with you to successfully finalize our project.\n",
"\n",
"Warm regards,\n",
"# Warm regards,\n",
"\n",
"[Your Name]\"\"\",\n",
" },\n",
" },\n",
"# [Your Name]\"\"\",\n",
"# },\n",
"# },\n",
" {\n",
" 'type': 'Toxicity',\n",
" 'value': 0.0,\n",
Expand Down

0 comments on commit 7770c33

Please sign in to comment.