forked from kookmin-sw/cap-template
-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ai readme 수정
- Loading branch information
Showing
1 changed file
with
69 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,27 +1,80 @@ | ||
| # Chat bot `KUKU` 소개 | ||
| # WorkFlow Overview | ||
|
|
||
| 국민대에 관한 모든 것을 물어보세요 ! | ||
| 국민대 관련 데이터를 기반으로 답변을 생성하는 챗봇 <쿠쿠>입니다. | ||
| **외국민 KUKU (OURS)** | ||
|
|
||
| # 학습한 데이터 | ||
|  | ||
|
|
||
| - [국민대 2023 요람 PDF](https://www.kookmin.ac.kr/comm/cmfile/thumbnail2.do?encSvrFileNm=223d2bdfdbb4df30ad85271267bd6e6a0a913159736ab9843bb76fc00eeeb5ddb1e88f35002b9cf1b749bfe96b751f16b8be21ad5273d348a74b10a57513dd4540bbcb178d3151db4d507c693a1f7ef9&encFileGrpSeq=8e8e9041def64eb5f7f8c21154bcff06&encFileSeq=cf9f1626435aafc6e0e182b36c8e23d9) | ||
| - [2023~2024.03.28 국민대 전체 공지사항](https://www.kookmin.ac.kr/user/kmuNews/notice/index.do) | ||
|
|
||
| # 실행 방법 | ||
| **Simple RAG** | ||
|
|
||
|  | ||
|
|
||
| 2024_03_30 아나콘다 환경 python 3.8 에서 실행 확인 | ||
| # Metrics | ||
|  | ||
| [RAGAS](https://docs.ragas.io/en/stable/concepts/metrics/index.html) | ||
|
|
||
| 1. /ai 경로로 작업 디렉토리 이동 | ||
| 2. 패키지 설치 `pip install -r requirements.txt` | ||
| 3. /ai 경로에 .env 파일 생성 (OPENAI_API_KEY = 'your_api') | ||
| 4. `python run_chatbot.py` | ||
| Ragas is a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. RAG denotes a class of LLM applications that use external data to augment the LLM’s context. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in. | ||
|
|
||
| # TEST | ||
|
|
||
|  | ||
| > __비교 모델__ (Comparison model) | ||
| - 외국민 KUKU (OURS) | ||
| - SimpleRAG | ||
| - ChatGPT(gpt-3.5-turbo) | ||
| - ON국민 쿠민이 | ||
|
|
||
|  | ||
|
|
||
|  | ||
| ## Faithfulness | ||
|
|
||
| This measures the factual consistency of the generated answer against the given context. It is calculated from answer and retrieved context. The answer is scaled to (0,1) range. Higher the better. | ||
| The generated answer is regarded as faithful if all the claims that are made in the answer can be inferred from the given context. To calculate this a set of claims from the generated answer is first identified. Then each one of these claims are cross checked with given context to determine if it can be inferred from given context or not. The faithfulness score is given by divided by | ||
|
|
||
|  | ||
| - Faithfulness는 챗봇의 답변 비율이 얼마나 사실에 기반했는지 평가하는 지표이다. | ||
| - 0~1 범위의 값을 가지며 높을 수록 좋다 | ||
|
|
||
|  | ||
|  | ||
|
|
||
| - 디스코드 봇을 통해 국민대 19학번으로 구성된 ‘외국민’ 팀원이 직접 사실관계를 평가하여 측정 | ||
|
|
||
|
|
||
| ## Answer Relevancy | ||
|
|
||
| The evaluation metric, Answer Relevancy, focuses on assessing how pertinent the generated answer is to the given prompt. A lower score is assigned to answers that are incomplete or contain redundant information and higher scores indicate better relevancy. This metric is computed using the question, the context and the answer. | ||
|
|
||
|  | ||
|  | ||
|
|
||
| The Answer Relevancy is defined as the mean cosine similartiy of the original question to a number of artifical questions, which where generated (reverse engineered) based on the answer | ||
|
|
||
| - Answer Relevance은 생성된 답변이 주어진 프롬프트와 얼마나 적절한지를 평가하는 데 초점을 맞춘다. | ||
| - 불완전하거나 중복 정보가 포함된 답변에는 낮은 점수가 할당되고, 더 높은 점수는 더 좋은 적합성을 나타낸다. | ||
| - 0~1 범위의 값을 가지며 높을 수록 좋다 | ||
|
|
||
|
|
||
|  | ||
|
|
||
| Answer Relevance는 Reverse Engineering을 통해 계산된다. 순서는 다음과 같다. | ||
| 1. 챗봇에게 질문 후 답변을 받는다. | ||
| 2. 답변을 llm의 입력으로 넣어 질문을 예측하게 한다. | ||
| 3. 예측된 질문과 원래의 질문의 유사도(코사인 유사도)를 비교한다. | ||
|
|
||
| # Latency | ||
| - 챗봇에게 쿼리를 보낸 후 답변 응답까지 걸린 평균 소요 시간(sec) | ||
| - 낮을 수록 좋다. | ||
|
|
||
|
|
||
|  | ||
|
|
||
|
|
||
| # Test Sample | ||
|
|
||
|  | ||
|  | ||
|
|
||
| # Test Log 일부 | ||
|
|
||
| `chatbot_result.xlsx` | ||
|
|
||
|  | ||
|
|