Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ragas with Llama-Index not working correctly due to misdefined scope #1695

Open
georgije-conv opened this issue Nov 21, 2024 · 1 comment
Open
Labels
bug Something isn't working module-testsetgen Module testset generation

Comments

@georgije-conv
Copy link

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When trying to generate a test set using generate_with_llamaindex_docs, the code fails with a NameError indicating that 'LCDocument' is not defined. This appears to be an internal import issue in the Ragas library where the Langchain Document class is not properly imported in the scope where it's being used.

Ragas version: 0.2.6
Python version: 3.10.11

Code to Reproduce

from llama_index.core import SimpleDirectoryReader
from ragas.testset import TestsetGenerator
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Load documents
documents = SimpleDirectoryReader("./nyc_wikipedia").load_data()

# Initialize generator components
generator_llm = OpenAI(model="gpt-4")
embeddings = OpenAIEmbedding(model="text-embedding-3-large")

# Create generator
generator = TestsetGenerator.from_llama_index(
    llm=generator_llm,
    embedding_model=embeddings,
)

# Generate testset
testset = generator.generate_with_llamaindex_docs(
    documents,
    testset_size=5,
)

Error trace

NameError                                 Traceback (most recent call last)
Cell In[4], line 2
      1 # generate testset
----> 2 testset = generator.generate_with_llamaindex_docs(
      3     documents,
      4     testset_size=5,
      5 )

File [...]/ragas/testset/synthesizers/generate.py:241, in TestsetGenerator.generate_with_llamaindex_docs(self, documents, testset_size, transforms, transforms_llm, transforms_embedding_model, query_distribution, run_config, callbacks, with_debugging_logs, raise_exceptions)
    240     transforms = default_transforms(
--> 241         documents=[LCDocument(page_content=doc.text) for doc in documents],
    242         llm=llm_for_transforms,
    243         embedding_model=embedding_model_for_transforms,
    244     )

NameError: name 'LCDocument' is not defined

Expected behavior
I expected it to generate a test set like described in the guide.

@georgije-conv georgije-conv added the bug Something isn't working label Nov 21, 2024
@dosubot dosubot bot added the module-testsetgen Module testset generation label Nov 21, 2024
@amanchaudhary-95
Copy link

Yes, I'm also facing the same issue with LlamaIndex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working module-testsetgen Module testset generation
Projects
None yet
Development

No branches or pull requests

2 participants