Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT support langsmith for tracing #11

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

leonj1
Copy link
Contributor

@leonj1 leonj1 commented Dec 26, 2024

Fixes #9

Since this is a langgraph project, then it's possible to use langsmith to see what's happening.
I do not know how to use langsmith (yet), but there is a connection now if needed for debugging.

Screenshot 2024-12-25 at 8 21 59 PM

@ai-christianson
Copy link
Owner

Looks like some conflicts (hopefully minor) after merging the logging PR.

I think the feature idea itself makes a ton of sense because right now we have zero visibility into the language model calls themselves.

Basically has been manual prompt tuning and making sure is works in real world scenarios.

This feature will give us visibility into the LLM calls which will be key for the bigger picture.

I'm going to put another ticket for benchmarking on SWE bench. Between langsmith visibility, logging, and benchmarks, we'll have a much more solid foundation for making sustainable progress.

@leonj1
Copy link
Contributor Author

leonj1 commented Dec 26, 2024

I'm unsure if I should proceed with PR since I don't know if langgraph is meant to integrate with langsmith. I have found no article that gives me confidence in this intent. I'd rather not force an integration that's not meant to be.

I fixed the conflicts if you still think it's reasonable to merge this.

@ai-christianson
Copy link
Owner

ai-christianson commented Dec 26, 2024

Yeah that's fair.

For now we can leave this issue open and treat it more like an exploration/research thread and figure out if it's something worth fully integrating.

Edit: but the general idea of tracing and visibility is a good one. It can get to be fairly opaque what is actually going on under the hood with all the prompting.

The visibility would also be instrumental in optimizing the agent for token use as well as making it work better with other models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Web UI
2 participants