Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TapeAgents export for experiment results #238

Merged
merged 22 commits into from
Nov 15, 2024

Conversation

ollmer
Copy link
Contributor

@ollmer ollmer commented Nov 4, 2024

This PR introduces a simple compatibility layer for exporting traces of the BrowserGym experiments in a format compatible with the Tapes of the TapeAgents framework.
What does it do:

  • convert a single trace of the experiment into a single Tape
  • decomposes each StepInfo step of the experiment into 3 steps of the tape agents: observation, thought, and action.
  • attempts to preserve as much metadata as possible for the step and for the whole trace.
  • parses action in the simplest possible way as a string with a single Python function call
  • if the think field is present in the agent info and not empty - it creates the thought step of the tape from it.
  • method to save the tape in json format

The MVP is ready but not yet tested.

Next steps in this PR:

  • test conversion over the real traces
  • collect feedback from the BrowserGym team
  • test tape loading on the TapeAgents side and rendering it with the tape browser

cc @recursix.

@ollmer ollmer changed the title [WIP] TapeAgents compatibility layer for experiment results TapeAgents export for experiment results Nov 8, 2024
@ollmer ollmer marked this pull request as ready for review November 8, 2024 15:49
@ollmer
Copy link
Contributor Author

ollmer commented Nov 8, 2024

I've simplified this PR down to <100 lines of changes in a single file and added support of tape_segments inside the agent_info, which could be produced by TapeAgent in the AgentLab.

Copy link
Collaborator

@gasse gasse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just made a few comments. Feel free to ping me for a chat @ollmer

browsergym/experiments/src/browsergym/experiments/loop.py Outdated Show resolved Hide resolved
browsergym/experiments/src/browsergym/experiments/loop.py Outdated Show resolved Hide resolved
browsergym/experiments/src/browsergym/experiments/loop.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@gasse gasse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gasse gasse merged commit 6a2a649 into ServiceNow:main Nov 15, 2024
10 checks passed
@ollmer
Copy link
Contributor Author

ollmer commented Nov 15, 2024

Thank you!

@ollmer ollmer deleted the tapes_integration branch November 15, 2024 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants