Add bad_output_process_model option and use_fixed_model_version option for all generation methods, to avoid future OpenAI API changes break Sotopia running.
#196
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…generate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly.
yangalan123
changed the title
use_fixed_model_version option for all generation methods, to avoid future OpenAI API changes break Sotopia running.bad_output_process_model option and use_fixed_model_version option for all generation methods, to avoid future OpenAI API changes break Sotopia running.
Codecov ReportAttention: Patch coverage is
@@ Coverage Diff @@
## release #196 +/- ##
===========================================
+ Coverage 69.33% 69.37% +0.03%
===========================================
Files 56 56
Lines 2971 2971
===========================================
+ Hits 2060 2061 +1
+ Misses 911 910 -1
|
|
@XuhuiZhou @ProKil Hi, this is my promised PR previously. And I think by merging this PR, now the user would be able to run Sotopia with their own specified OpenAI API versions, and API deprecation will no longer affect Sotopia from running (by setting |
ProKil
approved these changes
Sep 24, 2024
ProKil
added a commit
that referenced
this pull request
Oct 1, 2024
* bump the version, test release to PyPi * Update README.md * Update README.md * Update README.md * bumpy version to 0.0.9 * Update Sotopia presentation information in README.md * bump version to 0.0.10 * bump version * add merge release back to main action * change checkout v4->v3 * fix merge-back-to-main and pin mypy to <1.11.0 * merge bug fix * upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183) * update pull request -> pull request target * bump version * Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196) * Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly. * [autofix.ci] apply automated fixes --------- Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix gpt-3.5 * replace gpt3.5 turbo for tests * update gpt-3.5-turbo to gpt-4o-mini * bug fix for return fixed model version function --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
XuhuiZhou
added a commit
that referenced
this pull request
Oct 4, 2024
* bump the version, test release to PyPi * Update README.md * Update README.md * Update README.md * bumpy version to 0.0.9 * Update Sotopia presentation information in README.md * bump version to 0.0.10 * bump version * add merge release back to main action * change checkout v4->v3 * fix merge-back-to-main and pin mypy to <1.11.0 * merge bug fix * upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183) * update pull request -> pull request target * bump version * Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196) * Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly. * [autofix.ci] apply automated fixes --------- Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix gpt-3.5 * replace gpt3.5 turbo for tests * update gpt-3.5-turbo to gpt-4o-mini * bug fix for return fixed model version function * fix sampling error * fix rc.4 * new tag --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
XuhuiZhou
added a commit
that referenced
this pull request
Oct 14, 2024
* bump the version, test release to PyPi * Update README.md * Update README.md * Update README.md * bumpy version to 0.0.9 * Update Sotopia presentation information in README.md * bump version to 0.0.10 * bump version * add merge release back to main action * change checkout v4->v3 * fix merge-back-to-main and pin mypy to <1.11.0 * merge bug fix * upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183) * update pull request -> pull request target * bump version * Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196) * Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly. * [autofix.ci] apply automated fixes --------- Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix gpt-3.5 * replace gpt3.5 turbo for tests * update gpt-3.5-turbo to gpt-4o-mini * bug fix for return fixed model version function * fix sampling error * fix rc.4 * new tag * bump version * update workflow permission * add why sotopia * improve the why sotopia --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
XuhuiZhou
added a commit
that referenced
this pull request
Oct 19, 2024
* bump the version, test release to PyPi * Update README.md * Update README.md * Update README.md * bumpy version to 0.0.9 * Update Sotopia presentation information in README.md * bump version to 0.0.10 * bump version * add merge release back to main action * change checkout v4->v3 * fix merge-back-to-main and pin mypy to <1.11.0 * merge bug fix * upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183) * update pull request -> pull request target * bump version * Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196) * Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly. * [autofix.ci] apply automated fixes --------- Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix gpt-3.5 * replace gpt3.5 turbo for tests * update gpt-3.5-turbo to gpt-4o-mini * bug fix for return fixed model version function * fix sampling error * fix rc.4 * new tag * bump version * update workflow permission * add why sotopia * improve the why sotopia * bump version * further add clarification to the custom models --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
XuhuiZhou
added a commit
that referenced
this pull request
Jan 8, 2025
* Update langchain requirement from <0.3.0,>=0.2.5 to >=0.2.5,<0.4.0 (#236) * [Automated] Merge release into main (#235) * bump the version, test release to PyPi * Update README.md * Update README.md * Update README.md * bumpy version to 0.0.9 * Update Sotopia presentation information in README.md * bump version to 0.0.10 * bump version * add merge release back to main action * change checkout v4->v3 * fix merge-back-to-main and pin mypy to <1.11.0 * merge bug fix * upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183) * update pull request -> pull request target * bump version * Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196) * Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly. * [autofix.ci] apply automated fixes --------- Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix gpt-3.5 * replace gpt3.5 turbo for tests * update gpt-3.5-turbo to gpt-4o-mini * bug fix for return fixed model version function * fix sampling error * fix rc.4 * new tag * bump version * update workflow permission * add why sotopia * improve the why sotopia * bump version * further add clarification to the custom models --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * update macos test runner as macos-latest (#238) * update macos test runner as macos-latest * redis-stack-server * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(exp_eval): support tag for combo iteration (#245) * support tag for combo iteration * fix pre-commit error * [autofix.ci] apply automated fixes * fix mypy error --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * feat: Initial Message for LLM agnets in Sotopia Aact (Experimental) (#249) * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Rebasing --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: Adding Chat print node for pretty printing chat conversation between agents (#250) * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Adding scene context for agents * [autofix.ci] apply automated fixes * Added chat print node * [autofix.ci] apply automated fixes * Fixing the runtime channel * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Adding scene context for agents * Added chat print node * [autofix.ci] apply automated fixes * Moving chat node to experiment * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * feat: Initial Message for LLM agnets in Sotopia Aact (Experimental) (#249) * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Rebasing --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding scene context for agents * Added chat print node * [autofix.ci] apply automated fixes * Adding scene context for agents * [autofix.ci] apply automated fixes * Fixing the runtime channel * Moving chat node to experiment * Updating the toml file paths --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * doc: API endpoints for sotopia (#242) * api doc * add PUT * add an temp example for websocket * websocket * update readme * Update README.md * update websocket live simulation api doc * [autofix.ci] apply automated fixes * update websocket doc * add api server with websocket as well as a client * fix mypy errors * support stopping the chat * add 404 to the status code * fix mypy issue * remove files * update requirement * update return message from the server * add restful error code * Update README.md * remove paks --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: FastAPI Implementation of Sotopia Part One (wo websocket) (#246) * api doc * add PUT * add an temp example for websocket * websocket * update readme * Update README.md * fastapi server wo websocket * update project toml * add websocket api and a sample client sample * add initial test * post update * finalize the doc * fix the create * finish test * add files * change chata to api * fix mypy error * fix mypy bug * fix mypy * create mock agents * downgrade langchain openai upperbound to fix CI --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> * update the instruction to load existing data via docker (#255) * fix doc (#258) * Sotopia API and UI (#264) * feat: FastAPI Implementation of Sotopia Part Two (w websocket) (#252) * api doc * add PUT * add an temp example for websocket * websocket * update readme * Update README.md * update websocket live simulation api doc * [autofix.ci] apply automated fixes * update websocket doc * add api server with websocket as well as a client * fix mypy errors * support stopping the chat * add 404 to the status code * fix mypy issue * update the returned message types * redesign websocket api * update websocket, fix mypy error * add example of using websocket * clean code & change to existing functions for simulation * fix typing mismatch * update doc & mypy type fix * add type check for run_async_server * move example --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Add customizable evaluation dimensions (#256) * add customizable evaluation dimensions * add docs * fix mypy error & refactor examples * add docs for evaluation dimensions * update docs and examples * add test cases and fix mypy issue * fix mypy issue * Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) (#262) Co-authored-by: openhands <[email protected]> * Fix/custom eval dimension test (#263) * Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) * Update documentation for SotopiaDimension and EvaluationDimensionBuilder * [autofix.ci] apply automated fixes * Add API documentation for evaluation dimensions * Refine API documentation for evaluation_dimensions.py to match style * [autofix.ci] apply automated fixes --------- Co-authored-by: openhands <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * add doc --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: openhands <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Feat/addtional fast apis for non-streaming simulation and managing relationshio (#265) * temp run * add relationship api * fix mypy error * update relationship api * simulate episode non-streaming * modify sim episodes * add simulation status * task error * add background task * [autofix.ci] apply automated fixes * back to arun one episode * upload the code * use rq to execute background tasks * temp sol --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix ci error * solving pytests * improve the tests * add custom eval fast api (#268) * fix mypy error * aact moderator (#257) * initial framework * initial conv * fix module error * feat: Add 3 new features to Moderator (#266) * feat:introduce booting procedure, saving, and ending chat to moderator * fix: moderator will now ignore none AgentAction, Observations now don't have to include all channels in the mapping * merge changes of example into the original one * fix: 1. save() method now accepts push_to_db config 2. booting()'s waiting time is changed to 0.1 sec * fix: rewrite booting() so that different agent will receive different background information * fix: moderator now inherits from Node directly, instead of from BaseAgent --------- Co-authored-by: JXZhou <JXZhou> * add save condition for moderator * push to db false * to fully stop * stopping all agents * fix mypy * fix mypy error --------- Co-authored-by: JXZhou <[email protected]> * Deploy the api to modal (#267) * prototype for modal serving * add openai secret * fix type annotation * add doc * bug fix for simulation api * add customize model, evaluator model and evaluation dimensions * Implement modal API server with Redis integration and FastAPI setup - Added a new script for the modal API server that initializes a Redis instance. - Created a persistent volume for Redis data and included a function to download initial data if not present. - Configured a Docker image with necessary dependencies including Redis Stack and FastAPI. - Implemented a web API class that sets up and cleans up the Redis connection, ensuring readiness before serving requests. - Integrated the SotopiaFastAPI application within the modal framework. --------- Co-authored-by: XuhuiZhou <[email protected]> * Feature/sotopia demo UI (#261) * initial * initial ui * merge main * add new ui * switch to fastAPI * websocket check * fix render episode error * add page; make a simplified page and still WIP * [autofix.ci] apply automated fixes * fix simplified streaming version * semi-done character page + avatar assets * Fixed character card styling * [autofix.ci] apply automated fixes * unified rendering and chat display * updated chat character icons * add some tags * add typing * temp fix * add characters avatar to simulation * fix episode full avatar * go to modal config * clean up code * add modal streamlit app * clean codebase except websocket * remove repeated local css * clean websocket * fix get name error * fix errors * pre render scenario * add custom eval * change streamlit to dynamic path * new uv * revert to previous install commands * a fix for modal * add customized dimension * [autofix.ci] apply automated fixes * sort scenarios in simulation * for demo video * update deploy instruction * update intro page * update intro page * [autofix.ci] apply automated fixes * update intro page * add customized dimensions * update api link and modal environment * move folder * fix relative import * update modal image build * use uv to build environment * change folder name * change test * fix modal serve * environment change * refactor * fix ui --------- Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: astrophie <[email protected]> * remove dev tag * add custom eval * base dimension * fix ui mypy * fix mypy * add delete dimension * update streamlit ui * ignores the ui directory * Committing changes before push * pytest for eval dimension * fix mypy * clean up comments * run non streaming simulation * add pytest for websocket * fix mypy issue --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: openhands <[email protected]> Co-authored-by: JXZhou <[email protected]> Co-authored-by: astrophie <[email protected]> --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Haofei Yu <[email protected]> Co-authored-by: Arpandeep Khatua <[email protected]> Co-authored-by: Arpandeep Khatua <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: openhands <[email protected]> Co-authored-by: JXZhou <[email protected]> Co-authored-by: astrophie <[email protected]>
ProKil
added a commit
that referenced
this pull request
Jan 9, 2025
* bump the version, test release to PyPi * Update README.md * Update README.md * Update README.md * bumpy version to 0.0.9 * Update Sotopia presentation information in README.md * bump version to 0.0.10 * bump version * add merge release back to main action * change checkout v4->v3 * fix merge-back-to-main and pin mypy to <1.11.0 * merge bug fix * upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183) * update pull request -> pull request target * bump version * Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196) * Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly. * [autofix.ci] apply automated fixes --------- Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix gpt-3.5 * replace gpt3.5 turbo for tests * update gpt-3.5-turbo to gpt-4o-mini * bug fix for return fixed model version function * fix sampling error * fix rc.4 * new tag * bump version * update workflow permission * add why sotopia * improve the why sotopia * bump version * further add clarification to the custom models * UI update (#273) * Update langchain requirement from <0.3.0,>=0.2.5 to >=0.2.5,<0.4.0 (#236) * [Automated] Merge release into main (#235) * bump the version, test release to PyPi * Update README.md * Update README.md * Update README.md * bumpy version to 0.0.9 * Update Sotopia presentation information in README.md * bump version to 0.0.10 * bump version * add merge release back to main action * change checkout v4->v3 * fix merge-back-to-main and pin mypy to <1.11.0 * merge bug fix * upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183) * update pull request -> pull request target * bump version * Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196) * Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly. * [autofix.ci] apply automated fixes --------- Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix gpt-3.5 * replace gpt3.5 turbo for tests * update gpt-3.5-turbo to gpt-4o-mini * bug fix for return fixed model version function * fix sampling error * fix rc.4 * new tag * bump version * update workflow permission * add why sotopia * improve the why sotopia * bump version * further add clarification to the custom models --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * update macos test runner as macos-latest (#238) * update macos test runner as macos-latest * redis-stack-server * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(exp_eval): support tag for combo iteration (#245) * support tag for combo iteration * fix pre-commit error * [autofix.ci] apply automated fixes * fix mypy error --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * feat: Initial Message for LLM agnets in Sotopia Aact (Experimental) (#249) * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Rebasing --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: Adding Chat print node for pretty printing chat conversation between agents (#250) * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Adding scene context for agents * [autofix.ci] apply automated fixes * Added chat print node * [autofix.ci] apply automated fixes * Fixing the runtime channel * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Adding scene context for agents * Added chat print node * [autofix.ci] apply automated fixes * Moving chat node to experiment * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * feat: Initial Message for LLM agnets in Sotopia Aact (Experimental) (#249) * Adding OpenHands node * [autofix.ci] apply automated fixes * Added LLM Agent Node * [autofix.ci] apply automated fixes * removing openhands * [autofix.ci] apply automated fixes * Correcting mypy error in tick agent * Moving everything to examples since its not specific to sotopia * [autofix.ci] apply automated fixes * Adding some additional final checks * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Renaming to interview openhands * Adding documentation * [autofix.ci] apply automated fixes * Updating the readme * [autofix.ci] apply automated fixes * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding the initial message channel * Rebasing --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Adding scene context for agents * [autofix.ci] apply automated fixes * Renaming scene context to initial message * Adding scene context for agents * Added chat print node * [autofix.ci] apply automated fixes * Adding scene context for agents * [autofix.ci] apply automated fixes * Fixing the runtime channel * Moving chat node to experiment * Updating the toml file paths --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * doc: API endpoints for sotopia (#242) * api doc * add PUT * add an temp example for websocket * websocket * update readme * Update README.md * update websocket live simulation api doc * [autofix.ci] apply automated fixes * update websocket doc * add api server with websocket as well as a client * fix mypy errors * support stopping the chat * add 404 to the status code * fix mypy issue * remove files * update requirement * update return message from the server * add restful error code * Update README.md * remove paks --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: FastAPI Implementation of Sotopia Part One (wo websocket) (#246) * api doc * add PUT * add an temp example for websocket * websocket * update readme * Update README.md * fastapi server wo websocket * update project toml * add websocket api and a sample client sample * add initial test * post update * finalize the doc * fix the create * finish test * add files * change chata to api * fix mypy error * fix mypy bug * fix mypy * create mock agents * downgrade langchain openai upperbound to fix CI --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> * update the instruction to load existing data via docker (#255) * fix doc (#258) * Sotopia API and UI (#264) * feat: FastAPI Implementation of Sotopia Part Two (w websocket) (#252) * api doc * add PUT * add an temp example for websocket * websocket * update readme * Update README.md * update websocket live simulation api doc * [autofix.ci] apply automated fixes * update websocket doc * add api server with websocket as well as a client * fix mypy errors * support stopping the chat * add 404 to the status code * fix mypy issue * update the returned message types * redesign websocket api * update websocket, fix mypy error * add example of using websocket * clean code & change to existing functions for simulation * fix typing mismatch * update doc & mypy type fix * add type check for run_async_server * move example --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Add customizable evaluation dimensions (#256) * add customizable evaluation dimensions * add docs * fix mypy error & refactor examples * add docs for evaluation dimensions * update docs and examples * add test cases and fix mypy issue * fix mypy issue * Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) (#262) Co-authored-by: openhands <[email protected]> * Fix/custom eval dimension test (#263) * Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) * Update documentation for SotopiaDimension and EvaluationDimensionBuilder * [autofix.ci] apply automated fixes * Add API documentation for evaluation dimensions * Refine API documentation for evaluation_dimensions.py to match style * [autofix.ci] apply automated fixes --------- Co-authored-by: openhands <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * add doc --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: openhands <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Feat/addtional fast apis for non-streaming simulation and managing relationshio (#265) * temp run * add relationship api * fix mypy error * update relationship api * simulate episode non-streaming * modify sim episodes * add simulation status * task error * add background task * [autofix.ci] apply automated fixes * back to arun one episode * upload the code * use rq to execute background tasks * temp sol --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix ci error * solving pytests * improve the tests * add custom eval fast api (#268) * fix mypy error * aact moderator (#257) * initial framework * initial conv * fix module error * feat: Add 3 new features to Moderator (#266) * feat:introduce booting procedure, saving, and ending chat to moderator * fix: moderator will now ignore none AgentAction, Observations now don't have to include all channels in the mapping * merge changes of example into the original one * fix: 1. save() method now accepts push_to_db config 2. booting()'s waiting time is changed to 0.1 sec * fix: rewrite booting() so that different agent will receive different background information * fix: moderator now inherits from Node directly, instead of from BaseAgent --------- Co-authored-by: JXZhou <JXZhou> * add save condition for moderator * push to db false * to fully stop * stopping all agents * fix mypy * fix mypy error --------- Co-authored-by: JXZhou <[email protected]> * Deploy the api to modal (#267) * prototype for modal serving * add openai secret * fix type annotation * add doc * bug fix for simulation api * add customize model, evaluator model and evaluation dimensions * Implement modal API server with Redis integration and FastAPI setup - Added a new script for the modal API server that initializes a Redis instance. - Created a persistent volume for Redis data and included a function to download initial data if not present. - Configured a Docker image with necessary dependencies including Redis Stack and FastAPI. - Implemented a web API class that sets up and cleans up the Redis connection, ensuring readiness before serving requests. - Integrated the SotopiaFastAPI application within the modal framework. --------- Co-authored-by: XuhuiZhou <[email protected]> * Feature/sotopia demo UI (#261) * initial * initial ui * merge main * add new ui * switch to fastAPI * websocket check * fix render episode error * add page; make a simplified page and still WIP * [autofix.ci] apply automated fixes * fix simplified streaming version * semi-done character page + avatar assets * Fixed character card styling * [autofix.ci] apply automated fixes * unified rendering and chat display * updated chat character icons * add some tags * add typing * temp fix * add characters avatar to simulation * fix episode full avatar * go to modal config * clean up code * add modal streamlit app * clean codebase except websocket * remove repeated local css * clean websocket * fix get name error * fix errors * pre render scenario * add custom eval * change streamlit to dynamic path * new uv * revert to previous install commands * a fix for modal * add customized dimension * [autofix.ci] apply automated fixes * sort scenarios in simulation * for demo video * update deploy instruction * update intro page * update intro page * [autofix.ci] apply automated fixes * update intro page * add customized dimensions * update api link and modal environment * move folder * fix relative import * update modal image build * use uv to build environment * change folder name * change test * fix modal serve * environment change * refactor * fix ui --------- Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: astrophie <[email protected]> * remove dev tag * add custom eval * base dimension * fix ui mypy * fix mypy * add delete dimension * update streamlit ui * ignores the ui directory * Committing changes before push * pytest for eval dimension * fix mypy * clean up comments * run non streaming simulation * add pytest for websocket * fix mypy issue --------- Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: openhands <[email protected]> Co-authored-by: JXZhou <[email protected]> Co-authored-by: astrophie <[email protected]> --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hao Zhu <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Haofei Yu <[email protected]> Co-authored-by: Arpandeep Khatua <[email protected]> Co-authored-by: Arpandeep Khatua <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: openhands <[email protected]> Co-authored-by: JXZhou <[email protected]> Co-authored-by: astrophie <[email protected]> * bump release version --------- Co-authored-by: XuhuiZhou <[email protected]> Co-authored-by: Chenghao (Alan) Yang <[email protected]> Co-authored-by: Chenghao Yang <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Haofei Yu <[email protected]> Co-authored-by: Arpandeep Khatua <[email protected]> Co-authored-by: Arpandeep Khatua <[email protected]> Co-authored-by: Zhe Su <[email protected]> Co-authored-by: openhands <[email protected]> Co-authored-by: JXZhou <[email protected]> Co-authored-by: astrophie <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #
📑 Description
Two major updates:
add "bad_output_process_model" option to all
agenerate_xxx()methods so users can decide which model to use for handling bad outputs. By default, this is set to begpt-4o-mini.add
use_fixed_model_versionoption for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error.Document (
generation.md) has been updated for these two major changes correspondingly.✅ Checks
type/descript(e.g.feature/add-llm-agents)ℹ Additional Information