adding a script to fetch and convert devin's output for evaluation #81
Conversation
There was a problem hiding this comment.
How about we put this file to SWE-Bench/scripts?
There was a problem hiding this comment.
I'm not quite sure about this. It seems more reasonable to keep dataset related files in the dataset folder to me. @JustinLin610 @libowen2121 any thoughts on this?
There was a problem hiding this comment.
Ohh i suggest we do this: mv src/prepare_devin_outputs_for_evaluation.py scripts/prepare_devin_outputs_for_evaluation.py
There was a problem hiding this comment.
oh my bad, I thought we are moving it outside the evaluation folder. Will do
evaluation/README.md
Outdated
| - src | ||
| - `prepare_devin_outputs_for_evaluation.py`: script fetching and converting devin's output into the desired json file for evaluation. | ||
| - outputs: two json files under `evaluation/SWE-bench/data/` that can be directly used for evaluation |
There was a problem hiding this comment.
Can you upload the post-processed file to our huggingface datasets, and add curl or wget command here so people can directly download those for debugging? You can request to join if you haven't already: https://huggingface.co/OpenDevin
|
|
||
| with open(os.path.join(output_dir, "fail_output.json"), "w") as fail_file: | ||
| json.dump(failed_files_info, fail_file, indent=4) | ||
|
|
There was a problem hiding this comment.
I'm debating whether we want to make this two separate files, or just one file -- how about we merge them into one, and add an additional bool field like devin_pass?
There was a problem hiding this comment.
It only takes ~1 minute to fetch and process the files. The purpose of having two files is you can directly start from the passed files for pilot testing. I can generate another merged file and upload it to HF
There was a problem hiding this comment.
having both options is a good option! maybe we can add an argument in the script to switch that behavior; and we can upload both version to HF and have user decide which one they want to download
* a starting point for SWE-Bench evaluation with docker * fix the swe-bench uid issue * typo fixed * fix conda missing issue * move files based on new PR * Update doc and gitignore using devin prediction file from #81 * fix typo * add a sentence * fix typo in path * fix path --------- Co-authored-by: Binyuan Hui <[email protected]>
…ll-Hands-AI#81) * adding code to fetch and convert devin's output for evaluation * update README.md * update code for fetching and processing devin's outputs * update code for fetching and processing devin's outputs
* a starting point for SWE-Bench evaluation with docker * fix the swe-bench uid issue * typo fixed * fix conda missing issue * move files based on new PR * Update doc and gitignore using devin prediction file from All-Hands-AI#81 * fix typo * add a sentence * fix typo in path * fix path --------- Co-authored-by: Binyuan Hui <[email protected]>
No description provided.