Support Instance Level Images for SWE-Bench Evaluation

[Jiayi-Pan]

What is the problem that this fixes or functionality that this introduces? Does it fix any open issues?
Give a brief summary of what the PR does, explaining any non-trivial design decisions

We were limited to using the large, all-in-one ghcr.io/opendevin/eval-swe-bench:full-v1.2.1 for SWE-bench-lite evaluation, which is both heavy and inflexible.

This PR introduces support for using instance-level images for SWE-bench evaluation, which complements our previous approach. Each SWE-bench instance is now built into its own image (following SWE-bench's official format), enabling greater granularity and extensibility.

I've tested on 10 instances so far and plan to test on the entire swe-bench-lite and verify its effectivness.

Other references

[xingyaoww]

Mostly LGTM! Thanks for this!

[xingyaoww]

I can also help run an eval to validate the effectiveness of this when this PR is ready :)

[Jiayi-Pan]

All comments are addressed.

I can also help run an eval to validate the effectiveness of this when this PR is ready :)

Yeah, that would be helpful! Since I'm running out of my disk space for storing the images 😢

[xingyaoww]

LGTM! Let me run an eval first before merging it in!

[xingyaoww]

@Jiayi-Pan There seems to be some issue with exiting git diff commands.. did you run into those in your tests?

Also, I think instance_report won't work here, maybe we can disable instance_report for instance-level docker image?

[Jiayi-Pan]

Thanks, I've disabled instance_report for instance images. The git diff issue should be fixed after this
#2878

[xingyaoww]

LGTM! Pending evaluation - set to "request changes" in case of accidental merge.

[xingyaoww]

For some reason, the evaluation results are not as good (64/300) compared to the baseline results (74/300): https://huggingface.co/spaces/OpenDevin/evaluation/tree/main/outputs/swe_bench_lite/CodeActAgent/claude-3-5-sonnet%4020240620_maxiter_30_N_v1.8-no-hint

I've attached the evaluated outputs: claude-3-5-sonnet@20240620_maxiter_30_N_v1.8-no-hint-pr2874.tar.gz

Feel free to compare & quickly check the instances that failed.

[neubig]

I assigned you to this @xingyaoww , but feel free to reassign if you think someone else is more appropriate.

[xingyaoww]

After some offline discussion with Jiayi, i re-ran the eval and got 76/300 (25.3%)! Good to merge now!