[Bug]: eval-this workflow is not working

[neubig]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Describe the bug and reproduction steps

Currently the eval-this workflow is not working, so we should fix it.

Idea:

• Switch LM to Claude Haiku
• Reduce to use a subset of SWE-bench instances to make it affordable
• Run and make sure that it works

@csmith49 will take a look at this

OpenHands Installation

Docker command in README

OpenHands Version

No response

Operating System

None

Logs, Errors, Screenshots, and Additional Context

No response

[enyst]

The eval-this workflow has two parts:

• actual evaluation
• integration tests.

openhands-agent has split them here:

• Fix issue #5076: Integration test github action #5077

I can confirm that the new integration tests workflow works with Haiku:

• Integration tests (openhands fix issue 5076) enyst/playground#8 (comment)
• (browsing ones fail, I'm looking into that, that's a good thing in a way!)

I felt like we need the integration tests in their new form to be back in working state. We have removed them at some point from ./tests, and refactored them as external scripts like evals, also using Deepseek, but they weren't working either.

What do you think about this? Could we have a nightly for them - and maybe a label too, just in case needed -, also with Haiku?

IMHO it would be cool if we can also have a nightly on Deepseek or something. Because

• Haiku has native function calling
• Deepseek doesn't, so the runs use different prompt/code/conversion/pydantic serialization/etc (it really affects stuff IMHO)

These integration tests are just like 6 tests currently (and I'm working on a seventh), but they do try to cover some things in real-like use that we just don't have coverage elsewhere.

[enyst]

Oh, also: at this time, the Deepseek API key defined on this repo is depleted. I doubt it's the original reason why the eval workflow wasn't working, but it looks like the first reason currently. 😅 Cc: @neubig

[enyst]

This is my proposal on these: (source)

[neubig]

Thanks so much for digging in to this @enyst!
I unfortunately am a bit short on time to look at this, but @mamoodi if you'd be able to take a look I'd love your comments.