| Model | Mode | Acc | No answer | Total | Reason Lens |
|---|---|---|---|---|---|
| o1-preview-2024-09-12 | greedy | 92.84 | 0.61 | 2778 | 557.91 |
| claude-3-5-sonnet-20241022 | greedy | 88.91 | 0 | 2778 | 787.31 |
| chatgpt-4o-latest-24-09-07 | greedy | 88.88 | 0.07 | 2778 | 622.27 |
| gpt-4o-2024-08-06 | greedy | 88.26 | 0.04 | 2778 | 691.73 |
| gpt-4o-2024-05-13 | greedy | 88.01 | 0.14 | 2778 | 629.79 |
| o1-mini-2024-09-12 | greedy | 86.72 | 0.32 | 2778 | 464.2 |
| Llama-3.1-405B-Inst@sambanova | greedy | 86.21 | 0.25 | 2778 | 629.07 |
| gemini-1.5-pro-exp-0827 | greedy | 86.14 | 1.26 | 2778 | 858.08 |
| claude-3-5-sonnet-20240620 | greedy | 86 | 0.18 | 2778 | 907.1 |
| Llama-3.1-405B-Inst-fp8@together | greedy | 85.64 | 0.76 | 2778 | 449.71 |
| Qwen2.5-72B-Instruct | greedy | 85.57 | 0 | 2778 | 667.07 |
| gemini-1.5-pro-exp-0801 | greedy | 85.53 | 1.22 | 2778 | 656.64 |
| gpt-4-turbo-2024-04-09 | greedy | 85.31 | 0.04 | 2778 | 631.38 |
| Llama-3.1-405B-Inst@hyperbolic | greedy | 84.41 | 1.15 | 2778 | 563.13 |
| Qwen2.5-32B-Instruct | greedy | 83.66 | 0.04 | 2778 | 626.06 |
| Meta-Llama-3.1-70B-Instruct | greedy | 82.97 | 1.26 | 2778 | 668.14 |
| Mistral-Large-2 | greedy | 82.97 | 0.14 | 2778 | 490.94 |
| gemini-1.5-pro | greedy | 82.76 | 1.94 | 2778 | 666.7 |
| claude-3-opus-20240229 | greedy | 82.54 | 0.58 | 2778 | 500.35 |
| yi-large-preview | greedy | 82.15 | 0.14 | 2778 | 982.6 |
| gemini-1.5-flash-exp-0827 | greedy | 82.11 | 1.48 | 2778 | 794.53 |
| gpt-4-0314 | greedy | 81.64 | 0.04 | 2778 | 397.22 |
| Qwen2-72B-Instruct | greedy | 81.61 | 0.29 | 2778 | 486.41 |
| gpt-4o-mini-2024-07-18 | greedy | 81.5 | 0.07 | 2778 | 526 |
| yi-large | greedy | 81.17 | 0 | 2778 | 774.85 |
| deepseek-v2-chat-0628 | greedy | 80.81 | 0.11 | 2778 | 691.91 |
| deepseek-v2.5-0908 | greedy | 80.35 | 0.22 | 2778 | 641.15 |
| deepseek-v2-coder-0724 | greedy | 80.24 | 0.32 | 2778 | 650.65 |
| deepseek-v2-coder-0614 | greedy | 79.63 | 0.14 | 2778 | 704.72 |
| claude-3-5-haiku-20241022 | greedy | 79.63 | 0 | 2778 | 788.72 |
| Meta-Llama-3-70B-Instruct | greedy | 78.01 | 0.11 | 2778 | 520.77 |
| gemini-1.5-flash | greedy | 77.36 | 1.26 | 2778 | 583.45 |
| Athene-70B | greedy | 76.64 | 0.04 | 2778 | 552.61 |
| reka-core-20240501 | greedy | 76.42 | 0.76 | 2778 | 701.67 |
| gemma-2-27b-it | greedy | 75.67 | 0.61 | 2778 | 446.51 |
| Qwen2.5-7B-Instruct | greedy | 75.13 | 0.11 | 2778 | 759.05 |
| claude-3-sonnet-20240229 | greedy | 74.87 | 0.07 | 2778 | 671.75 |
| gemma-2-9b-it | greedy | 72.82 | 0.76 | 2778 | 499 |
| Yi-1.5-34B-Chat | greedy | 72.79 | 1.01 | 2778 | 620.1 |
| claude-3-haiku-20240307 | greedy | 72.32 | 0.04 | 2778 | 644.59 |
| Phi-3-mini-4k-instruct | greedy | 70.34 | 0.43 | 2778 | 677.09 |
| command-r-plus | greedy | 68.61 | 0 | 2778 | 401.51 |
| gpt-3.5-turbo-0125 | greedy | 68.36 | 0.04 | 2778 | 357.92 |
| Phi-3.5-mini-instruct | greedy | 67.67 | 4.5 | 2778 | 899.49 |
| Meta-Llama-3.1-8B-Instruct | greedy | 67.24 | 4.25 | 2778 | 732.7 |
| Qwen2-7B-Instruct | greedy | 66.92 | 0.72 | 2778 | 533.15 |
| Mistral-Nemo-Instruct-2407 | greedy | 66.88 | 0.47 | 2778 | 464.19 |
| Yi-1.5-9B-Chat | greedy | 65.05 | 4.61 | 2778 | 542.87 |
| reka-flash-20240226 | greedy | 64.72 | 0.32 | 2778 | 659.25 |
| Qwen2.5-3B-Instruct | greedy | 64.25 | 0.72 | 2778 | 708.95 |
| Mixtral-8x7B-Instruct-v0.1 | greedy | 63.17 | 5.51 | 2778 | 324.31 |
| Meta-Llama-3-8B-Instruct | greedy | 61.66 | 0.97 | 2778 | 600.81 |
| command-r | greedy | 61.12 | 0.04 | 2778 | 382.23 |
| Llama-3-Instruct-8B-SimPO-v0.2 | greedy | 55.22 | 1.19 | 2778 | 450.6 |
| gemma-2-2b-it | greedy | 51.94 | 2.59 | 2778 | 554.74 |
| Qwen2-1.5B-Instruct | greedy | 41.11 | 7.74 | 2778 | 280.56 |
| Rex-v0.1-1.5B | greedy | 39.09 | 4.79 | 2778 | 329.35 |