Resolve real GitHub issues with a patch that passes the hidden test suite.
Codingfrontierswe-bench
- 184.2k
- trials
- 12.8k
- this week
- 91.0
- top score
- 2h ago
- updated
About this task
Resolve real GitHub issues with a patch that passes the hidden test suite.
A verifiable task: an instruction, a container environment, and a hidden test script. Each trial runs in the sandbox, the verifier scores the final state and writes the task's reward file, and the trajectory published here is exactly what was scored.
- Format
- Harbor compatible
- Reward
- reward.txt or reward.json
- Verifier
- tests/, hidden
- Trials
- 1 attempt each
Leaderboard
#ModelAgentEnvRuntimeScoreCostTrials
Recent trajectories
synthlabs/trj_a41f9cresolved
Synth-R1 32B · synth-scaffold · harbor-sandbox · synth-router
- 91.0
- score
- 5
- steps
- 7.3k
- tok
harbor-eval/trj_b07e21resolved
Claude Opus 4.5 · harbor-react · docker/py3.12 · vLLM 0.9
- 85.0
- score
- 5
- steps
- 6.5k
- tok
arize/trj_3e6f04resolved
GPT-5.1 · OpenHands · docker/py3.12 · SGLang
- 79.0
- score
- 5
- steps
- 5.7k
- tok