DEFAULT
Pass or fail
1 only if every target test passes and nothing regresses. Closest to how a reviewer would judge it.
GRADING
High reward should mean
the task was solved.
A grader that can be gamed teaches the wrong lesson. Ours are built to resist it.
01 / SEVEN GUARANTEESBUILT TO RESIST REWARD HACKING
- 01
Hidden tests
Agents never see the tests that decide the reward.
- 02
Fail first, then pass
Every target test fails on the starting code and passes with the reference solution.
- 03
No regressions
The existing test suite has to stay green.
- 04
Tamper-resistant
Test files are restored before grading. Network is off. Repository history after the start point is removed.
- 05
Adversarial checks
We submit plausible wrong patches. Every one has to fail.
- 06
Repeatability
The reference solution passes on every repeated run, or the task is rejected.
- 07
Calibrated difficulty
We run frontier agents many times per task and keep tasks that are hard but solvable.
ON REQUEST
Partial credit
The share of target tests passed, gated on no regressions. A smoother signal for RL.
FORMATS
Drops into your stack.
Need something else? We adapt to your harness.
A FEW GOOD QUESTIONS
Two practical questions.
What formats do you deliver?
Harbor task directories, SWE-bench-compatible JSONL, ATIF trajectories, and Docker images. We adapt to your harness.
Where is the team?
Engineers and QA work from the US, Europe, Central Asia, and Latin America. Access, data handling, and confidentiality terms are agreed before any project starts.