TASKS

Open a task,
end to end.

Every task ships with its instruction, its environment, hidden tests, and measured results. Here’s one.

FlutterOffline syncExpert-authored

Offline expenses upload twice after a partial sync failure

A Flutter expense app loses track of what it already uploaded when the server rejects one item.

WHAT THE AGENT RECEIVES, WORD FOR WORD.

Expenses created offline get uploaded twice. Repro: go offline, add three expenses, have the server reject the second one, reconnect, then sync again later. Expected: expenses 1 and 3 are uploaded once; expense 2 stays queued with its error and uploads on the next sync. Actual: expense 1 is uploaded again on the next sync. Do not change SyncService’s public API.

Shows up late

The bug appears only on the second sync, after a partial server failure.

No file pointer

The report describes user behavior, not the code to change.

A tempting wrong fix

Retrying the whole batch passes a naive test and fails ours.

A fixed boundary

The sync service’s public API can’t change, so the fix has to live inside it.

WHAT SHIPS WITH IT.

  • instruction.mdThe text above
  • task.tomlMetadata, timeouts, resources
  • environment/DockerfilePinned Flutter SDK, warmed package cache, network off
  • solution/solve.shReference fix. Private, never shown to the agent
  • tests/test.shGrading script. Private
  • tests/hidden/*_test.dartHidden tests. Private

HOW IT’S GRADED.

  1. Restore the original test folder, undoing any agent edits.
  2. Copy in the hidden tests.
  3. Run the full suite. Reward is 1 only if everything passes.
test('partial failure never re-uploads synced items', () async {
  final api = FakeExpenseApi(rejectIds: {'e2'});
  final queue = InMemorySyncQueue([exp('e1'), exp('e2'), exp('e3')]);
  final sync = SyncService(api: api, queue: queue);

  await sync.run();
  api.rejectIds.clear();
  await sync.run();

  expect(api.uploadCount('e1'), 1);
  expect(api.uploadCount('e2'), 1);
  expect(await queue.pending(), isEmpty);
});

MEASURED RESULTS.

Calibration results, pass rates by agent and model over repeated runs, ship with every task set. This public sample isn’t calibrated yet.

How a task
is built.

The same seven steps for every task. People do the judgment. Automation does the repetition.

  1. 01
    Manual

    Author

    A senior engineer writes the bug or feature, the issue text, and the reference fix.

  2. 02
    Manual

    Hidden tests

    Tests that fail on the starting code and pass with the fix. Kept away from the agent.

  3. 03
    Automated

    Gold and empty runs

    The fix passes on every repeated run. Doing nothing fails.

  4. 04
    Manual

    Blind solve

    A second engineer solves it from the issue alone, and logs the time.

  5. 05
    Manual + automated

    Adversarial patches

    Plausible wrong fixes and test tampering. All of them must fail.

  6. 06
    Automated

    Calibration

    Frontier agents attempt it many times. We keep tasks that are hard but solvable.

  7. 07
    Automated

    Package and version

    Pinned environment, task files, hashes, and results, frozen together.

One task,
three uses.

The same task, with its hidden tests and trajectories, feeds all three.

SFT

Expert solve trajectories, replayable and labeled.

RL

Deterministic rewards from hidden tests, pass/fail or partial credit.

Evaluation

Pass rates, traces, and measured difficulty per task and model.

Request sample tasks.

Request sample tasks