LABS
ENVIRONMENTS
How the agent acts, and what it’s graded on. Pinned repositories in Docker today, and working apps on an emulator or simulator next, each reset to a known state for every attempt.
SWE-BENCH LINEAGE
The agent works inside a pinned codebase with a shell, reads the issue, and changes the code.
ANDROIDWORLD LINEAGE · IN DEVELOPMENT
The agent operates a working app on an emulator or simulator, the way a person does.
Working apps an agent operates the way a person does: on an Android emulator or iOS simulator, through taps, typing, and navigation. The grade comes from what actually changed in the app and its backend, not from what the agent says it did.
Scope a mobile environmentProduction-grade apps with realistic users, data, and history, reset to the same starting state for every attempt.
Multi-step flows across screens: onboarding, checkout, settings, offline use, permissions, and deep links.
Deterministic checks on the app database and backend records, with partial credit only where steps are truly separable.
We design the apps, tasks, and verifiers around the behaviors you want to train or measure.
We take on repository-level work in mainstream stacks. Where we go furthest is where we’ve shipped the most.
| Area | What we build tasks around |
|---|---|
| Flutter and Dart | State management, async and lifecycle bugs, offline sync, routing and deep links, package upgrades |
| Android | Kotlin and Jetpack Compose, Gradle builds, permissions, background work |
| iOS | Swift and SwiftUI, Xcode builds, app lifecycle, platform APIs |
| Shopify | Storefronts, apps, checkout and payment flows, inventory logic |
| Web and backend | APIs, data layers, and the services mobile apps depend on |
Public coding benchmarks are mostly Python and web libraries. Mobile apps, with their builds, lifecycles, and platform rules, barely appear. Where they’ve been measured, agents struggle.
