Benchmarks
Benchmarks that reflect the work, not just the answer.
Each benchmark is listed from the day it is planned, with its status. A result appears only with its methodology. Until a benchmark is released, this page shows the agenda and the method, never a chart.
The agenda
BenchmarkPlanned
Workflow evaluation suite
At least 200 labelled workflow scenarios, with documented methodology and held-out cases.
BenchmarkPlanned
Agent permission and misuse suite
Denied-action tests, cross-tenant checks, injected instructions, approval-parameter mutation, credential revocation.
BenchmarkPlanned
Execution recovery suite
Timeouts, stale data, retry duplication, partial writes, checkpoint restart, external API failure.
BenchmarkPlanned
Inference and cost-per-accepted-outcome benchmark
Quality, latency and cost across model choices and routing, at fixed acceptance criteria.
BenchmarkPending permission
Client-measured results
Baseline, outcome and limits from approved engagements.
Required fields when a benchmark is released
- task population and whether synthetic or authorised real data
- task and scenario versions
- systems, models, configurations and versions tested
- baseline, including a deterministic alternative where applicable
- who created and reviewed the ground truth
- definitions of success, severe failure and partial success
- run counts, seeds, uncertainty, hidden hold-out
- what pass@k does and does not imply
- environment artefacts and reproduction instructions
- cost assumptions, date and limitations
- security and privacy restrictions on traces and datasets