---
title: "Benchmarks | Chainfore"
url: https://chainfore.com/research/benchmarks
updated: 2026-10-22
summary: "Each benchmark is listed from the day it is planned, with its status. A result appears only with its methodology. Until a benchmark is released, this page shows the agenda and the method, never a chart."
---

Benchmarks

# Benchmarks that reflect the work, not just the answer.

Each benchmark is listed from the day it is planned, with its status. A result appears only with its methodology. Until a benchmark is released, this page shows the agenda and the method, never a chart.

## The agenda

- Workflow evaluation suite (Planned): At least 200 labelled workflow scenarios, with documented methodology and held-out cases.

- Agent permission and misuse suite (Planned): Denied-action tests, cross-tenant checks, injected instructions, approval-parameter mutation, credential revocation.

- Execution recovery suite (Planned): Timeouts, stale data, retry duplication, partial writes, checkpoint restart, external API failure.

- Inference and cost-per-accepted-outcome benchmark (Planned): Quality, latency and cost across model choices and routing, at fixed acceptance criteria.

- Client-measured results (Pending permission): Baseline, outcome and limits from approved engagements.

## Required fields when a benchmark is released

- task population and whether synthetic or authorised real data

- task and scenario versions

- systems, models, configurations and versions tested

- baseline, including a deterministic alternative where applicable

- who created and reviewed the ground truth

- definitions of success, severe failure and partial success

- run counts, seeds, uncertainty, hidden hold-out

- what pass@k does and does not imply

- environment artefacts and reproduction instructions

- cost assumptions, date and limitations

- security and privacy restrictions on traces and datasets