---
title: "Evaluation standard | Chainfore"
url: https://chainfore.com/research/standard
updated: 2026-10-22
summary: "We publish a result with its method, dataset, baseline, repeat count and limits. Until then, this page shows the standard we hold ourselves to, not a claim."
---

Evaluation standard

# Numbers only when they are measured.

We publish a result with its method, dataset, baseline, repeat count and limits. Until then, this page shows the standard we hold ourselves to, not a claim.

## How we measure

- Task success: All mandatory business conditions pass. Severe failures are reported separately.

- Permission compliance: Forbidden accesses or actions that were attempted, blocked or completed.

- Consistency: Repeated trials per task, so one success is not mistaken for reliable behaviour.

- Recovery: Success after a controlled timeout, stale record, tool error or checkpoint restart.

- Human burden: Review time, interventions, escalations and rework.

- Economics: Total operating expense divided by accepted outcomes, including failed runs.

- Latency: Completion-time distribution under a stated load, with p50 and p95 where meaningful.

## What zero failures does and does not show

With zero failures in 300 independent, representative trials, the exact one-sided 95% upper bound on the failure rate is about 1%. This is arithmetic on a sample size, not a measured result on any client workflow.

Limit: Real tasks may be correlated or unrepresentative. The correct statement is "zero observed failures on this suite under these conditions", not "safe".

## How to read a Chainfore number

Every figure we publish carries five fields: method, dataset or task distribution, baseline, repeat count, and limits. If a figure you see from us is missing one, tell us; it should not have been published.

method · dataset or task distribution · baseline · repeat count · limits

## Questions buyers ask

### What does Chainfore measure for an agent in production?

Seven measures: task success, permission compliance, consistency, recovery, human burden, economics and latency.

### What does zero failures in 300 trials show?

The exact one-sided 95% upper bound on the failure rate is about 1%. It is arithmetic on a sample size, not a measured result on any client workflow; the correct statement is "zero observed failures on this suite under these conditions".

### What must every published Chainfore figure include?

Its method, dataset or task distribution, baseline, repeat count and limits.

## Work with the programme

If you have a workflow and a question, write to research@chainfore.com.

Research collaboration: https://chainfore.com/contact?area=research-collaboration