AI Agent Evidence Benchmark 2026: baseline methodology

AI-service claims are often communicated as dashboards, screenshots or self-reported logs. Those formats may be useful operationally, but they make it difficult to determine which policy was applied, which evidence was supplied and whether the same inputs still produce the same result.

The benchmark therefore fixes the scenario inputs, expected outcome and verification sequence. A valid run must expose the exact command, environment, API response and receipt digest so another developer can challenge the result rather than merely trust a summary.

The healthy scenario remains within the published latency, uptime and freshness thresholds. The latency-breach scenario changes the measured response time while preserving the other availability signals. The output-integrity scenario preserves availability but supplies an observed digest that differs from the expected digest.

This structure tests whether each failure remains attributable to the rule that caused it. It does not attempt to benchmark model intelligence, response quality, benchmark contamination or the truth of an upstream measurement source.

A reviewable submission should identify the source commit, runtime version, command, execution timestamp, endpoint, complete bounded output and three receipt digests. Any modification, timeout, schema deviation or failed verification belongs in the record.

PactVerity will count an external run only after the supplied evidence can be reproduced. Applications, statements of intent and screenshots without the required record are not included as verified runs.

As published on 18 August 2026, the first-party suite contains three reference scenarios and the verified external-run count is zero. That zero is intentional: it prevents interest in the benchmark from being presented as completed independent validation.

Future editions can report reproducibility rates, environment differences, API-schema failures and reviewer findings only after the underlying records are available for inspection.

Download the public developer kit, inspect the scenario file, run the dry mode and then execute the explicit live mode when ready. Do not include credentials, private customer data or unredacted vulnerability information.

The useful contribution is not a positive result at any cost. A reproducible failure, compatibility problem or ambiguity in the evidence model is equally valuable when it is documented precisely.