September 2026: Agentic Testing Environments Monthly Benchmark Review

Executive readout

The leaders are a tie, not a near tie. Anthropic Fable 5.1 and OpenAI GPT-6 Astra both score 85.1 on the category-normalized index, and their six reported category scores are identical in this release. Anthropic Opus 5 follows at 83.4, then OpenAI GPT-5.6 Sol at 81.6.

The practical question is not only which curve is highest. It is where a model sits once cost, task duration, and the type of engineering work are considered together. That is why this report keeps the full category profile visible instead of declaring a winner from one aggregate number.

The source file contains validation index, average cost per task, and average task duration. It does not contain reliability/completion outcomes, prior-month results, error bars, confidence intervals, SUT latency, or fully versioned harness/configuration fields. This report therefore makes no reliability, significance, or month-over-month claims.

What the benchmark asks an agent to do

The unit under evaluation is a complete engineering task. A good textual answer cannot compensate for a wrong verdict, missing evidence, an unsafe action, or an uncleared test state.

The reported index is category-normalized: each of the six workflow scores contributes equally to the overall value. That prevents industrial data analysis, which has more scheduled cells in the source, from dominating the monthly headline.

The field at a glance

Industrial data analysis produces the highest reported scores in the release. V-cycle verification produces the lowest reported category scores. That observation describes this task mix; it does not establish that one family is inherently harder, nor that any difference is statistically significant.

Deployment guidance: three practical starting points

The joint leaders are suitable reference points for teams pursuing the highest observed validation index. For day-to-day deployment, the choice is more specific.

Sonnet 5 is the balanced Anthropic default in this release. It exceeds Terra in every reported category by 0.8–1.0 points, while costing $0.0616 less per task; its average task duration is 0.1 minute longer. It is the clearest starting point when teams want a strong index without paying the joint-leader price.

GPT-5.6 Terra is the recommended OpenAI-family baseline for teams standardizing on that family. Its 78.2 index and 6.0-minute average duration place it close to Sonnet’s profile, even though GPT-5.6 Sol is stronger on the supplied index and lower on cost. Terra should therefore be selected for OpenAI-family continuity and operating preference, not presented as the data leader.

GLM 5.3 Flash is the very-low-cost candidate: roughly 34.5× lower reported cost than Sonnet, with a 13.0-point category-normalized index tradeoff. For teams evaluating an open-weight path, it is the user-designated candidate in this release; licensing and deployment status should be independently confirmed before treating it as an open-weight option.

Category review

Industrial data analysis

Industrial data analysis asks an agent to discover a supplied BTMS recording, decode the available artifact, locate and quantify a seeded deviation, trace its root cause, cite evidence, and abstain correctly on a clean control. Detection is a gate: a missed seeded issue or false positive earns zero. Correct detections are weighted across detection (35%), localization (20%, ±500 ms), quantification (20%, ±15%), and root cause with cited evidence (25%).

Fable 5.1 and GPT-6 Astra tie at 93.9. Opus 5 follows at 92.2, GPT-5.6 Sol at 90.2, Sonnet 5 at 87.4, Terra at 86.4, and GLM 5.3 Flash at 73.0. This is the release’s highest-scoring category, but the three input encodings have identical supplied results per model/scenario, so the report does not claim a measured format-robustness advantage.

Automated test flows

Automated test flows evaluates whether an agent can turn a black-box requirement into a safe verification strategy, use a coherent timebase, perform live control where required, and return an evidence-backed verdict. The scenarios cover interlock timing and hysteresis, a charge-thermal watchdog that needs a command-refresh loop, and a clean-control false-positive check. Wrong or unsupported verdicts score zero; successful runs are weighted across correctness (40%), evidence window (20%), requirement citation (20%), and faithful observed behavior (20%).

Fable 5.1 and GPT-6 Astra tie at 83.5; Sonnet 5 is 77.5, Terra 76.7, and GLM 5.3 Flash 64.7. The category has the largest observed model spread in this release, so it is especially useful when a deployment needs live timing and evidence discipline rather than only offline reasoning.

Calibration & software

Calibration and software work tests baseline inspection, an explicitly permitted change, exact readback, behavioral verification with evidence, and restoration to the baseline. The published weighting is authorized change selection (25%), safety and evidence discipline (20%), exact readback (20%), behavioral verification (20%), and restoration evidence (15%). An unauthorized or out-of-range write, unsafe control action, or non-baseline cleanup is a hard zero.

Fable 5.1 and GPT-6 Astra tie at 87.3. Sonnet 5 reaches 81.0, Terra 80.1, and GLM 5.3 Flash 67.6. The supporting category documentation describes CS-01 as simulator-validated while other scenarios await smoke coverage, so this number should be read as the current calibration-change workflow result, not as a claim of full production calibration maturity.

Real-time stream validation

Real-time stream validation covers live thresholds, timing, and event sequences through Nexus. An agent must select a coherent timestamp source and an appropriate observation window, assess the required threshold, rate, or sequence, preserve a reviewable trace, return a supported terminal verdict, and restore the test state.

Fable 5.1 and GPT-6 Astra tie at 84.7; Sonnet 5 is 78.6, Terra 77.8, and GLM 5.3 Flash 65.6. The category README still marks fixtures and tolerances as planned for freeze, so the number is reported as the release’s stream-validation category score—not as a certified real-time performance result.

Safety & evidence quality

Safety and evidence quality evaluates appropriately bounded decisions: respecting operating limits, stopping when further action is unsafe, separating observations from assumptions and hypotheses, choosing a justified terminal state, preserving provenance, and giving an actionable next step.

Fable 5.1 and GPT-6 Astra tie at 85.4. Sonnet 5 reaches 79.3, Terra 78.5, and GLM 5.3 Flash 66.3. The supporting scope is still documented as planned, so this aggregate must not be represented as a certified safety or reliability result. It is a useful behavioral readout for the current benchmark corpus and nothing more.

V-cycle verification

V-cycle verification is the end-to-end path from requirement to observable acceptance condition, test strategy, bounded execution, evidence collection, verdict, cleanup, and next action. The intended coverage spans runtime behavior, diagnostics, calibration-dependent behavior, interfaces and timing, clean controls, and unavailable dependencies.

Fable 5.1 and GPT-6 Astra tie at 75.6, the lowest reported leader score of the six categories. Sonnet 5 reaches 70.2, Terra 69.4, and GLM 5.3 Flash 58.6. The category’s fixtures, weights, and tolerances are still described as awaiting freeze, so this is a report of the current V-cycle category result rather than a general claim about every migrated-ECU validation program.

How to use this release

For methodology and scoring definitions, see the Agentic Test & Validation Protocol. The solution link below takes the next step: turning a chosen model profile into a bounded, repeatable engineering workflow.