Hey folks,
In the first post about the Tactix agent architecture, we wrote that evaluation was how we planned to recover some of the determinism we gave up when we moved from traditional models to agents.
That sentence was directionally right and operationally incomplete.
The version in our heads was simple: save a good response, run the agent again, and compare the new answer to the old one. If they match closely enough, pass. If they do not, fail.
Then we built it and ran real cases.
The agent failed some of them. The more important discovery was that some of the cases failed too. A historical response could contain the wrong baseline. An approved label could disagree with the source evidence. A reference answer could sound exactly right while making a claim its own data could not prove.
That changed the job of the suite. It could not just ask, “Did the agent reproduce the answer we liked?” It had to ask a harder question:
Does this output satisfy a behavioral contract whose facts can be reconstructed from independent, frozen evidence?
The system we ended up with replays the actual Tactix agents in an isolated evaluation mode, serves their Model Context Protocol (MCP) calls from case-local snapshots, checks deterministic claims, scores the output across five dimensions, and publishes an immutable report tied to the exact code, image, cases, and evidence that produced it.
The established output-agent baseline now passes 50 out of 50 cases: 20 Anomaly Detection, 15 Critical Insights, and 15 Smart Actions. An independently forced Scheduler run passed the same 50 cases and delivered its Slack summary.
That is the result. The useful part of the story is why getting there required us to stop treating a saved answer as truth.
A service can complete successfully and still return the wrong number, label, explanation, action, or shape.
The first oracle was too easy to trust
Before the suite, we checked quality through a mix of unit tests, live data inspection, developer judgment, and feedback from operators. All of those checks still matter. None of them gives you a stable regression contract for a probabilistic system.
An HTTP 200 says the service returned something. A schema test says the response had the right fields. A green pipeline says the source data moved. None of them says the current value is correct, the comparison window is aligned, the alert classification is approved, or the narrative has not implied causation the evidence does not support.
So we started collecting what looked like good historical examples. Each case had a real prompt, a real agent response, and the MCP evidence used at the time. We thought of the response as the oracle.
That was the mistake.
A reference response is useful as a reviewed example of the behavior we want. It is not independent evidence that its own numbers are correct. If the response says revenue was 18% below baseline, the response cannot also be the proof that the baseline and percentage were right.
The first full replay passed 37 of 50 cases. We tightened the contracts, rebuilt the catalog, and the next run passed 36.
The score went down. The suite got better.
The stricter pass exposed cases whose numbers could not be reconstructed cleanly, labels inherited from older schemas, and subjective fields that had been promoted into factual checks. A looser evaluator would have called more of those cases green. A trustworthy one had to admit that it did not yet know.
The reference response shows the behavior we want. Independent evidence decides whether its claims deserve to become a regression contract.
What a golden case actually contains
We now treat a golden case as a versioned, content-bound contract. It contains:
- the redacted prompt and the business date the question refers to,
- frozen MCP requests and responses,
- deterministic calculations and invariants,
- a reviewed reference response,
- required and forbidden behaviors,
- the model and tool context needed for replay,
- and provenance for the exact content revision.
The state model matters as much as the payload.
Candidate means the evidence has been captured and redacted, but nobody has made a review decision. Approved means an authenticated reviewer accepted that exact revision under the current working policy. Trusted means the approval, evidence, reference, temporal alignment, deterministic proof, and content identity all agree.
Approval alone is intentionally insufficient.
If the prompt changes, the content identity changes. If the MCP snapshot changes, the identity changes. If a deterministic check changes, the identity changes. An approval for the old case cannot silently authorize the new one.
This sounds bureaucratic until you picture the failure it prevents. Someone fixes an evidence snapshot, leaves the approved flag in place, and the suite now reports confidence in a case nobody actually reviewed. Content-bound trust makes that impossible by design: the approval belongs to the exact contract, not to a friendly case ID.
The same discipline applies to redaction and isolation. Golden cases are built from DEV data, stored in a DEV-only bucket, and rejected if the runner points at a non-DEV environment. The replay can exercise real agent logic without writing memories, facts, product outputs, or operational state.
The run exercises the real thing
The lifecycle has six stages: capture, review, freeze, replay, evaluate, publish.
Capture makes the behavior observable. Review makes it intentional. Freeze makes it replayable. Evaluation makes it measurable. Publishing makes it explainable later.
An authenticated portal user can choose one case, an exact set, an agent family, or the established 50-case output-agent suite. The portal starts a Cloud Run Job and records the execution identity. The job downloads only the selected, approved, trusted cases and verifies their checksums and coverage policy before invoking an agent.
Each agent runs in evaluation mode. Its memory and writes are isolated. When it calls MCP, a frozen case-local server returns the captured evidence instead of reaching into mutable product data. The result goes through deterministic checks and the shared scorer, then the runner publishes an immutable bundle with the case results, summary, latency and cost, coverage gaps, flake register, and checksum manifest.
Manual and nightly runs use the same Cloud Run Job. Cloud Scheduler adds attribution and a best-effort Slack summary, but it does not change the evaluation logic.
GitHub Actions has a deliberately narrower job: build the image and run fast contract tests. It does not run the full model-backed regression.
That is a change from the shorthand in our first architecture post, where we described the eval suite as running in CI and blocking merges. The implementation we proved is more conservative: model-backed regression runs manually or nightly in Cloud Run, and the decision is advisory. Running the same suite in GitHub and Cloud Run would duplicate model spend and create two control planes for one quality signal.
One model-backed control plane is easier to explain. The portal and Scheduler are two triggers for the same run, while CI protects the container and contract around it.
We also set Cloud Run task retries to zero. That is unusual enough to explain.
Infrastructure retries are useful when work is idempotent and a transient failure should simply be tried again. A model-backed evaluation is also evidence. If a provider or contract fails on the first attempt, silently spending the tokens again can hide the failure we were trying to observe. Model-level retry behavior is explicit and bounded; the Cloud Run task itself gets one attempt.
This creates an important semantic split: Cloud Run SUCCESS means the runner completed and published a valid report. It does not mean every case passed. A healthy run can publish an advisory warning. Runtime status and quality decision are two different green lights.
Scoring without letting good prose hide bad facts
Each case is graded across five dimensions:
- Factual accuracy: 35% weight, 90% minimum floor
- Classification: 25% weight, 90% floor
- Narrative quality: 20% weight, 80% floor
- Actionability: 15% weight, 80% floor
- Completeness: 5% weight, 100% floor
The overall threshold is 85%, but the floors matter more than the weighted average. A fluent answer cannot compensate for a wrong number. A useful recommendation cannot compensate for the wrong alert type. A complete schema cannot compensate for an unsupported causal claim.
The weights tell us how to summarize quality. The floors decide what cannot be traded away.
We use deterministic checks wherever the evidence permits them: exact values within tolerances, date windows, percentage direction, allowed classifications, required fields, forbidden claims, scope, and hard invariants. The JudgeAgent handles the dimensions that genuinely require judgment, such as clarity, appropriate qualification, and usefulness.
That mix is deliberate. An LLM is good at deciding whether a recommendation is clear and appropriately hedged. It should not be asked to improvise whether 4,210 is actually 22% below 5,400 when the suite can calculate it directly.
The failures that made the suite trustworthy
Most of the hard work was not writing the runner. It was deciding what each failure meant.
Anomaly baselines and labels disagreed. Older cases crossed schema versions, period labels were inconsistent, and some alert flags were missing. We stopped trusting the historical narrative, reconstructed expected values from unique frozen performance rows, bound only evaluation-scoped unknown labels, and rejected known conflicts.
Critical Insights used the wrong date grain. The case builder selected insight_date, while the product content represented an end_date. The output looked plausible but referred to the wrong business window. We changed selection to the true business period and made percentage sign and magnitude semantics explicit.
Smart Actions had “facts” that were really judgments. Subjective historical labels had been presented as factual truth, while action dates and execution windows were conflated. We kept only independently supported store and time evidence as factual checks, then treated subjective labels as behavioral regression.
An optional holiday table returned 404. The first candidate build aborted even though holiday context was not required for those cases. We now freeze optional holiday evidence as empty rather than failing unrelated capture.
Cost displayed as zero. The estimator did not recognize the deployed model name. The evaluation itself was fine; the artifact contract was not. We added the model’s official input and output rates and tests for the mapping.
A green runtime looked like a green evaluation. The portal followed the Cloud Run execution state and made it too easy to conflate completion with quality. We now publish and display both: infrastructure status and the decision inside the report.
These were not side quests. They were the work. The suite became trustworthy by learning to distinguish at least three things that initially looked like the same red cell: an agent regression, an invalid case, and an infrastructure failure.
The improvement path was not monotonic
The sequence tells the story better than the final score.
A lower score can be progress when the evaluator has stopped rewarding claims it cannot prove.
The early full replay passed 37 of 50. The rebuilt catalog passed 36. Evidence and schema fixes moved it to 44. A qualified proof replay reached 48. The final established baseline passed 50 of 50. Then an independently triggered Scheduler execution passed the same 50 and logged real Slack delivery.
The verified full run completed in 296.2 seconds, used 1,521,130 tokens, made zero retries, and produced an estimated model cost of about $0.48. Half of measured case calls completed in roughly 6.1 seconds or less; the p95 was about 7.4 seconds, while the slowest one percent reached roughly 15.4 seconds.
Those numbers are useful because they sit beside provenance. The report records the exact agent/workflow commit, immutable container image digest, case checksums, manifest checksums, trigger source, and execution identity. “50/50” is not a free-floating screenshot. It is a result we can reproduce and interrogate.
What 50/50 does—and does not—mean
The established baseline covers the three output agents: Anomaly Detection, Critical Insights, and Smart Actions. Co-Pilot has a separate provisional 10-case qualification slice because conversational orchestration, tool routing, write-isolated memory, scope, redaction, grounding, and usefulness are a different contract. We did not quietly add those cases to the 50 and move the goalposts.
The suite is also DEV-only and advisory.
It is not yet a protected production release gate. There is no sealed holdout cohort, signed approval manifest, role-separated Product and Data Science review, or calibrated blocking threshold backed by a long history of nightly runs. Branch protection does not yet depend on it.
That boundary is intentional. A weak gate creates ceremony. A premature gate creates false confidence. We want repeated evidence on flakiness, dimension drift, cost, and tail latency before this system is allowed to control a release.
The next phase is governance: a stable Co-Pilot catalog, validated customer incidents as permanent cases, a sealed challenge cohort, role-aware approval, protected signing, and advisory results in release dashboards before anything becomes blocking.
The principle is the same one the suite taught us at the case level: a status should never claim more trust than the evidence underneath it can support.
Takeaways
A few things we would carry into the next agent evaluation system.
- A golden case is a governed contract, not a saved answer. The prompt, evidence, checks, reference, applicability, and content identity all have to agree.
- Evaluate the cases as hard as the agents. A failed assertion can reveal a stale schema or invalid oracle instead of a model regression.
- Reference output cannot prove itself. Reconstruct factual claims from independent frozen evidence and use the reference for behavior, not truth.
- Run the real agent against a controlled world. Isolate writes and memory, replay MCP from frozen snapshots, and keep the product code path intact.
- Use deterministic graders for deterministic claims. Let an LLM judge clarity and usefulness; calculate values, dates, classifications, and hard invariants directly.
- Separate runtime success from quality. A completed job can publish an advisory warning. Infrastructure health and output quality are different signals.
- Bind every result to provenance. Commit, image digest, case checksums, artifacts, trigger source, and execution identity turn a score into evidence.
- Lower scores can be progress. If a stronger contract exposes weak cases, believe the diagnostic before changing the expectations.
- Stay advisory until the governance is real. Calibrate on repeated runs, add a sealed cohort and independent approval, then decide whether the gate deserves to block.
The headline result is 50 out of 50.
The more important result is that we can explain what those fifty passes mean, what they do not mean, and exactly which evidence made each one pass.
If you are building evals for agents that touch operational data, we'd love to compare notes.
Further reading
- Anthropic, Demystifying evals for AI agents. A useful vocabulary for tasks, trials, graders, transcripts, outcomes, and the evaluation harness around an agent.
- Google, Agent evaluation guide. The evaluation-and-fix loop, trace grading, comparison, and failure analysis in Google’s current agent tooling.
- Google Cloud, Set maximum retries for Cloud Run jobs. The task-level retry behavior behind our deliberate zero-retry runner configuration.
- Model Context Protocol, Specification. The tool protocol whose request and response evidence we freeze and replay per case.
Until next time,
Amit & Sahej