Evaluation Operator Topology and Verbatim Evidence

Overview

v1.1.1 is a patch release on top of v1.1.0. Evaluation campaigns now resolve their Inference and Data Operators as one typed topology, and the Gateway's embedded Operator, which always carries Data and Inference and now works in its own .ztd/operators/embedded-operator directory, can serve model-driven runs. Campaign traces carry the signed receipt of every inference call, including failures, and the exact model input. Public evaluation evidence is published verbatim: the restricted-text screen is removed, and the complete agent trace and frozen scenario fixture become proof artifacts the Evaluation Explorer renders. Eval backups now carry their vault key chain, so a snapshot restores after the runtime tree is wiped. The website adds Stripe purchase links and OEM terms.

The wire changes are additive protobuf fields. Evaluation evidence written by v1.1.0 is not carried forward: agent-trace.v2 traces and eval backup manifests before 3.0.0 are rejected (see Changed).

Highlights

  • Typed campaign Operator topology — CampaignOperatorTopology resolves the Inference and Data runtimes with enrollment-bound actuator anchors. A shared runtime pins exactly one actuator key (INV-EVAL-CAMP-35).
  • Embedded Operator for evals — The embedded Operator always carries data and inference, runs from .ztd/operators/embedded-operator, and is the default Data Operator, so make gw alone can run a model campaign (INV-EVAL-CAMP-32).
  • Receipted inference evidence — agent-trace.v4 records a signed ActionReceipt for every model call, including FAILED ones, plus the exact first-turn model input. On a shared runtime, the model-call commitments are claimed in strict ledger order (INV-EVAL-CAMP-36).
  • Verbatim public evidence — Model responses, transcripts, grader detail, the full agent trace and the scenario fixture are published unscreened, bounded only by size (INV-SPEC-DATA-02/03). ztd public verify-agent-trace checks a downloaded trace offline.
  • Restorable eval backups — Snapshots record the vault key chain, and a restore reseals the evidence under the current vault (INV-EVAL-EVID-05).

Added

  • Protocol — PublicSemanticGradeSummary.detail (eval.proto) carries the grader output verbatim. InferenceDispatchFailure.receipt (operator.proto) carries the Operator's signed FAILED receipt when dispatch reached the Operator. Go and TypeScript bindings and the protocol reference are regenerated.
  • CampaignOperatorTopology (internal/services/evaluation/campaign_operator_topology.go) — ResolveCampaignOperatorTopology, WithAnchors and SharedRuntime replace the CLI's per-role operator session resolution.
  • Public proof artifacts — assignment-<id>.agent-trace.json (binding assignment_agent_trace), and the campaign-frozen scenario fixture scenario-<id>.input.json and scenario-<id>.gold.json (bindings scenario_input_fixture and scenario_gold_criteria). Public record schema versions 1.7.0 and 1.8.0 are accepted.
  • ztd public verify-agent-trace --trace <file> — Verifies evidence_digest, each model and tool receipt signature against the trace's actuator anchors, and the sentinel bracket order.
  • Eval app enrollment approval — The ztd-eval app identity auto-enrolls with owner approval (AppEnrollmentOptions.Approver, authcmd.OwnerAppEnrollmentApprover), matched by request ID and CSR fingerprint.
  • Evaluation Explorer — Assignment Details shows the full agent trace (model input, every turn's thinking and text, tool arguments and results, signed receipts, anchors and the sentinel bracket), the scenario fixture and the evidence data. Evaluations has new cards and a methodology view. Wire enums are generated from the protocol (proto-enums.generated.ts, make explorer-catalog-*), and make explorer-embed-check fails on a stale embed.
  • Operator capabilities in listings — ztd operator list and ztd operator bind list share one table with a Capabilities column. ztd gw status shows the embedded Operator's directory.
  • Website — Stripe purchase links (/buy/<sku> 303 redirects from src/payment-links.json, maintained by scripts/stripe-payment-links.sh), /terms/ and /terms/oem/ (legal/oem-license-agreement.md), a new logo and favicon, and the post "The day my evals deleted the repo".
  • Constants — ErrEvaluationBackupKeyChainMissing, ErrEvaluationBackupKeyChainInvalid, EmbeddedOperatorDirname, and the vault key chain paths (VaultHeaderRelPath, VaultKeyRelPath, MasterKeyDPAPIRelPath, MasterKeySSHAgentRelPath).

Changed

  • Embedded Operator — It always carries data and inference (WithEmbeddedBaseline, replacing WithData); the gw start --capabilities default is now data,inference. Its working directory is .ztd/operators/embedded-operator, not the project root. RegistrationService.RecordEmbeddedActuatorKey binds the Gateway actuator key to the embedded Operator's document.
  • Data Operator binding — CLI session refresh prefers the embedded Operator, then a dedicated Data Operator (PreferEmbeddedDataOperator, INV-AUTH-OP-BIND-04). ValidateAgentDataOperator admits the embedded Operator for model-driven runs (INV-EVAL-CAMP-32).
  • Heartbeat pwd — Every Operator heartbeat reports its configured WorkDir, not the host process cwd.
  • Campaign traces (breaking for stored evidence) — Traces are sealed as agent-trace.v4. agent-trace.v3 traces still verify. agent-trace.v2 traces are malformed evidence and import as INVALID_EVIDENCE.
  • Failed receipts reach callers — governance.RejectionError is renamed ReceiptError, and WithFailedReceipt/FailedReceipt carry any signed FAILED receipt, so a failed inference dispatch returns its receipt and the trace can claim its commitment.
  • Eval backups (breaking for stored snapshots) — The manifest schema is 3.0.0 and records the vault key chain. A restore opens each snapshot's vault through the host key store and reseals the evidence under the current vault. Older manifests are not restored.
  • Public feed — Evidence is bounded only by size (structural strings ≤512 B, records ≤256 KiB). RoleTranscript is exported, grade detail is allowed, and PublicFeedProofMaxArtifacts rises from 8192 to 12288. The ztd-adapter contract pack is regenerated without the disclosure matrix and the restricted-text event schema entries.
  • Explorer release filter — Defaults to the newest attributed release in the public feed (INV-REL-VER-06). ci-explorer runs explorer-embed-check instead of refreshing the embed.
  • Website deploy — The deploy workflow passes ZTD_SMOKE_WORKER_VERSION, and the live smoke checks the x-ztd-worker-version header (CF_VERSION_METADATA). Reduced motion slows the hero grid instead of freezing it, Windows visitors land on the Windows install tab, and the page script moved to src/site.js.

Removed

  • Public restricted-text screening — ErrPublicFeedProofRestricted, ErrPublicFeedRestrictedField and result_redaction.
  • Actuator_pub.pem export — The actuator public key is exported only as JSON (ActuatorPubPEMFilename removed).
  • File-edit runtime-directory block — Removed from the file edit target. Only the protected runtime path check and the controlled evaluation target check remain.
  • CLI eval session plumbing — operatorSessions and resolveOperatorSessions, replaced by resolveOperatorTopology. PreferDedicatedDataOperators, replaced by PreferEmbeddedDataOperator.
  • .cloudflare/cache/cloudflare-account.json — Deleted from the tree; .cloudflare/ is ignored.

Fixed

  • INVALID_EVIDENCE in the live view — The Explorer did not recognize the INVALID_EVIDENCE lifecycle status or the CONTAINMENT scenario category. Wire enums are now generated from the protocol, and containment has public criteria. The explorer view schema is now 1.8.0.
  • Eval backup restore — A snapshot was unreadable once the runtime tree, and the vault in it, was wiped and recreated. Snapshots now carry the sealed key chain, and a clear ExportRun bundle under a sealed root is restored as stored after payload digest verification.
  • Heartbeat working directory — The heartbeat reported the Gateway process cwd, so Data Operator binding saw the project root instead of the directory where commands run.
  • Explorer version display — The bundle labelled the wrong release as current.
  • Compliance audit-chain export — ztd compliance release-prepare failed with audit chain segment: sequence gap when other chained events, such as command audit records, were interleaved with receipt facts in the window. The operational export kept only receipt facts. It now exports every chained event from the first to the last receipt fact in the window, and only receipt facts carry content_sha256. The window bounds now use the stored fixed-microsecond timestamp format; a variable-width bound admitted a fact up to a millisecond past the window end, which then failed the graph freshness check. An invalid analysis graph now reports its failures in the error.
  • Windows release smoke — Log handles stay open until the process group has exited and the child is reaped (closeLogs, stopGroup).
  • Website — Windows reduced-motion handling and CI timing.

Tests

  • New coverage for the operator topology (shared and split runtimes, anchor mismatch), agent-trace.v3/v4 validation and ledger-order claims, eval backup key chains and restore after a wipe, operational snapshot export across interleaved audit events, owner-approved app enrollment, the explorer campaign adapter, and agent trace and scenario fixture rendering.
  • Aligned existing unit and integration fixtures with v1.1.1 contracts: sealed backup file sizes (destructive_evidence_backup_test.go), verbatim grade detail publication (campaign_assignment_projection_test.go, evidence_files_integration_test.go), inference Operator actuator key anchors (test_support_test.go, runs_env_test.go), and unscreened public mirror ingest (public_mirror_test.go).
  • Removed the obsolete internal/cli/serve/sessions_test.go and dead certificate tests in cert_test.go.

Candidate verification (Linux, 2026-10-15, binary SHA-256 25267ab2c75f39c88b786bb63b82681d4b822bb6403e01127be811f94f9bd76a):

Command Result
./ztd test unit Pass
./ztd test integration --pkg <package> (each of ./internal/services/storage, ./internal/services/compliance/evidence, ./internal/services/compliance/report, ./internal/cli/cmd/compliance, ./internal/services/evaluation, ./internal/cli/cmd/eval, ./internal/cli/auth, ./internal/services/publicdisclosure, ./internal/cli/cmd/shared, ./internal/services/gateway) Pass, race detector enabled
./ztd test lint 0 issues
make console-lint, make console-test Pass (44/44)
make explorer-lint, make explorer-test Pass (680/680)
make website-test Pass (154/154)
make website-build Pass
make version-sync-check Pass (1.1.1, 16 declarations)
make proto-generate No diff
make console-embed-check, make explorer-embed-check, make explorer-catalog-check, make constants-check, make doctrines-validate Pass
  • Release smoke (./ztd test release-smoke --root .ztd/operators/release-smoke-v111-20261015) — Pass (exit 0, result: PASS in .ztd/operators/release-smoke-v111-20261015/evidence/smoke-summary.json, linux/amd64, posture doctrine, 2026-10-15T23:56:11.139434112Z to 2026-10-15T23:56:12.632855479Z): 3 scenarios (file-create, document-merge, embedded-file-create), 6 local receipts (2 worker, 4 gateway, all EXECUTION_STATUS_COMPLETED), 0 failed rows, 0 skipped rows across both Gateway and worker reports.

Documentation

  • CHANGELOG.md — Added the 1.1.1 row.
  • VERSION — Bumped to v1.1.1. make version-sync updated the console, evaluation explorer, adapter and protocol/node packages and the protocol/docs/ headers.
  • docs/architecture/evals.md (INV-EVAL-CAMP-32/35/36, agent trace v4, backups), public_spectator.md (INV-SPEC-DATA-02/03, proof artifacts, verify-agent-trace), operator.md and overview.md (embedded Operator capabilities and working directory), auth.md (INV-AUTH-OP-BIND-04), governance.md (ReceiptError), gateway.md, protocol.md, storage.md, encryption.md, docs/devs/codemap.md, docs.md and release_process.md.
  • Guides: unified_stack.md (eval app enrollment, embedded Data and Inference), getting_started.md and build_operator.md (install examples and the cosign identity at v1.1.1), README.md, the Explorer runbook and the website README.

Compliance Evidence

Operational evidence bundle generated by ./ztd compliance release-prepare:

  • Scope: gateway-operational-20261015 (window 2026-10-15T23:55:11.871Z to 2026-10-15T23:55:41.871Z, 2 receipts, 3 commitments, 2 source signer keys).
  • Public bundle: release-evidence/v1.1.1/report-bundle/manifest.json.
  • Offline verification: compliance_bundle@2.0.0 valid=true, failures=0, checksum root 187a28f510a50ccfe26a15e233e29605fdbb9780a2669f5e4155a8ad9d91d11f.
  • Source revision: 884678f3cc2673feb2688ef892b9e554af0e2e9d.
  • Canonical projections: docs/release_notes/v1.1.x/v1.1.1-compliance-evidence.md and docs/release_notes/v1.1.x/v1.1.1-compliance-evidence.csv.

Deferred