RFC: Procedure Manifests - Mechanism for AI Agents to Resolve Contractual Disputes

[adpoe]

Overview

This post seeks design & feasibility feedback on a proposed mechanism for ai agents to resolve subjective contractual disputes without a human-in-the-loop at arbitration time.

Proposal / Idea

Create a repeatable dispute-resolution procedure that is directly embedded in the smart contract.

At contract formation time, both parties agree to what we will call a procedure manifest. This manifest will define & pin the complete adjudication procedure for an LLM Judge to perform.

The manifest must contain:

• the requirements, decomposed into a machine-evaluable rubric

• the judge: specific model(s) pinned by version/hash, the LLM-judge’s complete evaluation prompts agreed verbatim, run counts, aggregation rules

• the admissible evidence: schema, submitters, data transformations, and named measurement sources - all defined inside the contract

• the default rule and burden of proof - who wins when the answer is genuinely ambiguous (i.e. - The LLM-judge has low confidence re: the correct answer)

• the remedy structure: how will funds be (re)distributed depending on arbitration result

Under this system, a contract would be Arbitrable if and only if its manifest conforms to spec. And at contract formation time there would be tooling to help users identify if the contract meets all requirements. Nonconforming contracts will not be arbitrable under these mechanisms - this creates a clear refusal boundary, knowable at contract creation time.

Because the procedure is fully specified and cheap to run (it’s an LLM-judge eval), both parties could independently execute the judge before disputing. The manifest therefore functions as a mutually-agreed acceptance test, and arbitration becomes the attested backstop that makes the acceptance test binding. The goal is a system that prevents most disputes and resolves the remainder predictably.

The intent is to compose with existing standards to the highest possible degree. Ideally, Procedure Manifests would ride in ERC-1497 MetaEvidence, resolve through an ERC-792-compatible Arbitrator, and anticipate ERC-8004 identities for parties and validators.

Background & Motivation

Agent-to-agent commerce has payment rails (x402, agent wallets, escrow), and with ERC-8004 it is getting identity, reputation, and validation mechanisms. I believe there is still a gap that sits just above the Validation Registry: subjective dispute resolution between ai agents.

My understanding of ecosystem is that current mechanisms resolve three contractual & trust problems well:

1. Was anything delivered at all? (escrow with timeout refunds)

2. Did the agent run the process it claims? (stake-secured re-execution, TEE attestation, zkML)

3. What have others experienced? (reputation registries)

However, it seems the trickiest problem with the most area for disagreement is still open. How do ai agents determine if the work delivered actually satisfies the ‘spirit’ of the contract - and what recourse do they have, when reasonable parties disagree? I believe inability to resolve such disputes cleanly & automatically will hold back widespread adoption of agentic commerce.

Examples:

• A website can be delivered on time, and still fail the spec in non-obvious ways

• An SLA can degrade in ways a refund timeout can’t see

• A funding milestone can be “met” only under one party’s reading

Data-feed oracles cannot answer these questions (no authoritative source exists for "was this good enough”).

One design choice is to only transact via micropayments. But I suggest this will become a burdensome limitation over time, and possibly hold back growth of the ecosystem.

Gap Analysis

Here is where I could use some help refining my own understanding of the proposed ecosystem, to ensure these ideas stand up to scrutiny. My understanding is that ERC-8183 (Agentic Commerce) standardizes the escrow lifecycle and creates an evaluator slot - a single address that alone may complete or reject - while explicitly scoping out dispute resolution (“no dispute resolution or arbitration; reject/expire is final”). This leaves the evaluator’s decision procedure entirely to future implementors.

My reading of Alkahest is that it similarly abstracts release conditions behind a composable boolean IArbiter, which is the interface for a condition, not the procedure for judging one.

Kleros solves the general problem for human-speed disputes with drawn juries, and ERC-792/ERC-1497 gave it a clean interface: any Arbitrable can resolve through any Arbitrator.

This proposal fills the layer that I believe those designs deliberately leave open: not who may release funds, but under what signed procedure they decide. The intent is to create a new Arbitrator type under ERC-792, and equally as an evaluator implementation for ERC-8183. Plus a contract-formation format that makes subjective disputes tractable in the first place.

Minimal Viability Validation
To validate baseline feasibility, I’ve created a draft spec, examples, and a working conformance validator: GitHub - formulary-systems/spec: Agentic arbitration procedure manifest spec · GitHub

• JSON Schema for the manifest, and 12 conformance rules constituting the executable refusal boundary

• Three example manifests: 2 conforming and 1 non-conforming, with explanations. These can be executed against the conformance validator already.

• An ERC-792/1497 compatibility sketch: the manifest rides as MetaEvidence; rulings map to the standard enum

Design intent is that anyone can implement a conforming Arbitrator.

Footnotes

An historical parallel shaped this design, and gave me a name for it. While researching this problem I saw some superficial similarities with the Roman law system’s Formulary procedure. I’m not a scholar in this area, so the analogy is surely inexact. But in principle, a magistrate published a formula stating: the exact question to be decided, the name of a private judge that both parties accepted, a damages cap, and a default rule. The private judge then rendered an arbitration following that exact formula. That’s, in essence, what I am proposing here.

My reading of this historical legal mechanism is that was intended to solve problems of scale. Namely, it allowed Rome to scale their legal system for use by non-citizens (peregrini), mostly for commercial law cases, as I understand it. (My knowledge here is wikipedia level, not research level - I’m happy to be schooled by someone who actually knows this in detail, but it’s not the main point.)

To finalize this analogy and why I see it as relevant - AI agents are similar to the peregrini: no legal citizenship, and no standing in existing courts. The analogy is directional, but the architecture is similar. Citizens publish the procedures; non-citizens transact safely under them; judgment delegates to agreed-upon judges. This is a guardrails system.

Questions for this forum

1. Prior art. If something here duplicates existing work - deployed or published - links are the most valuable reply this thread can get.

2. Feasibility & Safety, Security. This currently exists only at the level of a sketch. It is intending to solve what I see as a very hard problem - placing reliable guardrails on agentic ai coordinating serious commercial work together, with mechanisms for resolving disputes in a ‘fair’ way. At this point, I’m exploring the problem space. I welcome any feedback about why this is a potentially naive or impossible solution before anyone spends significant time on it. (Including me.)

3. For anyone who has operated or attacked deployed arbitration/oracle systems (Kleros, UMA, reality.eth, Aragon Court): which failure modes am I missing? Bribery and judge-shopping shaped several rules in my sketch. But I’m sure there is a lot more complexity here.

4. Ecosystem Integration. Am I misreading how this idea could be integrated into the wider agentic economy ecosystem that Ethereum is building? Is this actually a problem that people are eager to solve?

Again, I’d like to battle-test and sanity check this idea - as well as determine if other folks are interested in these topics. Please feel free to point me in a different direction if that is the appropriate response.

---

Changelog v0.0.2 (Sept/14 2026) — PR #2: closure counterexample by @chugarchugarr accepted and adopted as regression fixture #1. Confidence removed from the binding projection; ambiguity is structural (declared dissent caps; any non-unanimous panel is UNRESOLVED, per @cedricbrown taken to the strict limit); policy_on_unresolved in the literal policy(UNRESOLVED) shape. resolve_against_burden implemented, three values reserved as open contribution surface; normative state machine + Judge Result Contract; implementation-equivalence conformance with replay harness; evidence provenance pins; universal judge liveness; REPRODUCED/VERIFIED trust vocabulary per @babyblueviper1; ERC-8210 reconciled as composable remedy rail.

Proposed for v0.0.3 (under review in-thread): the acquisition boundary, authorized-run slot schedule + declared provenance binding - answering @SergeevDmitry’s selective-rerun question via @chugarchugarr’s strengthened invariant.

[chugarchugarr]

I have some adjacent implementation evidence that may be useful here. I don’t think it duplicates Procedure Manifests, but it exercises a few of the state boundaries that seem relevant to what you’re proposing.

Executable x402 Resolution Receipt reference implementation:
https://github.com/chugarchugarr/-x402-resolution-receipt

x402 Foundation discussion:
https://github.com/x402-foundation/x402/issues/2833

Surviving upstream proposal:
https://github.com/x402-foundation/x402/pull/3291

The implementation preserves independently signed verifier findings, conflicting findings on the same subject, explicit resolution states including UNRESOLVED, and hash-linked corrections rather than overwriting earlier conclusions.

That work came from a somewhat different problem, so I would treat it as adjacent implementation evidence rather than prior art for the complete manifest idea.

Reading your proposal, I think one thing you are already separating conceptually is especially important to preserve mechanically:

what the judge established != what the agreed procedure says to do about it.

You already have the ingredients for this in the manifest: evaluation procedure, default/burden rule, and remedy are separate fields. I would make that separation explicit in the resulting state machine.

For example, if the evidence genuinely does not establish whether the work satisfied the contractual requirement:

resolution = UNRESOLVED

while the pre-agreed manifest can still deterministically specify:

policy(UNRESOLVED) = REFUND

There is no human required at arbitration time, and there is no loss of determinism. But the record also does not convert “we could not establish it” into the stronger factual claim “the payer was right.”

I would probably preserve one additional boundary after that:

resolution
→ authorized remedy
→ execution
→ evidence that execution actually manifested

In other words:

judge returned X != contract was authorized to do X != X actually happened

That could matter once another ERC-8004 agent or another protocol consumes the result. A downstream system should be able to tell whether it is inheriting a judgment, an authorization, or an observed state change.

Your refusal boundary also looks important to me. “Nonconforming = not arbitrable under this mechanism” is preferable to forcing a result from a procedure that cannot support one.

A few failure modes I would specifically try to falsify:

1. Evaluator optimization / Goodharting

Because the exact rubric, prompts, model, transformations, aggregation rules and thresholds are intentionally known in advance, they become an optimization surface.

That is not automatically a vulnerability — an acceptance test is supposed to be known. But for subjective requirements there is a dangerous case:

optimize artifact to satisfy evaluator

without necessarily:

satisfy underlying contractual intent

If those diverge, the problem is no longer judge nondeterminism; it is that the machine-evaluable proxy failed to capture the contract.

I would test manifests adversarially for that divergence.

2. Evidence transformation becoming part of the attack surface

You include data transformations in the manifest, which I think is necessary. I would preserve both the original submitted evidence and the transformed representation, plus the exact transformation/version used.

Otherwise a deterministic judge can still produce a perfectly reproducible answer over a lossy or adversarial transformation.

3. “Pinned model” reproducibility

For locally reproducible/open models, model hashes are straightforward conceptually. Hosted models seem harder: identical model names/prompts do not necessarily establish identical weights, serving stack, system wrappers, decoding behavior or inference environment.

So I think the specification may eventually need to distinguish:

model identity

from

reproducible execution environment / attested execution.

Otherwise two parties can conform to the same manifest and still receive different evaluations for reasons outside the manifest.

4. Aggregate certainty hiding disagreement

If several judge runs or models disagree, I would preserve the individual findings rather than only the aggregate ruling. The aggregate can determine the contractual result, but the disagreement remains useful evidence.

That also gives you a cleaner path for later corrections without rewriting history.

5. Procedural determinacy vs fairness

I think Procedure Manifest conformance can establish something very strong: the adjudication procedure was completely specified and executable before the dispute.

I’m less sure conformance alone can establish that the procedure is substantively “fair.” Two parties can deterministically agree to a badly specified rubric, biased evidence transformation or pathological default rule.

It may therefore be worth keeping the spec’s claim narrow:

conforming → procedurally closed/executable

rather than:

conforming → fair/correct.

On ecosystem placement, I don’t think I see an obvious contradiction in the layer you’re describing. ERC-792 can remain the arbitration interface, ERC-1497 can carry dispute metadata, ERC-8004 can provide identities/validation context, and ERC-8183 can consume the resulting evaluator/arbitration decision.

What Procedure Manifests seem to add is the signed, precommitted procedure governing how the decision is produced.

The piece I would preserve very carefully is the boundary between:

evidence → judgment → authorized consequence → manifested consequence

because those become different claims as soon as the result is consumed by another agent or protocol.

[cedricbrown]

ran an llm judge as an acceptance gate on a delivery contract last year. pinned snapshot, temperature 0, the whole setup. what broke me was the reruns. same rubric, same evidence, different verdicts. only about 4 percent of cases, but they weren’t scattered. they were the half-met borderline ones, which is the only population that ever reaches a dispute anyway. so both parties running the judge before disputing works fine on the cases nobody argues about, and falls apart where the money is.

that’s also why i wouldn’t hang the default rule off the judge’s self-reported confidence. tried that. got a cheerful 0.9 on items that flipped on the very next run. you’re already specifying run counts and aggregation, so the split rate across those runs is sitting right there for free, and a manifest that says a k-of-n split fires the default rule turns judge instability into a declared outcome instead of a coin flip.

on prior art, one you might not have. ERC-8183 post 51 (drmattbarney, march). he argues a multi-judge setup is basically Classical Test Theory and points at Rasch and item response models instead, which lands right on how you pin aggregation. ERC-8210 Agent Assurance also carries an arbitration role callable when a party disputes the Evaluator decision, worth checking whether that’s already sitting in your slot.

last thing, and it’s boring but it’ll get you. pinning a model by version or hash pins the name, not the behaviour, and snapshots get retired. contract formed today, model gone in nine months, and it’s either unarbitrable or quietly substituted. probably wants a liveness rule and a stated fallback, agreed at formation rather than discovered at dispute.

[babyblueviper1]

Real, checkable mechanism behind that 4%, not just anecdote: temp=0 doesn’t guarantee determinism on a hosted API because dynamic batch size at inference time changes the GPU reduction kernel’s numeric path (RMSNorm/matmul/attention reduction trees differ by batch composition) – same prompt, different server load, can diverge during greedy decoding even at temp=0. This is well-documented (Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference,” Sept 2025) and it’s an opaque, uncontrollable property from the caller’s side – you have no visibility into what else was batched with your request, and most hosted reasoning-tier models don’t expose a temperature override at all. The fix (batch-invariant kernels, forcing one reduction strategy regardless of batch size) exists and is demonstrated bit-identical over 1,000 runs, but costs 34-60%+ throughput – not something most API consumers can turn on themselves.

Concrete implication for a dispute mechanism specifically, from measuring the same failure mode on our own /review verdicts: reruns on borderline cases flip on confidence and issue-count, but the discrete verdict (approve/reject) itself is far more stable than the reasoning trace around it. If Procedure Manifests pins the LLM Judge’s full evidence+rubric+temperature, that pins the INPUT, not the reduction path – so the manifest’s committed output should probably be the narrow decision only (the binding both parties can dispute over), with the full trace kept as supporting evidence, not itself the thing re-run and diffed for equality. Rerunning the whole judge and expecting byte-identical output is asking for more determinism than temp=0 on a hosted model can actually deliver.

[chugarchugarr]

Following the latest exchange, I went back to the current Procedure Manifest repository and tried to resolve the question mechanically rather than add another interpretation to the thread. I think the result is now narrow enough to falsify against the draft itself.

Evidence and provenance

The current README says a procedure manifest pins the complete adjudication procedure and explicitly identifies examples/website-delivery.manifest.json as a conforming LLM-judged example. The conformance rules describe conformance as establishing a procedure that is complete, closed, and executable, with the outcome-relevant procedure defined in advance.

Current sources:

README

Conformance rules

Procedure Manifest schema

Conforming website-delivery example

The example commits three judges, five runs per judge, within_judge = majority, across_panel = majority, temperature = 0, a per-requirement confidence_threshold = 0.7, a payer default, requirement weights of 25/20/25/20/10, and an overall weighted passing threshold of 8000 basis points. Its prompt asks each LLM run to return PASS or FAIL, a confidence in [0,1], and a rationale.

The schema explicitly says that confidence below the threshold causes the requirement to resolve through default_rule, while the problem statement separately acknowledges under OQ4 that self-reported confidence semantics are unresolved. What I cannot find in the current schema, conformance rules, example, or reference conformance validator is the operation that converts multiple run-level confidence observations into the single per-requirement confidence consumed by that threshold, or a rule fixing where that confidence operation occurs relative to within-judge and across-panel aggregation. That is the boundary I tested.

PredictionLock

If the current procedure is closed in the strong sense intended by conformance, then once the manifest and the complete evaluation observations are fixed, there should be no remaining implementation choice capable of changing the contractual result.

Equivalently, two implementations following the published manifest should not be able to consume the same fixed observations and derive opposite remedies because an outcome-relevant transformation was left unspecified.

I therefore froze the manifest and constructed one fixed set of run observations for R1. R2 through R5 are held PASS in both executions.

J1

PASS 0.99

PASS 0.99

PASS 0.69

FAIL 0.69

FAIL 0.69

J2

PASS 0.99

PASS 0.99

PASS 0.99

PASS 0.99

PASS 0.99

J3

FAIL 0.99

FAIL 0.99

FAIL 0.99

FAIL 0.99

FAIL 0.99

The discrete majority results themselves are not ambiguous. J1 is PASS by 3–2, J2 is PASS unanimously, and J3 is FAIL unanimously. The only question is how the committed confidence threshold consumes the five confidence observations associated with a repeated judge execution.

Executable counterexample

One implementation can reduce J1’s five confidence values by arithmetic mean:

(0.99 + 0.99 + 0.69 + 0.69 + 0.69) / 5 = 0.81

That is above the manifest’s 0.7 threshold. J1 therefore retains its PASS majority. Across the panel the requirement becomes PASS by 2–1. Because R2 through R5 are also PASS, the weighted score is 100/100, which exceeds the committed 80% threshold. Under the example’s ruling map, the payee-side outcome releases the funds.

A second implementation can reduce the exact same five confidence values by median:

median(0.99, 0.99, 0.69, 0.69, 0.69) = 0.69

That is below 0.7, so the confidence rule enters the committed payer-default path. If that default is interpreted as resolving R1 against the payee, R1 contributes no passing weight and the remaining requirements total 75/100, below the required 80%, producing the payer-side outcome and refund. If the default is instead interpreted as immediately selecting the payer-side ruling, the final remedy is still refund.

Mean and median are not being proposed as competing normative choices. Neither is the proposed fix. They are witnesses. The relevant fact is that the current procedure does not select the confidence-reduction function or its location in the adjudication sequence, while that undeclared choice is capable of changing contractual state.

The manifest is unchanged. The evidence is unchanged. The models and prompts are unchanged. Every PASS/FAIL value is unchanged. Every confidence value is unchanged. The complete observations are unchanged. Only an unspecified transformation changes, and the resulting remedy changes from release to refund.

Observed result

This falsifies the stronger procedural-closure condition for the current v0.0.1 draft.

The result does not depend on LLM nondeterminism. The LLMs can finish first and every output can be frozen permanently. The divergence remains after observation because the deterministic procedure consuming those observations is not yet completely specified.

OQ4 already correctly identifies confidence semantics as open. The counterexample establishes why that open question is not merely about whether confidence is well calibrated. As long as confidence remains outcome-relevant, its reduction semantics are part of the state-transition procedure and therefore part of the closure boundary.

The same inspection exposes a neighboring issue. The example prompt asks for PASS/FAIL, confidence, and rationale, but the current manifest schema does not define a normative machine-level judge-result object, parser, authoritative projection, or canonical representation for those produced values. The prompt constrains what the model is asked to say; it does not yet define the exact result contract that independent implementations must consume.

Resolution

I think the minimum invariant is therefore:

Every evaluation-produced value capable of altering aggregation, resolution, remedy selection, or contractual state must enter through an explicitly defined authoritative result contract, and every transformation of those authoritative values capable of altering the outcome must itself be committed by the procedure.

That means the manifest does not need to require the entire model trace to be deterministic. It needs to specify exactly which projection of the observation is authoritative, how that projection is parsed and represented, and every reduction that can affect the next contractual state.

For v0, there are two coherent ways to resolve OQ4. If self-reported confidence remains authoritative, then the procedure must define its exact semantics: which confidence observations participate, the reduction function, the order in which reduction and majority aggregation occur, and what a below-threshold result means at the requirement and contract levels. If those semantics are not ready to be standardized, confidence can remain preserved in the transcript as evidence while being removed from the binding projection.

The same rule applies to rationale, issue lists, diagnostics, or any richer model output. They can be retained completely without acquiring undeclared authority over the contract.

The resulting boundary is:

evidence

↓

judge observation

↓

authoritative result

↓

within-judge aggregation

↓

across-panel aggregation

↓

requirement resolution

↓

contract outcome

↓

authorized remedy

↓

manifested consequence

Each arrow is a separate transformation. If an arrow can change the contractual state, its semantics belong inside the committed procedure.

The corresponding conformance test becomes:

Given the same committed manifest and the same complete evaluation observations, two independent conforming implementations must necessarily derive the same requirement resolutions, the same contractual outcome, and the same authorized remedy.

If that does not hold, some outcome-relevant procedure remains outside the manifest.

The current website-delivery example fails that test at the confidence-reduction edge. This is also consistent with the adjacent implementation evidence I linked earlier in this thread, where I was already separating verifier findings, resolution state, authorized consequence, and what actually manifested. I do not think that prior implementation needs to be imported into Procedure Manifests; the useful part here is the invariant, and this counterexample establishes it independently.

Current conclusion

The current Procedure Manifest draft already recognizes nondeterministic judges and already recognizes confidence semantics as unresolved. The remaining step is to make the state boundary exact.

The model does not have to be bit-deterministic for the contractual procedure to be closed. But once any part of a model observation is allowed to influence contractual state, every transformation from that observation to the authorized remedy has to be part of the procedure the parties committed to.

Preserve everything the judge produced. Give contractual authority only to the values whose state-transition semantics are completely specified. Under that condition, independent implementations can disagree in supporting traces without being allowed to disagree about what the contract does next.

[babyblueviper1]

@chugarchugarr that’s a genuinely clean falsification – freezing the observations and varying only the undeclared reduction function isolates it well, and the arrow-chain (evidence → judge observation → authoritative result → aggregation → resolution → outcome → remedy) is a good normal form for “what has to be inside the committed procedure.”

We hit an adjacent version of this boundary for a related reason and it might sharpen where the line falls. Our production verdict pipeline (/review) has to decide exactly this: which fields get contractual authority (bound into a commitment a third party checks) vs. which are preserved as evidence only. The real preimage, verbatim from the running code (services/proof_signing.py, DECISION_REF_PREIMAGE_FIELDS)

artifact_hash, artifact_type, policy_version, verdict, source_class, vantage_limitation, related_decision_ref, intended_audience, confidentiality_tier, disclosed_summary, intended_verifier, policy_commitment, verified_at, registry_as_of, registry_snapshot_sha256, action_binding_tool_hash, action_binding_args_hash, action_binding_agent_id, action_binding_nonce, freshness_beacon_hash, epistemic_basis

verdict IS in that list – it’s part of the committed hash, same as your “authoritative result” needing to be explicit. But we draw a second, narrower line inside that: every field in the preimage is REPRODUCED in the sense that a third party can recompute the exact hash bytes with zero trust in us (canonical JSON + sha256, a pure function). The verdict’s correctness, though, is only VERIFIED – authentic, attributable, tamper-evident via our signature – never claimed as independently re-derivable by rerunning the judgment. We disclose this per-verdict via a vantage_limitation field (itself in the preimage) naming what our own judgment’s vantage couldn’t establish.

Mapped onto your invariant: committing a value into the manifest’s binding hash (REPRODUCED-shaped) is a different trust primitive than claiming two independent judge runs will converge on it (a determinism claim). Your counterexample shows confidence can’t get contractual authority as an aggregation input without specifying the reduction – but our read is that even after fixing the reduction, the resulting number stays only VERIFIED-shaped (attributable to that specific committed run), not REPRODUCED-shaped (recomputable by a second, independent judge execution), because LLM judgment isn’t a pure function of its stated inputs – only the hash of the manifest+observations is. That argues for treating confidence the way we treat verdict+vantage_limitation: bind it for attribution, but never let a re-run’s disagreement with a frozen observation count as a spec violation – only disagreement with the signed record of what that specific run actually said should.

Real prior-art pointer directly on point: we found and helped close an almost identical gap yesterday in a TEE-attestation conformance spec (trustless-ai/recompute-kit) – a leaf could drop its “not judged correct” disclaimer and still pass conformance, because execution-integrity (ran on unmodified hardware/model) and judgment-correctness were being silently conflated in one attestation. Same shape of bug, different domain.

[SergeevDmitry]

I may have missed this in the original post. What prevents selective reruns by the arbitration executor?

Suppose the manifest requires five judge runs. The executor performs twenty runs and submits only the five that favour user 1 over user 2. Every submitted output could be authentic and the aggregation could follow the manifest exactly.

How would the contract distinguish those five outputs from the five authorized runs?

Would execution need to commit to a dispute-specific run schedule before any results are known? And how should failed attempts and retries be recorded so that a retry policy cannot become a way to discard an unfavourable verdict?

[chugarchugarr]

My earlier conformance test started with “the same manifest and the same complete evaluation observations” and then asked whether two implementations could derive different remedies. I froze the observation set to isolate the downstream procedure.

Your example asks the missing question one step earlier: what establishes that those observations are the complete authorized observation set in the first place?

I don’t think runs = 5 answers that. It tells aggregation how many observations it expects, but five authentic outputs selected from twenty can satisfy that cardinality just as easily as the five intended runs.

I also don’t think the Procedure Manifest itself should try to prove that the executor never queried a model sixteenth or twentieth time. The original proposal already separates procedural specification from execution-integrity mechanisms such as re-execution/attestation. The manifest’s job is narrower: define which run instances are allowed to acquire contractual authority, and require evidence sufficient to bind each admitted observation to one of those instances.

I would therefore put one more boundary before judge observation:

evidence

→ authorized run

→ attested observation

→ authoritative result

→ aggregation

→ resolution

For a dispute, each authorized run should be identifiable before its result exists, e.g.

run_id = H(manifest_hash, dispute_id, requirement_id, judge_id, run_index)

A result counts only if the chosen execution/provenance mechanism binds that result to that run_id. An arbitrary authentic result from the same model and inputs is not enough.

That also answers the retry case. A completed verdict consumes the authorized run slot. A retry is permitted only for failure states that the manifest defined before execution, and the failed attempt stays attributable in the transcript/lineage. An unfavorable completed verdict is not a failure condition and cannot open another attempt.

The important limitation is that this property depends on the execution mechanism. If a hosted model allows an executor to invoke the same authorized run repeatedly and later attach the run_id to whichever output it prefers, then the procedure cannot establish selective-rerun resistance for that judge. The correct result there is the same refusal boundary the proposal already uses elsewhere, not pretending aggregation can repair missing execution provenance.

So I would strengthen my previous closure invariant to:

Every outcome-relevant transformation must be committed by the procedure, and every observation admitted to those transformations must be proven to originate from an execution instance authorized by that procedure.

The falsification test is then exactly your twenty-run example: can an executor replace the observation occupying an authorized run slot with a different completed invocation after seeing both results, while all conformance checks still pass? If yes, the acquisition side of the procedure remains open. The procedure must constrain both which observations may enter and what transformations may act on them.

[adpoe]

Thank you everyone for the thoughtful reviews and comments.

In response, I’ve created a new PR (https://github.com/formulary-systems/spec/pull/2), adopting many of the suggestions in the thread so far.

High Level Points
the v0.0.1 closure claim was falsified by a counterexample from @chugarchugarr. An unspecified confidence-reduction step allowed two conforming implementations to derive opposite remedies from identical frozen observations (mean vs median on confidence). In response, v0.0.2 adopts the following invariant: every outcome-relevant value enters through an explicit result contract, and every outcome-relevant transformation is committed in the manifest.

Moreover, unanimity is now required across the panel of judges. Therefore any judge disagreement produces UNRESOLVED, and the burden allocation - not an implementation’s silent choice - decides the outcome. Critically, this makes `burden_of_proof` and `policy_on_unresolved` the most consequential fields in the manifest. Parties should negotiate them with the same care as price.

Clarification adopted: conformance establishes that the procedure is complete, closed, and executable - never that it is fair, wise, or correct. Two parties can deterministically agree to a bad rubric, a biased evidence transformation, or a burden allocation one of them will regret. Execution of the authorized remedy (escrow release, bond claim, stream halt) and its on-chain manifestation are downstream of this spec.

Changelog

• Confidence removed from the binding projection (confidence_threshold deleted). Self-reported confidence is evidence-only - preserved in transcripts with zero contractual authority - per convergent thread feedback (@chugarchugarr, @cedricbrown , @babyblueviper1 ).

• Ambiguity is now structural: aggregation is fully parameterized as within_judge: majority_with_dissent_cap (declared max_dissents) and across_panel: unanimous. Any non-unanimous panel yields UNRESOLVED (@cedricbrown‘s split-rate proposal, taken to its strict limit).

• UNRESOLVED is first-class at run, judge, and requirement level, with declared `policy_on_unresolved` in @chugarchugarr policy(UNRESOLVED) shape.

• Introduce a Judge Result Contract (spec/judge-result.schema.json): authoritative result object, strict final-line parse rule, no salvage parsing. Nonconforming runs are UNRESOLVED runs.

• Introduce a normative state machine (spec/adjudication.md) with the semantic conformance property: same manifest + same frozen observations ⇒ same authorized remedy across independent implementations.

• Added a Replay harness + regression fixtures (validator/src/replay.ts, validator/fixtures/). Fixture 1 is the falsifying observation set; tests assert outcome-invariance under adversarial permutation of (evidence-only) confidence values.

• Introduce evidence provenance: non-trivial transformations require a `transformation_pin`. originals, transformed representations, and pins are all preserved.

• Universal judge liveness: every manifest requires a fallback ladder - self-hosted judges included. A pinned hash pins a name, not continued availability (as Noted by @cedricbrown).

• REPRODUCED vs. VERIFIED trust vocabulary adopted (from @babyblueviper1); the spec explicitly disclaims byte-identical rerun expectations (per the batch-variance analysis of hosted inference).

• Add an ERC-8210 composition stub (arbitrator.erc8210_profile, draft, non-normative): a manifest ruling as EvaluatorDispute attestation; bonded_finality compiling to an AAP JobAssurance.

• Scope language narrowed throughout: conforming ⇒ procedurally closed and executable; never ⇒ fair or correct.

Remaining Open Questions & Proposed Path Forward

The current PR does not yet address @SergeevDmitry’s question: what stops doing twenty runs and picking the most favorable five?

@chugarchugarr‘s framing of it is the one I’d like to adopt. The earlier design specified using "the same complete evaluation observations”. But this example attacks one step earlier, asking what establishes that an observation set is the complete authorized set at all.

My first (incorrect) instinct was to use a result-linked chain. Each run’s identity hashed over the results of all prior runs, blockchain-style, so the transcript couldn’t be reordered or edited. But a result-chain doesn’t stop the attack. The executor holds the pen, and nothing prevents an executor from discarding the entire chain after an unfavorable run and regrinding from run one until the whole sequence pleases them. Hash-linking makes history tamper-evident to someone who already holds the true head; it doesn’t stop the author from writing a fresh history. It also serializes runs and makes legitimate retries cascade.

For that reason, I believe @chugarchugarr‘s slot-schedule construction is the right authorization primitive. And that’s precisely because run_id = H(manifest_hash, dispute_id, requirement_id, judge_id, run_index) contains no results. Every authorized slot is identifiable before anything executes.

Where I think the chain idea does belong is one layer down, in transcript lineage. Which can use append-only, hash-linked attempt records. This could also ensure retries stay attributable.

So here is the proposed v0.0.3 direction, posted here for design review, deliberately not pushed to a PR yet:

• Adopt the strengthened invariant from post 8 verbatim: every outcome-relevant transformation committed by the procedure, and every admitted observation proven to originate from an execution instance authorized by that procedure.

• Pre-computable slot schedule: run_ids as above; an observation acquires contractual authority only by binding to an unconsumed slot.

• Declared provenance_binding per judge (deterministic_reexecution | signed_receipt | tee_attestation), checked for coherence with hosting by a new conformance rule. The observation that I think ties this thread together: the grinding surface is exactly the judge’s nondeterminism. I believe a self-hosted judge on a pinned, batch-invariant serving stack with a committed seed has one valid output per authorized slot. Twenty runs would produce the same result twenty times, selective submission is pointless, and any admitted observation is checkable by re-execution (REPRODUCED-shaped binding, free with the open-weight tier this spec already defaults to). Hosted nondeterministic judges make every invocation a fresh lottery ticket, so binding must come from the execution side: provider-signed inference receipts tying a request nonce (= run_id) to an output hash, or TEE attestation. Where a provider offers neither, that judge class cannot establish selective-rerun resistance, and the answer is the same refusal boundary used everywhere else, per post 8’s limitation paragraph.

• Slot-consumption retry semantics: a completed verdict consumes its slot; v0.0.2’s parse rule already anticipated this. A parse failure is a completed run yielding UNRESOLVED, not a retryable event (if it were retryable, inducing parse failures becomes its own grinding lottery). Retryable states are enumerated ex-ante and limited to infrastructure failures where no result exists (timeout, transport), with every failed attempt kept attributable in the hash-linked lineage.

Is there anything else from the discussion that I am overlooking?

@SergeevDmitry, does this close the schedule-before-results gap as you intended it, or is there a selection channel this still leaves open?

@chugarchugarr, does the binding-tiered version survive your falsification test? And more directly: you’ve supplied the closure invariant, the acquisition boundary, and the lineage pattern this spec is converging on; if you’d want to land the acquisition boundary as a PR, or co-draft the state-machine section, I’d welcome those contributions.

Next Steps

I’m inviting reviews on PR #2. It holds everything described above. Not merging until it survives review/discussion. Review from anyone here is welcome: scope, rules, fixtures, or the judgment calls.

Try to falsify it. For anyone who wants to poke at the mechanism directly: the validator is deliberately boring TypeScript. ~500 lines, no network, no LLM calls. There are two parts.

1. Static conformance: check the manifest against a JSON Schema, then fifteen hand-written rules (C1–C15) enforce the cross-reference and closure properties a schema can’t express. Every LLM-judged requirement must name an agreed prompt, every measurement a declared source, dissent caps must leave a strict majority possible, only implemented unresolved-policies are arbitrable, non-trivial evidence transformations must be pinned. Failures come back as rule-tagged defects; that list is the refusal boundary.

2. Semantic replay: a pure function implementing the normative state machine. Parse each frozen observation per the Judge Result Contract, aggregate with the declared dissent cap, require panel unanimity, resolve UNRESOLVED against the burden, apply the outcome rule and ruling map, emit the authorized remedy. Fixtures are frozen observation bundles with expected outcomes, so `npm test` re-derives remedies deterministically.

–

The validation walkthrough describes how to write your own observation bundle against one of the example manifests, run it through the replay harness, and check whether any implementation choice of yours can produce a different authorized remedy from the same inputs. If you find one, an outcome-relevant transformation has escaped the manifest - open an issue with the bundle attached.

[babyblueviper1]

Real, shipped data point that seems relevant to the declared-provenance-binding work for v0.0.3, and to @chugarchugarr’s evidence → judgment → authorized consequence → manifested consequence boundary above.

In our own `/review` verdict pipeline (a live, signed pre-action verdict, not a paper design), a verdict can be fully VERIFIED – signed, attributable, tamper-evident – while still carrying an explicit `vantage_limitation` field for irreversible-class artifacts, disclosing that the verdict is occurrence evidence, not an absence/completeness claim. Separately, an `execution_binding` field can be set to `external` when the judgment’s own execution was delegated elsewhere rather than run by us directly.

Why this seems relevant to the provenance-binding work: the coherence gap isn’t only “which hosting model produced this judge run” (self-hosted deterministic vs. TEE vs. signed receipts) – it can also live inside the judgment artifact itself, as an explicit disclosed scope limit that a consuming Guard/manifest needs to check before treating a VERIFIED-shaped result as sufficient to authorize a given remedy. A judge run can be perfectly attributable and tamper-evident and still be scoped narrower than the remedy structure assumes.

Concrete question for the manifest schema: is there room in the Judge Result Contract for a verdict-level scope/limitation disclosure field (distinct from confidence, which v0.0.2 correctly removed from the binding projection), or would that need its own extension point?

[chugarchugarr]

@adpoe I ran the binding-tiered version through the same falsification test. As written, I don’t think it survives yet, but I think the remaining gap is exact. @babyblueviper1’s scope example exposes the corresponding boundary from the other direction.

The common rule is:

authenticity is necessary for authority, but it is not sufficient for authority.

On the acquisition side, run_id fixes which slot is authorized before results exist, but a signed receipt of (run_id, output_hash) only proves that the provider produced that output for that nonce. Unless signed_receipt also means provider-enforced non-equivocation, the provider can produce and sign both (r, hash(A)) and (r, hash(B)) for the same authorized slot. The executor sees both and submits the preferred one. Both receipts are authentic and the selective-rerun attack survives.

So I would move slot consumption one step earlier. The slot must be consumed when execution authority is acquired, not when the observation is later admitted.

I think the hosted form wants two linked records:

claim_receipt = Sig(run_id, attempt_id, request_hash, accepted_at)

followed by exactly one:

terminal_receipt = Sig(claim_receipt_hash, terminal_status, output_hash?)

where the execution mechanism MUST reject a second claim for the same authorized attempt. request_hash commits the exact evidence root, prompt/material, judge execution profile, sampling parameters and every other outcome-relevant execution input. Binding only the slot without binding the request still allows the right nonce to authenticate the wrong execution.

There is a second selection channel here too: withholding. Even if the provider produces only one result, if only the executor can see the terminal receipt, an unfavorable result can be suppressed and presented as timeout/transport failure. So selective-rerun resistance also requires terminal state to be independently verifiable from the provenance mechanism. A caller-local timeout cannot prove that no result was produced.

That makes the execution lifecycle:

AUTHORIZED → CLAIMED → TERMINAL_RESULT | TERMINAL_UNRESOLVED | ATTESTED_NO_RESULT

TERMINAL_RESULT and TERMINAL_UNRESOLVED consume the logical run. Only an execution-side ATTESTED_NO_RESULT – or an equivalent independently verifiable absence condition committed by that provenance mechanism – can open the next preauthorized attempt. A fresh dispute_id also cannot be executor-selectable after results exist, or the executor simply regenerates the entire slot namespace one level up.

@babyblueviper1 , I think vantage_limitation then answers a different question and should stay distinct from execution_binding.

execution_binding says where/how this observation came from. That belongs to provenance/acquisition.

vantage_limitation says what this observation actually establishes. That belongs to authority eligibility.

I would absolutely leave room for that in the Judge Result Contract, but I would not make an outcome-relevant limitation a free-text extension. If a limitation can determine whether a verdict is sufficient to authorize a remedy, then under the closure invariant it has acquired contractual authority and its semantics have to be committed too.

So I would split it:

• free-form limitations may remain evidence-only;
• a limitation that can affect contractual state becomes a structured scope/authority_scope field in the authoritative projection;
• the manifest commits the required_scope for the requirement and the exact predicate for satisfying it;
• in v0, the simplest deterministic rule is scope insufficient → UNRESOLVED.

That prevents a VERIFIED result from silently escalating beyond what it established. A verdict can be perfectly signed, attributable, uniquely acquired and still fail the eligibility check because its scope does not support the contractual consequence being requested.

So I would now write the full boundary as:

evidence

→ authorized execution

→ attested observation

→ authority eligibility

→ authoritative result

→ aggregation

→ resolution

→ contract outcome

→ authorized remedy

→ manifested consequence

The eligibility step is where the two questions meet:

eligible = authorized_execution ∧ exact_request_binding ∧ unique_terminal_execution ∧ sufficient_scope

Only an eligible observation is allowed into the authoritative result path.That gives me a stronger closure invariant:

Every outcome-relevant transformation must be committed by the procedure; every observation admitted to those transformations must originate from a uniquely authorized execution; and every condition deciding whether that observation is sufficient to carry contractual authority must itself be committed and machine-evaluable.

With single-use acquisition, independently verifiable terminality, exact request binding, and the scope-eligibility check above, I no longer have a selective-rerun or scope-escalation counterexample inside the stated model.

I’d co-draft the state-machine section around that lifecycle. I think that is the acquisition boundary to land once the wording converges.

[babyblueviper1]

@chugarchugarr’s AUTHORIZED → CLAIMED → TERMINAL_RESULT | TERMINAL_UNRESOLVED | ATTESTED_NO_RESULT lifecycle maps closely onto a real, live mechanism we ship for exactly the “authentic-but-swappable” problem, so sharing it as a data point.

Our /review endpoint durably records a “pending” admission row the moment a request is accepted – BEFORE verdict computation runs, not after. Each admission gets an index, a canonical request digest, and receipt_hash = sha256(admission_index, accepted_at, request_digest, prev_receipt_hash) – chaining every admission to the one before it. Real example, from a /review call I made minutes before writing this:

admission_index: 33
accepted_at: 1789490106
request_class: standard
response_deadline: 1789490166
receipt_hash: d709c99a...
prev_receipt_hash: 498d7e96...

Periodically (currently every ~15min, live right now) a batch of receipt hashes is Merkle-rooted and the root is OTS-stamped to Bitcoin, so a third party can check chain integrity + index continuity + checkpoint inclusion + the OTS proof without ever calling us.

Where this maps onto your lifecycle, and where the honest boundary sits: the chain proves an ADMITTED request’s record cannot be silently altered or deleted after acceptance – a real answer to “provider signs both (r, hash(A)) and (r, hash(B)), executor submits the preferred one,” since the admission (the claim) is committed and hash-chained before the verdict (the terminal result) exists. It does NOT prove every incoming request WAS admitted – a request that errors or is silently dropped before reaching the admission step leaves zero trace, and no external verifier can detect an event that was never recorded. That’s exactly the edge your ATTESTED_NO_RESULT state is reaching for, and it’s still open on our side too.

Closing it fully seems to need the CLIENT side to also commit something checkable before sending, not just the provider – an executor that made a request and got silence needs its OWN receipt to point to. Is your claim_receipt = Sig(run_id, attempt_id, request_hash, accepted_at) meant to be signed by the requester specifically for this reason? If so that’s the missing half our own mechanism doesn’t yet have.

Prior art worth citing if useful: this exact “OTS-stamp every receipt vs. checkpoint-and-stamp-the-root” tradeoff was raised against an earlier design of ours by @imran-siddique (agentbaseline/agentbaseline#26) – the counter-spec there (index-before-computation, hash-chain, periodic Merkle+OTS checkpoint) is what’s now live.

[chugarchugarr]

claim_receipt should not be requester-signed. It attests that the provider accepted the request into the authorized execution slot, so the provider is the authority that must sign it.

A requester signature can prove only attempted submission, e.g. a separate submission_commitment.

So:

submission_commitment = requester proves "I sent this"

claim_receipt = provider proves "I accepted this"

If there is a requester submission commitment but no provider claim receipt, that proves attempted submission – not admission, and not ATTESTED_NO_RESULT. Silence only acquires authority if the committed provenance mechanism makes that absence independently verifiable

[babyblueviper1]

@chugarchugarr, that split is right, and it answers the question I ended #12 with: no, the provider signs the claim, and the requester’s object is a different thing. I had folded two claims together.

For the record, here is what our /review mechanism does today, checked against the code rather than from memory. At admission it assigns an index and records (admission_index, accepted_at, request_digest, prev_receipt_hash) as a hash chain, before the verdict runs, with periodic Merkle checkpoints stamped to Bitcoin. That is your claim_receipt side: the provider commits “I accepted this” in a form a third party can check without calling us. There is no submission_commitment side at all. A caller that sends a request and gets silence has nothing of its own to point to. And admission happens inside the handler, so a failure before that point leaves no trace either. Both limits are stated in the module header, so this is a known edge, not a new finding.

Your last line is the part I think we can close: silence only carries weight if the absence is independently verifiable. Two separate pieces:

1. Requester side: an optional client-signed submission_commitment over (request_digest, attempt_id, sent_at) that the provider folds into the admission’s request digest. A claim_receipt then covers a specific commitment, and a commitment with no receipt evidences attempted submission only, as you say.
2. Provider side: index earlier, so errors after arrival are still recorded. That shrinks “silent drop” to network-level loss, which is the case only a requester-held commitment can evidence.

I haven’t built either. Before I do: does the formulary spec want submission_commitment as its own typed object with a field in claim_receipt that binds to it, or as an attribute of claim_receipt? I’d rather match your shape than invent ours.

[chugarchugarr]

I’d make submission_commitment its own typed object, with claim_receipt carrying a hash/ID that binds provider admission to that exact requester object. The reason is that submission_commitment has to remain independently meaningful precisely when no claim_receipt ever exists: it proves only “I attempted this exact submission,” while claim_receipt proves “I admitted this exact submission.” If submission_commitment exists only as an attribute of claim_receipt, then the pre-admission failure case loses the requester-side evidence we just separated out. So I’d keep them as separate authority-bearing objects, bind the receipt to the commitment when present, and never let the commitment alone imply admission or ATTESTED_NO_RESULT.

[babyblueviper1]

@chugarchugarr shipped it as you specified: submission_commitment its own typed object, claim_receipt carries a hash/ID binding to it, never the reverse, and the commitment alone never implies admission or a resolved state.

Live, code checked just now (not from memory):

submission_commitment = {type, version, request_digest, attempt_id, sent_at, requester_pubkey, sig}   -- requester-signed, BIP-340
submission_commitment_ref = sha256(canonical(full object))

/review accepts it as an optional field. When it verifies and commits to exactly the submitted artifact, our admission receipt (the hash-chained claim_receipt from the earlier part of this thread) binds to it by hash — folded into receipt_hash, and so into the periodic Merkle/OTS checkpoints. Real call, live server, this session:

• valid commitment → submission_commitment_status: "verified", receipt carries the bound submission_commitment_ref.
• same attempt_id replayed → duplicate_of_admission:<n> — one attempt binds to one admission, never re-bound.
• commitment for a different artifact → invalid:request_digest_mismatch, and the request is still admitted, just unbound. An invalid or missing commitment never blocks the review it rides on — matches “never let the commitment alone imply admission” the other direction too: absence or invalidity of a commitment doesn’t imply anything bad about the request either.

Named states throughout, no shared null: absent | verified | invalid:<reason> | duplicate_of_admission:<n>.

Backward-compatible: compute_receipt_hash only includes submission_commitment_ref in the preimage when one was bound, so every receipt hash issued before this change recomputes byte-for-byte identically.

Offline verifier (stdlib only, no install) + fixtures, cross-checked byte-for-byte against the production module:

preaction-governance-conformance/examples/submission-commitment at...

python3 tools/verify_submission_commitment.py examples/submission-commitment/events/valid_commitment.json / tampered_commitment.json — PASS / FAIL respectively, exit codes 0/1.

Real gap I found doing this, not asking you to solve it: claim_receipt here is currently derived from a provider-side hash chain (admission_chain.py), not itself a discrete signed object the way submission_commitment is. If the formulary spec wants claim_receipt to be independently portable (something a requester can hold and show a third party without querying us), that’s a different shape than what I have — happy to build that too if useful, but wanted to flag the asymmetry rather than imply parity that isn’t there yet.