RFC: Procedure Manifests - Mechanism for AI Agents to Resolve Contractual Disputes

[adpoe]

Overview

This post seeks design & feasibility feedback on a proposed mechanism for ai agents to resolve subjective contractual disputes without a human-in-the-loop at arbitration time.

Proposal / Idea

Create a repeatable dispute-resolution procedure that is directly embedded in the smart contract.

At contract formation time, both parties agree to what we will call a procedure manifest. This manifest will define & pin the complete adjudication procedure for an LLM Judge to perform.

The manifest must contain:

• the requirements, decomposed into a machine-evaluable rubric

• the judge: specific model(s) pinned by version/hash, the LLM-judge’s complete evaluation prompts agreed verbatim, run counts, aggregation rules

• the admissible evidence: schema, submitters, data transformations, and named measurement sources - all defined inside the contract

• the default rule and burden of proof - who wins when the answer is genuinely ambiguous (i.e. - The LLM-judge has low confidence re: the correct answer)

• the remedy structure: how will funds be (re)distributed depending on arbitration result

Under this system, a contract would be Arbitrable if and only if its manifest conforms to spec. And at contract formation time there would be tooling to help users identify if the contract meets all requirements. Nonconforming contracts will not be arbitrable under these mechanisms - this creates a clear refusal boundary, knowable at contract creation time.

Because the procedure is fully specified and cheap to run (it’s an LLM-judge eval), both parties could independently execute the judge before disputing. The manifest therefore functions as a mutually-agreed acceptance test, and arbitration becomes the attested backstop that makes the acceptance test binding. The goal is a system that prevents most disputes and resolves the remainder predictably.

The intent is to compose with existing standards to the highest possible degree. Ideally, Procedure Manifests would ride in ERC-1497 MetaEvidence, resolve through an ERC-792-compatible Arbitrator, and anticipate ERC-8004 identities for parties and validators.

Background & Motivation

Agent-to-agent commerce has payment rails (x402, agent wallets, escrow), and with ERC-8004 it is getting identity, reputation, and validation mechanisms. I believe there is still a gap that sits just above the Validation Registry: subjective dispute resolution between ai agents.

My understanding of ecosystem is that current mechanisms resolve three contractual & trust problems well:

1. Was anything delivered at all? (escrow with timeout refunds)

2. Did the agent run the process it claims? (stake-secured re-execution, TEE attestation, zkML)

3. What have others experienced? (reputation registries)

However, it seems the trickiest problem with the most area for disagreement is still open. How do ai agents determine if the work delivered actually satisfies the ‘spirit’ of the contract - and what recourse do they have, when reasonable parties disagree? I believe inability to resolve such disputes cleanly & automatically will hold back widespread adoption of agentic commerce.

Examples:

• A website can be delivered on time, and still fail the spec in non-obvious ways

• An SLA can degrade in ways a refund timeout can’t see

• A funding milestone can be “met” only under one party’s reading

Data-feed oracles cannot answer these questions (no authoritative source exists for "was this good enough”).

One design choice is to only transact via micropayments. But I suggest this will become a burdensome limitation over time, and possibly hold back growth of the ecosystem.

Gap Analysis

Here is where I could use some help refining my own understanding of the proposed ecosystem, to ensure these ideas stand up to scrutiny. My understanding is that ERC-8183 (Agentic Commerce) standardizes the escrow lifecycle and creates an evaluator slot - a single address that alone may complete or reject - while explicitly scoping out dispute resolution (“no dispute resolution or arbitration; reject/expire is final”). This leaves the evaluator’s decision procedure entirely to future implementors.

My reading of Alkahest is that it similarly abstracts release conditions behind a composable boolean IArbiter, which is the interface for a condition, not the procedure for judging one.

Kleros solves the general problem for human-speed disputes with drawn juries, and ERC-792/ERC-1497 gave it a clean interface: any Arbitrable can resolve through any Arbitrator.

This proposal fills the layer that I believe those designs deliberately leave open: not who may release funds, but under what signed procedure they decide. The intent is to create a new Arbitrator type under ERC-792, and equally as an evaluator implementation for ERC-8183. Plus a contract-formation format that makes subjective disputes tractable in the first place.

Minimal Viability Validation
To validate baseline feasibility, I’ve created a draft spec, examples, and a working conformance validator: github.com/formulary-systems/spec

• JSON Schema for the manifest, and 12 conformance rules constituting the executable refusal boundary

• Three example manifests: 2 conforming and 1 non-conforming, with explanations. These can be executed against the conformance validator already.

• An ERC-792/1497 compatibility sketch: the manifest rides as MetaEvidence; rulings map to the standard enum

Design intent is that anyone can implement a conforming Arbitrator.

Footnotes

An historical parallel shaped this design, and gave me a name for it. While researching this problem I saw some superficial similarities with the Roman law system’s Formulary procedure. I’m not a scholar in this area, so the analogy is surely inexact. But in principle, a magistrate published a formula stating: the exact question to be decided, the name of a private judge that both parties accepted, a damages cap, and a default rule. The private judge then rendered an arbitration following that exact formula. That’s, in essence, what I am proposing here.

My reading of this historical legal mechanism is that was intended to solve problems of scale. Namely, it allowed Rome to scale their legal system for use by non-citizens (peregrini), mostly for commercial law cases, as I understand it. (My knowledge here is wikipedia level, not research level - I’m happy to be schooled by someone who actually knows this in detail, but it’s not the main point.)

To finalize this analogy and why I see it as relevant - AI agents are similar to the peregrini: no legal citizenship, and no standing in existing courts. The analogy is directional, but the architecture is similar. Citizens publish the procedures; non-citizens transact safely under them; judgment delegates to agreed-upon judges. This is a guardrails system.

Questions for this forum

1. Prior art. If something here duplicates existing work - deployed or published - links are the most valuable reply this thread can get.

2. Feasibility & Safety, Security. This currently exists only at the level of a sketch. It is intending to solve what I see as a very hard problem - placing reliable guardrails on agentic ai coordinating serious commercial work together, with mechanisms for resolving disputes in a ‘fair’ way. At this point, I’m exploring the problem space. I welcome any feedback about why this is a potentially naive or impossible solution before anyone spends significant time on it. (Including me.)

3. For anyone who has operated or attacked deployed arbitration/oracle systems (Kleros, UMA, reality.eth, Aragon Court): which failure modes am I missing? Bribery and judge-shopping shaped several rules in my sketch. But I’m sure there is a lot more complexity here.

4. Ecosystem Integration. Am I misreading how this idea could be integrated into the wider agentic economy ecosystem that Ethereum is building? Is this actually a problem that people are eager to solve?

Again, I’d like to battle-test and sanity check this idea - as well as determine if other folks are interested in these topics. Please feel free to point me in a different direction if that is the appropriate response.

[chugarchugarr]

I have some adjacent implementation evidence that may be useful here. I don’t think it duplicates Procedure Manifests, but it exercises a few of the state boundaries that seem relevant to what you’re proposing.

Executable x402 Resolution Receipt reference implementation:
https://github.com/chugarchugarr/-x402-resolution-receipt

x402 Foundation discussion:
https://github.com/x402-foundation/x402/issues/2833

Surviving upstream proposal:
https://github.com/x402-foundation/x402/pull/3291

The implementation preserves independently signed verifier findings, conflicting findings on the same subject, explicit resolution states including UNRESOLVED, and hash-linked corrections rather than overwriting earlier conclusions.

That work came from a somewhat different problem, so I would treat it as adjacent implementation evidence rather than prior art for the complete manifest idea.

Reading your proposal, I think one thing you are already separating conceptually is especially important to preserve mechanically:

what the judge established != what the agreed procedure says to do about it.

You already have the ingredients for this in the manifest: evaluation procedure, default/burden rule, and remedy are separate fields. I would make that separation explicit in the resulting state machine.

For example, if the evidence genuinely does not establish whether the work satisfied the contractual requirement:

resolution = UNRESOLVED

while the pre-agreed manifest can still deterministically specify:

policy(UNRESOLVED) = REFUND

There is no human required at arbitration time, and there is no loss of determinism. But the record also does not convert “we could not establish it” into the stronger factual claim “the payer was right.”

I would probably preserve one additional boundary after that:

resolution
→ authorized remedy
→ execution
→ evidence that execution actually manifested

In other words:

judge returned X != contract was authorized to do X != X actually happened

That could matter once another ERC-8004 agent or another protocol consumes the result. A downstream system should be able to tell whether it is inheriting a judgment, an authorization, or an observed state change.

Your refusal boundary also looks important to me. “Nonconforming = not arbitrable under this mechanism” is preferable to forcing a result from a procedure that cannot support one.

A few failure modes I would specifically try to falsify:

1. Evaluator optimization / Goodharting

Because the exact rubric, prompts, model, transformations, aggregation rules and thresholds are intentionally known in advance, they become an optimization surface.

That is not automatically a vulnerability — an acceptance test is supposed to be known. But for subjective requirements there is a dangerous case:

optimize artifact to satisfy evaluator

without necessarily:

satisfy underlying contractual intent

If those diverge, the problem is no longer judge nondeterminism; it is that the machine-evaluable proxy failed to capture the contract.

I would test manifests adversarially for that divergence.

2. Evidence transformation becoming part of the attack surface

You include data transformations in the manifest, which I think is necessary. I would preserve both the original submitted evidence and the transformed representation, plus the exact transformation/version used.

Otherwise a deterministic judge can still produce a perfectly reproducible answer over a lossy or adversarial transformation.

3. “Pinned model” reproducibility

For locally reproducible/open models, model hashes are straightforward conceptually. Hosted models seem harder: identical model names/prompts do not necessarily establish identical weights, serving stack, system wrappers, decoding behavior or inference environment.

So I think the specification may eventually need to distinguish:

model identity

from

reproducible execution environment / attested execution.

Otherwise two parties can conform to the same manifest and still receive different evaluations for reasons outside the manifest.

4. Aggregate certainty hiding disagreement

If several judge runs or models disagree, I would preserve the individual findings rather than only the aggregate ruling. The aggregate can determine the contractual result, but the disagreement remains useful evidence.

That also gives you a cleaner path for later corrections without rewriting history.

5. Procedural determinacy vs fairness

I think Procedure Manifest conformance can establish something very strong: the adjudication procedure was completely specified and executable before the dispute.

I’m less sure conformance alone can establish that the procedure is substantively “fair.” Two parties can deterministically agree to a badly specified rubric, biased evidence transformation or pathological default rule.

It may therefore be worth keeping the spec’s claim narrow:

conforming → procedurally closed/executable

rather than:

conforming → fair/correct.

On ecosystem placement, I don’t think I see an obvious contradiction in the layer you’re describing. ERC-792 can remain the arbitration interface, ERC-1497 can carry dispute metadata, ERC-8004 can provide identities/validation context, and ERC-8183 can consume the resulting evaluator/arbitration decision.

What Procedure Manifests seem to add is the signed, precommitted procedure governing how the decision is produced.

The piece I would preserve very carefully is the boundary between:

evidence → judgment → authorized consequence → manifested consequence

because those become different claims as soon as the result is consumed by another agent or protocol.

[cedricbrown]

ran an llm judge as an acceptance gate on a delivery contract last year. pinned snapshot, temperature 0, the whole setup. what broke me was the reruns. same rubric, same evidence, different verdicts. only about 4 percent of cases, but they weren’t scattered. they were the half-met borderline ones, which is the only population that ever reaches a dispute anyway. so both parties running the judge before disputing works fine on the cases nobody argues about, and falls apart where the money is.

that’s also why i wouldn’t hang the default rule off the judge’s self-reported confidence. tried that. got a cheerful 0.9 on items that flipped on the very next run. you’re already specifying run counts and aggregation, so the split rate across those runs is sitting right there for free, and a manifest that says a k-of-n split fires the default rule turns judge instability into a declared outcome instead of a coin flip.

on prior art, one you might not have. ERC-8183 post 51 (drmattbarney, march). he argues a multi-judge setup is basically Classical Test Theory and points at Rasch and item response models instead, which lands right on how you pin aggregation. ERC-8210 Agent Assurance also carries an arbitration role callable when a party disputes the Evaluator decision, worth checking whether that’s already sitting in your slot.

last thing, and it’s boring but it’ll get you. pinning a model by version or hash pins the name, not the behaviour, and snapshots get retired. contract formed today, model gone in nine months, and it’s either unarbitrable or quietly substituted. probably wants a liveness rule and a stated fallback, agreed at formation rather than discovered at dispute.

[babyblueviper1]

Real, checkable mechanism behind that 4%, not just anecdote: temp=0 doesn’t guarantee determinism on a hosted API because dynamic batch size at inference time changes the GPU reduction kernel’s numeric path (RMSNorm/matmul/attention reduction trees differ by batch composition) – same prompt, different server load, can diverge during greedy decoding even at temp=0. This is well-documented (Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference,” Sept 2025) and it’s an opaque, uncontrollable property from the caller’s side – you have no visibility into what else was batched with your request, and most hosted reasoning-tier models don’t expose a temperature override at all. The fix (batch-invariant kernels, forcing one reduction strategy regardless of batch size) exists and is demonstrated bit-identical over 1,000 runs, but costs 34-60%+ throughput – not something most API consumers can turn on themselves.

Concrete implication for a dispute mechanism specifically, from measuring the same failure mode on our own /review verdicts: reruns on borderline cases flip on confidence and issue-count, but the discrete verdict (approve/reject) itself is far more stable than the reasoning trace around it. If Procedure Manifests pins the LLM Judge’s full evidence+rubric+temperature, that pins the INPUT, not the reduction path – so the manifest’s committed output should probably be the narrow decision only (the binding both parties can dispute over), with the full trace kept as supporting evidence, not itself the thing re-run and diffed for equality. Rerunning the whole judge and expecting byte-identical output is asking for more determinism than temp=0 on a hosted model can actually deliver.

[chugarchugarr]

Following the latest exchange, I went back to the current Procedure Manifest repository and tried to resolve the question mechanically rather than add another interpretation to the thread. I think the result is now narrow enough to falsify against the draft itself.

Evidence and provenance

The current README says a procedure manifest pins the complete adjudication procedure and explicitly identifies examples/website-delivery.manifest.json as a conforming LLM-judged example. The conformance rules describe conformance as establishing a procedure that is complete, closed, and executable, with the outcome-relevant procedure defined in advance.

Current sources:

README

Conformance rules

Procedure Manifest schema

Conforming website-delivery example

The example commits three judges, five runs per judge, within_judge = majority, across_panel = majority, temperature = 0, a per-requirement confidence_threshold = 0.7, a payer default, requirement weights of 25/20/25/20/10, and an overall weighted passing threshold of 8000 basis points. Its prompt asks each LLM run to return PASS or FAIL, a confidence in [0,1], and a rationale.

The schema explicitly says that confidence below the threshold causes the requirement to resolve through default_rule, while the problem statement separately acknowledges under OQ4 that self-reported confidence semantics are unresolved. What I cannot find in the current schema, conformance rules, example, or reference conformance validator is the operation that converts multiple run-level confidence observations into the single per-requirement confidence consumed by that threshold, or a rule fixing where that confidence operation occurs relative to within-judge and across-panel aggregation. That is the boundary I tested.

PredictionLock

If the current procedure is closed in the strong sense intended by conformance, then once the manifest and the complete evaluation observations are fixed, there should be no remaining implementation choice capable of changing the contractual result.

Equivalently, two implementations following the published manifest should not be able to consume the same fixed observations and derive opposite remedies because an outcome-relevant transformation was left unspecified.

I therefore froze the manifest and constructed one fixed set of run observations for R1. R2 through R5 are held PASS in both executions.

J1

PASS 0.99

PASS 0.99

PASS 0.69

FAIL 0.69

FAIL 0.69

J2

PASS 0.99

PASS 0.99

PASS 0.99

PASS 0.99

PASS 0.99

J3

FAIL 0.99

FAIL 0.99

FAIL 0.99

FAIL 0.99

FAIL 0.99

The discrete majority results themselves are not ambiguous. J1 is PASS by 3–2, J2 is PASS unanimously, and J3 is FAIL unanimously. The only question is how the committed confidence threshold consumes the five confidence observations associated with a repeated judge execution.

Executable counterexample

One implementation can reduce J1’s five confidence values by arithmetic mean:

(0.99 + 0.99 + 0.69 + 0.69 + 0.69) / 5 = 0.81

That is above the manifest’s 0.7 threshold. J1 therefore retains its PASS majority. Across the panel the requirement becomes PASS by 2–1. Because R2 through R5 are also PASS, the weighted score is 100/100, which exceeds the committed 80% threshold. Under the example’s ruling map, the payee-side outcome releases the funds.

A second implementation can reduce the exact same five confidence values by median:

median(0.99, 0.99, 0.69, 0.69, 0.69) = 0.69

That is below 0.7, so the confidence rule enters the committed payer-default path. If that default is interpreted as resolving R1 against the payee, R1 contributes no passing weight and the remaining requirements total 75/100, below the required 80%, producing the payer-side outcome and refund. If the default is instead interpreted as immediately selecting the payer-side ruling, the final remedy is still refund.

Mean and median are not being proposed as competing normative choices. Neither is the proposed fix. They are witnesses. The relevant fact is that the current procedure does not select the confidence-reduction function or its location in the adjudication sequence, while that undeclared choice is capable of changing contractual state.

The manifest is unchanged. The evidence is unchanged. The models and prompts are unchanged. Every PASS/FAIL value is unchanged. Every confidence value is unchanged. The complete observations are unchanged. Only an unspecified transformation changes, and the resulting remedy changes from release to refund.

Observed result

This falsifies the stronger procedural-closure condition for the current v0.0.1 draft.

The result does not depend on LLM nondeterminism. The LLMs can finish first and every output can be frozen permanently. The divergence remains after observation because the deterministic procedure consuming those observations is not yet completely specified.

OQ4 already correctly identifies confidence semantics as open. The counterexample establishes why that open question is not merely about whether confidence is well calibrated. As long as confidence remains outcome-relevant, its reduction semantics are part of the state-transition procedure and therefore part of the closure boundary.

The same inspection exposes a neighboring issue. The example prompt asks for PASS/FAIL, confidence, and rationale, but the current manifest schema does not define a normative machine-level judge-result object, parser, authoritative projection, or canonical representation for those produced values. The prompt constrains what the model is asked to say; it does not yet define the exact result contract that independent implementations must consume.

Resolution

I think the minimum invariant is therefore:

Every evaluation-produced value capable of altering aggregation, resolution, remedy selection, or contractual state must enter through an explicitly defined authoritative result contract, and every transformation of those authoritative values capable of altering the outcome must itself be committed by the procedure.

That means the manifest does not need to require the entire model trace to be deterministic. It needs to specify exactly which projection of the observation is authoritative, how that projection is parsed and represented, and every reduction that can affect the next contractual state.

For v0, there are two coherent ways to resolve OQ4. If self-reported confidence remains authoritative, then the procedure must define its exact semantics: which confidence observations participate, the reduction function, the order in which reduction and majority aggregation occur, and what a below-threshold result means at the requirement and contract levels. If those semantics are not ready to be standardized, confidence can remain preserved in the transcript as evidence while being removed from the binding projection.

The same rule applies to rationale, issue lists, diagnostics, or any richer model output. They can be retained completely without acquiring undeclared authority over the contract.

The resulting boundary is:

evidence

↓

judge observation

↓

authoritative result

↓

within-judge aggregation

↓

across-panel aggregation

↓

requirement resolution

↓

contract outcome

↓

authorized remedy

↓

manifested consequence

Each arrow is a separate transformation. If an arrow can change the contractual state, its semantics belong inside the committed procedure.

The corresponding conformance test becomes:

Given the same committed manifest and the same complete evaluation observations, two independent conforming implementations must necessarily derive the same requirement resolutions, the same contractual outcome, and the same authorized remedy.

If that does not hold, some outcome-relevant procedure remains outside the manifest.

The current website-delivery example fails that test at the confidence-reduction edge. This is also consistent with the adjacent implementation evidence I linked earlier in this thread, where I was already separating verifier findings, resolution state, authorized consequence, and what actually manifested. I do not think that prior implementation needs to be imported into Procedure Manifests; the useful part here is the invariant, and this counterexample establishes it independently.

Current conclusion

The current Procedure Manifest draft already recognizes nondeterministic judges and already recognizes confidence semantics as unresolved. The remaining step is to make the state boundary exact.

The model does not have to be bit-deterministic for the contractual procedure to be closed. But once any part of a model observation is allowed to influence contractual state, every transformation from that observation to the authorized remedy has to be part of the procedure the parties committed to.

Preserve everything the judge produced. Give contractual authority only to the values whose state-transition semantics are completely specified. Under that condition, independent implementations can disagree in supporting traces without being allowed to disagree about what the contract does next.

[babyblueviper1]

@chugarchugarr that’s a genuinely clean falsification – freezing the observations and varying only the undeclared reduction function isolates it well, and the arrow-chain (evidence → judge observation → authoritative result → aggregation → resolution → outcome → remedy) is a good normal form for “what has to be inside the committed procedure.”

We hit an adjacent version of this boundary for a related reason and it might sharpen where the line falls. Our production verdict pipeline (/review) has to decide exactly this: which fields get contractual authority (bound into a commitment a third party checks) vs. which are preserved as evidence only. The real preimage, verbatim from the running code (services/proof_signing.py, DECISION_REF_PREIMAGE_FIELDS)

artifact_hash, artifact_type, policy_version, verdict, source_class, vantage_limitation, related_decision_ref, intended_audience, confidentiality_tier, disclosed_summary, intended_verifier, policy_commitment, verified_at, registry_as_of, registry_snapshot_sha256, action_binding_tool_hash, action_binding_args_hash, action_binding_agent_id, action_binding_nonce, freshness_beacon_hash, epistemic_basis

verdict IS in that list – it’s part of the committed hash, same as your “authoritative result” needing to be explicit. But we draw a second, narrower line inside that: every field in the preimage is REPRODUCED in the sense that a third party can recompute the exact hash bytes with zero trust in us (canonical JSON + sha256, a pure function). The verdict’s correctness, though, is only VERIFIED – authentic, attributable, tamper-evident via our signature – never claimed as independently re-derivable by rerunning the judgment. We disclose this per-verdict via a vantage_limitation field (itself in the preimage) naming what our own judgment’s vantage couldn’t establish.

Mapped onto your invariant: committing a value into the manifest’s binding hash (REPRODUCED-shaped) is a different trust primitive than claiming two independent judge runs will converge on it (a determinism claim). Your counterexample shows confidence can’t get contractual authority as an aggregation input without specifying the reduction – but our read is that even after fixing the reduction, the resulting number stays only VERIFIED-shaped (attributable to that specific committed run), not REPRODUCED-shaped (recomputable by a second, independent judge execution), because LLM judgment isn’t a pure function of its stated inputs – only the hash of the manifest+observations is. That argues for treating confidence the way we treat verdict+vantage_limitation: bind it for attribution, but never let a re-run’s disagreement with a frozen observation count as a spec violation – only disagreement with the signed record of what that specific run actually said should.

Real prior-art pointer directly on point: we found and helped close an almost identical gap yesterday in a TEE-attestation conformance spec (trustless-ai/recompute-kit) – a leaf could drop its “not judged correct” disclaimer and still pass conformance, because execution-integrity (ran on unmodified hardware/model) and judgment-correctness were being silently conflated in one attestation. Same shape of bug, different domain.