i turned the SHALL rules in the Specification section into a foundry test suite and ran it on forks against five escrows: the reference code in the ERC, Circle’s arc testnet deployment, ArcBounty’s arc mainnet escrow, G2 Jobs on arc mainnet (@apexfaucet-hub) and Virtuals’ AgenticCommerceV3 on base. nothing is sent on chain, every job lives and dies on the fork.
of 29 SHALL rules, the reference code, Circle and ArcBounty pass 25, G2 28 and Virtuals 22. the matrix and what each gap means are in docs/CONFORMANCE.md of github.com/vijaygopalbalasa/judge-protocol
the four the reference code misses are worth fixing in the ERC itself, since deployments copy it. fund has no expectedBudget, the front-running check @javierpmateos raised in #425, and it’s the only gap here that can move money: a client who approved more than the agreed price can be charged a budget the provider raised just before. only the provider can call setBudget. fund accepts a zero budget. submit accepts an Open job whose budget is 0.
G2’s one gap and most of Virtuals’ look deliberate: settle-on-silence instead of a refund for submitted work, the 15 minute evaluator grace period, jobs with no evaluator.
the suite checks itself on every run: an escrow written from the spec alone passes all 30 tests, and 34 planted single-rule violations each make at least one fail. adding another deployment is one small adapter contract. happy to add yours, or to hear where i read a rule wrong.
Thanks @vijaygopalbalasa for running this. I checked the four reference-code gaps against the current erc-8183.md and they hold:
fund: the Specification says it SHALL revert on a zero budget and when job.budget != expectedBudget, but the reference fund(uint256 jobId, bytes optParams) takes no expectedBudget and transfers whatever job.budget is at execution. The shorter snippet earlier in the ERC, fund(uint256 jobId, uint256 expectedBudget), does have the check, so the document currently disagrees with itself.setBudget: the Specification allows client or provider; the reference reverts unless the caller is the provider.submit: the Specification requires Funded; the reference also accepts Open with a zero budget.
One thing for whoever fixes it: adding expectedBudget changes the selector of fund, so escrows that already copied the reference keep the old ABI and the old behavior. Until they redeploy, the protection a client actually has is its ERC-20 allowance: approving exactly the agreed amount makes a raised budget fail at transferFrom instead of charging more.
Happy to open a PR against the reference implementation if the authors want it.
@javierpmateos thanks for checking them against the text. one data point for whoever writes the fix: of the five escrows in the suite, two already have the front-running guard, G2 Jobs on arc mainnet and Virtuals’ AgenticCommerceV3 on base, and both take fund(uint256,uint256,bytes), the same shape as fund(jobId, expectedBudget, optParams?) in the Specification section. so a reference that adopts that signature would match an ABI that’s already deployed. the three without the guard are the reference code, circle’s arc testnet deployment and arcbounty’s mainnet escrow.
agreed on the allowance: docs/CONFORMANCE.md already tells clients of those three to approve exactly the agreed budget right before funding. if the PR goes ahead, i’ll run the suite against your branch and post the result here.