CAAS: Continuous Audit as a Service — reproducible security evidence for open-source cryptographic libraries

Magicians Primordial Soup
, , , ,
1

Hi everyone,

I would like to share the audit model I am using in UltrafastSecp256k1: CAAS — Continuous Audit as a Service.

The idea is simple:

Security claims should not live only in a PDF. They should become reproducible, executable evidence.

This post is not an argument against human review. Human review is valuable. The problem is treating a one-time external audit report as the final source of truth for a living cryptographic project.

For a project that changes continuously, a static audit naturally decays over time.

CAAS is my attempt to make the audit process continuous, public, reproducible, and extensible.

Motivation

The traditional audit model often looks like this:

  1. project reaches a snapshot

  2. external auditor reviews it

  3. PDF report is published

  4. project continues changing

  5. the PDF becomes stale

  6. new bugs may appear outside the audited snapshot

This model also creates a barrier for independent open-source developers. If a project cannot afford an expensive audit, it may be treated as unreviewable regardless of how much reproducible evidence it provides.

I think open-source crypto needs a better default model.

The goal should be:

Do not trust the author. Do not trust the auditor blindly. Reproduce the evidence.

Core principle

CAAS follows one rule:

Every security claim should become executable evidence.

That means:

  • every invariant should become a test

  • every known exploit class should become a regression

  • every fixed bug should become permanent coverage

  • every backend difference should be tracked

  • every benchmark claim should be reproducible

  • every audit artifact should include provenance

  • every reviewer objection should become either a test, a fix, a documented limitation, or a benchmark correction

The audit is not something attached to the repository after the fact.

The audit is part of the repository.

What CAAS includes

In UltrafastSecp256k1, CAAS currently includes several layers:

  • invariant tests

  • exploit-style regression tests

  • differential testing

  • Wycheproof vectors

  • BIP-340 / RFC6979 / Ethereum vector coverage

  • invalid input grammar tests

  • fuzzing

  • LibFuzzer harnesses

  • mutation testing

  • static analysis

  • CodeQL / SonarCloud / coverage tooling

  • constant-time testing

  • dudect-style timing checks

  • ctgrind / Valgrind style checks

  • Cryptol formal specs

  • backend parity matrices

  • C ABI negative tests

  • GPU backend parity checks

  • audit-quality scanners

  • provenance reports

  • CI gating policy

  • bug capsule format

  • research monitoring

The production library does not depend on this tooling at runtime. CAAS is the audit and development layer, not the shipped runtime.

Bug handling philosophy

The project does not claim to be bug-free.

That would not be credible.

The actual security posture is:

Bugs can exist. When they are found, they must become reproducible, fixed, documented, and permanently added to the regression/audit corpus.

A good bug report should produce one or more of:

  • a failing test

  • an exploit PoC

  • a regression vector

  • a new invariant

  • a code fix

  • a benchmark correction

  • a documented limitation

  • a new CI gate

In other words:

Bug-free software is not a credible claim. Public, reproducible bug handling is.

Why this matters for Ethereum

Ethereum has a large and diverse software ecosystem:

  • clients

  • wallets

  • account abstraction tooling

  • relayers

  • rollups

  • proving systems

  • bridges

  • signing services

  • hardware wallets

  • libraries

  • developer tools

Many of these systems depend on cryptographic libraries or cryptographic assumptions. A traditional one-time audit can be useful, but it does not scale well across a fast-moving ecosystem.

For Ethereum infrastructure, I think we need more projects to expose:

  • exact security claims

  • executable invariants

  • differential references

  • regression history

  • known limitations

  • backend parity results

  • reproducible benchmark artifacts

  • public bug-fix history

This is especially important for libraries that aim to support many platforms, such as x86-64, ARM64, RISC-V, embedded devices, CUDA, OpenCL, and Metal.

A library that runs everywhere needs an audit model that follows it everywhere.

AI-assisted review

CAAS can use AI-assisted review, but AI findings are not treated as truth.

AI is useful for exploration:

  • finding suspicious patterns

  • proposing missing invariants

  • generating adversarial cases

  • comparing code paths

  • reviewing documentation consistency

  • suggesting exploit classes

  • checking whether a fix should become a regression test

But an AI output only becomes part of the audit model if it turns into deterministic evidence:

  • a test

  • a PoC

  • a formal property

  • a CI gate

  • a documented limitation

  • a reproducible benchmark correction

The authority is not the model.

The authority is the evidence.

Audit-quality testing

One important part of CAAS is testing the tests themselves.

It is possible to have many tests that do not prove much. For example:

  • tests that always pass

  • tests that ignore return values

  • adversarial tests that do not assert rejection

  • statistical tests with weak thresholds

  • tests where the condition and message disagree

  • tests that exercise only happy paths

CAAS includes audit-quality scanning to detect these patterns.

The goal is not only to count tests. The goal is to improve the quality of the evidence.

Static audits vs living audit corpus

A static audit asks:

Was this snapshot reviewed?

A living audit corpus asks:

What claims are currently covered? What changed? What regressed? What new exploit class was added? Which platforms diverged? Which assumptions are still untested?

That is a different security model.

It is not a replacement for expert review. It is a better substrate for expert review.

A human reviewer should be able to say:

Here is a missing assumption. Add it as a test.

And then that assumption should remain permanently covered.

What I am looking for

I would like feedback from Ethereum developers, auditors, wallet developers, ZK engineers, and client/tooling maintainers.

Questions:

  1. What would make CAAS-style evidence useful for Ethereum libraries?

  2. Which exploit classes should every secp256k1/ECDSA library permanently test?

  3. Which Ethereum-specific signing/recovery vectors should be mandatory?

  4. What should a reproducible audit artifact contain?

  5. How should projects present known limitations without creating unnecessary fear?

  6. Can this model help smaller open-source projects get credible review without requiring a large up-front audit budget?

  7. What would you expect before relying on such a library in wallet, ZK, relayer, or infrastructure tooling?

Repository

Project:
https://github.com/shrec/UltrafastSecp256k1

I am especially interested in adversarial feedback: missing gates, missing Ethereum vectors, weak claims, unclear boundaries, and anything that should be converted into deterministic evidence.

The goal is not to say “trust this library.”

The goal is:

make the claims reproducible, make the bugs permanent regressions, and make the review process continuous.