AI Assurance Catalog
An open list of what must be true of an AI application before it is trusted in production.
1 · Purpose
The AI Assurance Catalog (AAC) is an open, machine-readable list of test obligations for AI applications, sorted by the kind of application: chatbot, document extractor, tool-using agent and so on. It is for architects, test leads and auditors who must say which tests an AI system owes before it goes live. Standards such as ISO/IEC 42001 and the EU AI Act say a system must be validated, and threat lists say what can go wrong, but nobody publishes which tests a given kind of application owes. This catalog is that missing link, and it turns a team's existing test results into one coverage report, gaps included.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Write a build option for one obligation that has none
- Check one crosswalk row against the standard's text
- Add a test case to the promptfoo adapter
4 · Requirements
What it must do
- Lists 118 obligations, each with a permanent identifier (AAC-0001 and so on) that never changes.
- Marks 36 obligations as owed by every AI application and tags the rest by the kind of application that owes them.
- States for each obligation which kind of check can produce a verdict and at which stage of the life cycle.
- Maps obligations to OWASP, NIST AI RMF, ISO/IEC 42001 and the EU AI Act by identifier, and lists what does not map.
- Turns results from JUnit, promptfoo, DeepEval, Langfuse and LangSmith into one coverage report.
- Offers concrete build options for 60 of the obligations and plain-language guidance for 9.
- Renders the whole catalog as a filterable web page.
How well it must do it
- It is a join table, not a framework: it cites controls, threats and metrics and never restates them.
- A linter fails the build if a product name appears in an obligation's normative text.
- Specification text is CC BY 4.0 and tooling is Apache 2.0, so companies can adopt both.
- The repository never ships scorers, runners or dashboards; if its code ever judges a system, the boundary is crossed.
- Every release is archived on Zenodo with a DOI, so a specific version can be cited.
- The obligations, the mappings and the build options are versioned separately.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS |
|---|---|---|---|
| Obligation files | YAML in Git ✓ | — | — |
| Schema and linter | JSON Schema, Node.js linter ✓ | — | — |
| Evidence adapters | JUnit, promptfoo, DeepEval, Langfuse, LangSmith ✓ | Azure AI evaluation results as evidence (not built) | Bedrock evaluation results as evidence (not built) |
| Report builder | Node.js report tool and GitHub Action ✓ | — | — |
| Renderer | static HTML on GitHub Pages ✓ | — | — |
7 · What needs to be done
Phases 0 to 5: the catalog itself 5 of 5 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Taxonomy, schema and linter | done | M | Specification design |
| 118 obligations with permanent ids | done | L | AI testing |
| Crosswalks to OWASP, NIST AI RMF, ISO/IEC 42001 and the EU AI Act | done | L | Standards and compliance |
| Adapters for JUnit, promptfoo, DeepEval, Langfuse and LangSmith | done | M | JavaScript |
| Release 0.17.1 archived on Zenodo with a DOI | done | S | Release management |
Build options and guidance 1 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Build options for 60 of 118 obligations | done | L | AI engineering |
| Write build options for the 58 obligations that have none | todo | L | AI engineering |
| Write plain-language guidance for more obligations (9 written so far) | todo | M | Technical writing |
Patterns as portable detection rules 0 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Define a rule format over OpenTelemetry GenAI traces | todo | M | Observability |
| Rewrite the support agent's online rules in that format | todo | M | Python |
| Run the same rules on a second backend and compare verdicts | todo | L | Observability |
| Collect a trace for every rule that must fire it and one that must not | todo | M | AI testing |
Adoption 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Find three teams that publish a coverage report with their gaps | todo | L | Community |
| Add an adapter for one more test tool, such as Ragas or Inspect | todo | S | JavaScript |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Obligation author | 2 | 3-5 hours/week | Writes and reviews obligations and build options from real failures. |
| Standards mapper | 1 | 2-4 hours/week | Keeps the crosswalks to ISO, NIST, OWASP and the EU AI Act accurate. |
| Adapter developer | 1 | 2-4 hours/week | Writes adapters that turn another tool's results into evidence. |
9 · What you'll learn
- How AI risk frameworks, threat lists and test tools fit together, and where they leave a gap.
- How to write a requirement that can be tested, without naming a product.
- How to map one standard to another honestly, including what does not map.
- How an audit treats a known gap differently from an unknown one.
- How to keep a public specification machine-readable and linted.
Level-up questions
- Which obligations does a retrieval chatbot owe that a document extractor does not, and why?
- Why does the catalog use flat ids instead of ids grouped by application type?
- What makes a coverage report acceptable to an ISO/IEC 42001 auditor?
- Where does an obligation end and a metric implementation begin?