AI Assurance Catalog

An open list of what must be true of an AI application before it is trusted in production.

Active · join meAI Engineering

6 of 14 tasks done

1 · Purpose

The AI Assurance Catalog (AAC) is an open, machine-readable list of test obligations for AI applications, sorted by the kind of application: chatbot, document extractor, tool-using agent and so on. It is for architects, test leads and auditors who must say which tests an AI system owes before it goes live. Standards such as ISO/IEC 42001 and the EU AI Act say a system must be validated, and threat lists say what can go wrong, but nobody publishes which tests a given kind of application owes. This catalog is that missing link, and it turns a team's existing test results into one coverage report, gaps included.

For: AI architects, QA and test leads, risk and compliance teams, auditors preparing ISO/IEC 42001 or EU AI Act evidence.

2 · Business benefits

Audit evidence without a Word documentTest results become a coverage report that maps to ISO/IEC 42001, NIST AI RMF, OWASP and the EU AI Act.
Fewer surprises in productionThe obligations come from real agent failures, so known risks are tested before launch, not found by customers.
Faster design reviewsAn architect picks the application type and gets the list of tests it owes in minutes.
Honest gaps are a strengthA named, owned, dated gap is acceptable to an auditor; a gap nobody noticed is a finding.
Reuse the tools you already pay forResults from promptfoo, DeepEval, Langfuse, LangSmith or JUnit feed the report as they are.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Write a build option for one obligation that has none
  • Check one crosswalk row against the standard's text
  • Add a test case to the promptfoo adapter

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Obligation filesOne YAML file per obligation:statement, level, tags,Schema and linterChecks every file against theschema and the boundary rules.CrosswalksMap obligation ids to outsidestandards, with an explicitEvidence adaptersTranslate a test tool's resultsinto evidence for namedReport builderJoins evidence with theobligations an application owesRendererBuilds the browsable catalogpage from the YAML.validated on etest results as ev
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Obligation files → Crosswalks: ids referenced · Obligation files → Report builder: what is owed · Obligation files → Renderer: published page

6 · Tech stack, per platform

ComponentOpen stackAzureAWS
Obligation filesYAML in Git ✓——
Schema and linterJSON Schema, Node.js linter ✓——
Evidence adaptersJUnit, promptfoo, DeepEval, Langfuse, LangSmith ✓Azure AI evaluation results as evidence (not built)Bedrock evaluation results as evidence (not built)
Report builderNode.js report tool and GitHub Action ✓——
Rendererstatic HTML on GitHub Pages ✓——

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Phases 0 to 5: the catalog itself 5 of 5 done

TaskStatusSizeSkill
Taxonomy, schema and linterdoneMSpecification design
118 obligations with permanent idsdoneLAI testing
Crosswalks to OWASP, NIST AI RMF, ISO/IEC 42001 and the EU AI ActdoneLStandards and compliance
Adapters for JUnit, promptfoo, DeepEval, Langfuse and LangSmithdoneMJavaScript
Release 0.17.1 archived on Zenodo with a DOIdoneSRelease management

Build options and guidance 1 of 3 done

TaskStatusSizeSkill
Build options for 60 of 118 obligationsdoneLAI engineering
Write build options for the 58 obligations that have nonetodoLAI engineering
Write plain-language guidance for more obligations (9 written so far)todoMTechnical writing

Patterns as portable detection rules 0 of 4 done

TaskStatusSizeSkill
Define a rule format over OpenTelemetry GenAI tracestodoMObservability
Rewrite the support agent's online rules in that formattodoMPython
Run the same rules on a second backend and compare verdictstodoLObservability
Collect a trace for every rule that must fire it and one that must nottodoMAI testing

Adoption 0 of 2 done

TaskStatusSizeSkill
Find three teams that publish a coverage report with their gapstodoLCommunity
Add an adapter for one more test tool, such as Ragas or InspecttodoSJavaScript

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Obligation author23-5 hours/weekWrites and reviews obligations and build options from real failures.
Standards mapper12-4 hours/weekKeeps the crosswalks to ISO, NIST, OWASP and the EU AI Act accurate.
Adapter developer12-4 hours/weekWrites adapters that turn another tool's results into evidence.

9 · What you'll learn

Level-up questions

10 · Links