AI Harness Catalog
An open list of the parts that must exist around a model call for production-grade AI.
1 · Purpose
The AI Harness Catalog (AHC) lists the components that must exist around a model call: context assembly, tool calls, retries, cost limits, human approval, logging and more, across sixteen layers. It is for architects and engineers building AI applications, and for teams choosing an agent framework. Teams often build logging and evaluation and call the job done, then meet outages in the layers they skipped. The catalog names every layer, says what breaks without each part, and shows what a chosen framework leaves for you to build.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Draft the capability for AAC-0109 (repetition detected and broken)
- Assess one agent framework against three layers
- Fix unclear wording in one layer's capabilities
4 · Requirements
What it must do
- Lists 118 capabilities with permanent ids across 16 harness layers and 10 kinds of application.
- States for each capability what it must do, what breaks without it, and the design decisions it forces.
- Defines 18 ports, the seams where a harness meets a service it does not own.
- Provides a profile format in which a team declares what it built, what it accepts as a gap and what does not apply.
- Links each capability to the AI Assurance Catalog obligations it helps meet.
- Ships blueprints for two kinds of application: document extraction pipelines and tool-using agents.
How well it must do it
- It describes components and never ships them as a library.
- A linter rejects verification words in a requirement, so build and test statements stay in separate catalogs.
- It depends on the assurance catalog in one direction only.
- Named products come last, because they change faster than the requirements.
- Specification text is CC BY 4.0 and tooling is Apache 2.0; releases carry a DOI.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS |
|---|---|---|---|
| Capability files | YAML in Git ✓ | — | — |
| Schema and linter | JSON Schema, Node.js linter ✓ | — | — |
| Profile format | harness-profile.yaml, used by the support agent ✓ | the claims agent's profile on the Azure stack ✓ | not built |
| Renderer | static HTML on GitHub Pages ✓ | — | — |
7 · What needs to be done
Core catalog 4 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Taxonomy of 16 layers and 8 positions, schema and linter | done | M | Specification design |
| 118 capabilities across all layers and application types | done | L | AI architecture |
| Settle all 118 open design decisions | done | M | AI architecture |
| Release 0.3.1 archived on Zenodo with a DOI | done | S | Release management |
Close the link to the assurance catalog 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Write a capability for repetition detection (AAC-0109) | todo | S | AI architecture |
| Write a capability for autonomy that widens only on measured agreement (AAC-0118) | todo | S | AI architecture |
Blueprints 1 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Blueprints for document extraction and tool-using agents | done | M | AI architecture |
| Write a blueprint for a retrieval chatbot | todo | M | AI architecture |
| Generate blueprints for the remaining application types | todo | L | Technical writing |
Realizations and coverage deltas 0 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| List, per agent framework, which capabilities it does not provide | todo | L | Agent frameworks |
| Name product options per capability in one dated pass | todo | L | Cloud and open source |
| Decide whether the ports become a standard of their own | todo | M | Architecture |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Capability author | 2 | 3-5 hours/week | Writes capabilities and design decisions from systems they have built. |
| Framework assessor | 2 | 2-4 hours/week | Compares one agent framework with the catalog and writes up what it leaves out. |
| Reviewer | 1 | 1-2 hours/week | Reviews pull requests for the boundary rules and plain wording. |
9 · What you'll learn
- What an agent harness is, layer by layer, beyond the model call.
- Why ports and adapters make AI systems cheaper to change.
- How to tell a build requirement from a test obligation.
- How to assess an agent framework by what it does not give you.
- How a profile declares accepted gaps honestly.
Level-up questions
- Which of the 16 layers does your current agent framework hide from you?
- When should a model choke point be in-process, at a gateway, or both?
- Why are named products the last thing added to the catalog?
- What is the difference between a gap you accept and a capability that does not apply?