AI Harness Catalog

An open list of the parts that must exist around a model call for production-grade AI.

Active · join meAI Engineering

5 of 12 tasks done

1 · Purpose

The AI Harness Catalog (AHC) lists the components that must exist around a model call: context assembly, tool calls, retries, cost limits, human approval, logging and more, across sixteen layers. It is for architects and engineers building AI applications, and for teams choosing an agent framework. Teams often build logging and evaluation and call the job done, then meet outages in the layers they skipped. The catalog names every layer, says what breaks without each part, and shows what a chosen framework leaves for you to build.

For: AI architects, platform engineers, teams choosing or reviewing an agent framework.

2 · Business benefits

Fewer outages from missing partsEvery capability states the failure that follows when it is absent, so gaps are found at design time.
Know what a framework leaves outComparing a framework to the catalog shows what you still have to build before you commit.
Cheaper vendor changesPorts name the seams where the system meets outside services, so a provider can be swapped without a rewrite.
Linked to testingEach capability names the assurance obligations it helps meet, so build and test plans line up.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Draft the capability for AAC-0109 (repetition detected and broken)
  • Assess one agent framework against three layers
  • Fix unclear wording in one layer's capabilities

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Capability filesOne YAML file per capability:requirement, failure mode,PortsThe named seams to outsideservices, with the standardsSchema and linterChecks capabilities and theboundary with the assuranceProfile formatWhere an adopter records whatis met, accepted as a gap, orBlueprintsPer application type, thecapabilities it owes and howRendererBuilds the filterable catalogpage.
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Capability files → Schema and linter: validated · Ports → Capability files: seams referenced · Capability files → Blueprints: collated per application type · Profile format → Schema and linter: adopter's claims checked · Capability files → Renderer: published page

6 · Tech stack, per platform

ComponentOpen stackAzureAWS
Capability filesYAML in Git ✓——
Schema and linterJSON Schema, Node.js linter ✓——
Profile formatharness-profile.yaml, used by the support agent ✓the claims agent's profile on the Azure stack ✓not built
Rendererstatic HTML on GitHub Pages ✓——

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Core catalog 4 of 4 done

TaskStatusSizeSkill
Taxonomy of 16 layers and 8 positions, schema and linterdoneMSpecification design
118 capabilities across all layers and application typesdoneLAI architecture
Settle all 118 open design decisionsdoneMAI architecture
Release 0.3.1 archived on Zenodo with a DOIdoneSRelease management

Close the link to the assurance catalog 0 of 2 done

TaskStatusSizeSkill
Write a capability for repetition detection (AAC-0109)todoSAI architecture
Write a capability for autonomy that widens only on measured agreement (AAC-0118)todoSAI architecture

Blueprints 1 of 3 done

TaskStatusSizeSkill
Blueprints for document extraction and tool-using agentsdoneMAI architecture
Write a blueprint for a retrieval chatbottodoMAI architecture
Generate blueprints for the remaining application typestodoLTechnical writing

Realizations and coverage deltas 0 of 3 done

TaskStatusSizeSkill
List, per agent framework, which capabilities it does not providetodoLAgent frameworks
Name product options per capability in one dated passtodoLCloud and open source
Decide whether the ports become a standard of their owntodoMArchitecture

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Capability author23-5 hours/weekWrites capabilities and design decisions from systems they have built.
Framework assessor22-4 hours/weekCompares one agent framework with the catalog and writes up what it leaves out.
Reviewer11-2 hours/weekReviews pull requests for the boundary rules and plain wording.

9 · What you'll learn

Level-up questions

10 · Links