AI governance · safety · security · reliability · free guide · grows over time
Running AI you can trust
Eighteen things an organization needs so that its AI is governed, safe, secure and reliable. Each one starts with the question a leadership team asks, goes down to what to build, names the open standards and tools, and ends with how you would know it works.
Who this is for. Anyone who has to answer for AI across a whole organization, not for one system: many users, many tools, several clouds, many functions. It covers bought-in tools, staff using public assistants, classical models, language-model apps and agents.
How each capability is laid out. The question, a plain explanation, what to decide, what to build, open standards and tools (open source and cloud), a way to make it better than the usual, how to prove it works, and how much of it can be automated.
People use these terms loosely. They are five angles on the same job: AI that you can rely on and defend.
Term
The question it answers
Where it lives below
AI governance
Who decides what AI may be used, who is accountable, and how we prove it.
The whole guide; strongest in capabilities 1 to 7 and 16.
Responsible AI
Fair, explainable, private, and with a person answerable for the result.
Capabilities 6, 8, 14, 15.
AI safety
The system does not cause harm, even without an attacker, and we can stop it.
Capabilities 9, 10, 12, 13.
AI security
An attacker cannot steal, poison, trick or hijack the AI or its data.
Capabilities 2, 7, 10, 11, 12.
Reliable AI
It gives correct, consistent answers and keeps doing so as things change.
Capabilities 9, 11, 13, 17.
Do you need to learn NIST, ISO/IEC 42001 and the EU AI Act?
Yes, but not to the same depth. The three do different jobs. The EU AI Act is a law, so it says what you must have. NIST AI RMF is a free framework, so it gives you a way to organize the work. ISO/IEC 42001 is an auditable standard, so it lets you prove to others that you do the work. Learn the law in depth, NIST as your working vocabulary, and ISO/IEC 42001 at overview level.
Framework
What it is
What it is for
How deep to go
Verdict
EU AI Act (Regulation (EU) 2024/1689)
A law. It sorts AI systems into prohibited, high-risk, limited-risk and minimal-risk, and lists what providers and deployers must do for each.
Tells you what you are legally required to have if you sell to or operate in the EU.
Deep. Read the risk tiers, Articles 9 to 15 (what a high-risk system needs), Article 26 (deployer duties) and Article 50 (transparency). Skim the rest.
Pursue. It is the only one with legal force and the most concrete list of what must exist.
NIST AI Risk Management Framework 1.0 (and the generative AI profile, NIST AI 600-1)
A free, voluntary US framework with four functions: Govern, Map, Measure, Manage.
Gives you a plain working vocabulary and a way to organize the whole program.
Working knowledge. About 40 pages for the core document; read the profile if you build with language models.
Pursue. It is free, readable, and a good backbone for explaining everything else.
ISO/IEC 42001 (AI management system)
A certifiable standard in the same family as ISO/IEC 27001. It checks that policies, roles, risk processes and records exist and are kept up.
Lets an organization prove to customers and auditors that it governs AI in a repeatable way.
Overview. Learn the clause structure and the nine control areas of Annex A. The text is paid, so use the public summaries first.
Pursue the overview. Buyers ask for it, but it is about process, not about how to test a model.
OWASP LLM Top 10 and MITRE ATLAS
Lists of attacks and weaknesses specific to AI systems.
Feeds the security testing and runtime controls.
Read when you build or test language-model apps.
Pursue later, once the three above are clear.
OECD AI Principles, UNESCO ethics recommendation
High-level principles that the laws and standards above build on.
Background only.
One line each.
Drop as study items. Know they exist.
Regional rules: India's DPDP Act, GDPR, UK, Singapore, Australia, US states
Privacy law and local AI guidance.
Applies to the data and the people your system touches.
Only the ones your learners work under. A short section, not a course.
Pursue for your own region, drop the rest.
One job, three vocabularies
The three mostly ask for the same things in different words. Once you see that, you stop memorizing three lists. The table lists fifteen things a trustworthy AI program needs. The second column links to the capability on this page that covers it. The other three columns say where each framework asks for it.
Clause 9.2 (internal audit), 9.3 (management review) and the certification audit
Two notes on reading it. First, the EU AI Act puts most duties on providers (who build or sell the system) and some on deployers (who use it), so check which you are. Second, where a column names a broad area (a whole clause or Annex A group) the framework asks in general words; where it names an article, the ask is specific.
Worked example: a CV-ranking assistant
Take one system and follow it through all three. A company uses an AI assistant that ranks job applicants' CVs for its recruiters. It is bought from a vendor, and the recruiters make the final decision.
Step
What the team does
NIST AI RMF
EU AI Act
ISO/IEC 42001
1. Register it
Add it to the AI register with an owner, the vendor, the purpose and the data it sees (capability 1).
Map: write down the context and who is affected.
You must know what you deploy (Article 26).
Clause 4 and Annex A.4.
2. Sort it by risk
Employment and worker management is on the high-risk list, so treat it as the top tier (capability 3).
Map and Measure: rate the risk.
Annex III lists employment uses as high-risk, so Articles 9 to 15 apply to the provider and Article 26 to you.
Clause 6: record the risk and why.
3. Check the vendor
Ask for the vendor's documentation, test results and instructions for use; test the ranking on your own sample (capability 7).
Govern: third-party risk.
Providers owe instructions for use (Article 13); you follow them as the deployer (Article 26).
Annex A.10.
4. Assess the impact
Ask who could be wrongly rejected and how you would notice (capability 6).
Map: impacts on people.
A fundamental rights impact assessment may be required of some deployers (Article 27). Check whether you are one.
Clause 6.1.4 and Annex A.5.
5. Keep people in charge
Recruiters see why a CV was ranked, can override it, and a person signs every rejection (capability 10).
Manage: controls while it runs.
Human oversight (Article 14) and deployer oversight duties (Article 26).
Annex A.9.
6. Watch it
Keep logs, sample decisions each month and compare outcomes across groups (capability 11).
Measure and Manage: monitor.
Logs kept (Articles 12 and 26) and post-market monitoring (Article 72, provider).
Clause 9.
7. Handle problems
If it keeps rejecting a group unfairly, pause it, tell the vendor and record what you did (capability 12).
Manage: respond and learn.
Serious incidents are reported (Article 73).
Clause 10.
8. Tell people and train staff
Tell applicants that AI helps screen them and train the recruiters (capabilities 14 and 15).
Govern: transparency and skills.
Transparency (Article 50 and Article 26) and AI literacy (Article 4).
Annex A.8 and Clause 7.
Notice that the team does one piece of work at each step, and all three frameworks are satisfied by it. That is the point of the crosswalk.
What to read, and in what order
Read
Cost
Time
Note
Links
NIST AI RMF 1.0 and the NIST AI 600-1 generative AI profile
Read NIST first, then the EU risk tiers, then come back to the crosswalk above. The article and clause numbers here follow the official texts as we understand them; check each one against the source before you quote it in a compliance document.
Start with the full spread of AI use
A program built only for the AI you built yourself misses most of the exposure. Six kinds of use need covering, and they need different tools.
Kind of AI use
What it looks like
What makes it hard
Bought-in AI
AI inside products you buy: a CRM that scores leads, an HR tool that ranks CVs, a help desk that drafts replies.
You cannot test it, only question the vendor and limit its data.
Staff using public tools
People pasting work into chat assistants, writing assistants and meeting recorders on their own.
You cannot see it unless you look; policy and a safe approved option are the main levers.
Classical models
Scoring, forecasting and recommendation models that are not language models: credit, churn, demand, fraud.
Older rules (model risk management) already apply; fairness and drift are the main worries.
Language-model apps
Your own summarisers, extractors, search-and-answer tools and chat assistants.
Wrong answers, leaks and tricks through the text they read.
Agents
Systems where the model chooses tools and takes actions: refunds, bookings, record changes.
The largest risk, because a mistake changes something in the real world.
Models you train or tune
Models trained or fine-tuned on your data, including open-weight models you host.
Training-data rights, poisoning, model theft and reproducibility.
The eighteen, in six groups
The groups follow the life of an AI use: you find it, set the rules, prove it, run it, tell people, and keep cost and platforms in order.
Know what you have
You cannot govern what you cannot see. These three produce the list that everything else hangs on.
1. Inventory (the AI register)
Can you list every AI use in the organization, who owns it, and what data it touches?
A single maintained list of AI uses. One row per use, not per tool: the same chat assistant used for drafting emails and for screening applicants is two rows, because the risk is different.
Decide
What counts as an entry: any use where AI output affects a decision, a customer, or a document that leaves the building.
The unit of the register: the use (purpose + system + data + audience), not the product.
Who must register, and what happens to a use that is not registered after a set date.
How entries stay true: owners confirm each quarter, and a change in model or data triggers an update.
Build
Start with a spreadsheet or table with fixed columns: id, purpose, owner, system, model and version, vendor, data classes, users, region, autonomy level, risk tier, status, last review.
Move to a small service with an API once there are more than a few dozen entries, so pipelines can read and write it.
Feed it from places that already know: cloud billing for model spend, API gateway logs, identity provider app lists, procurement records.
Publish an AI bill of materials per system (model, datasets, libraries) so an entry can be traced to its parts.
Standards and tools
Open standards. ISO/IEC 42001 asks for a documented inventory of AI systems; EU AI Act expects providers and deployers to know their systems; CycloneDX (ML-BOM) and SPDX 3.0 (AI profile) describe an AI bill of materials.
Open source. Backstage catalog or any metadata catalog (DataHub, OpenMetadata) with a custom AI entity; MLflow Model Registry for models you train.
Cloud. AWS SageMaker Model Registry and Bedrock model inventory; Azure AI Foundry and Purview; Databricks Unity Catalog for models and endpoints; Google Vertex AI Model Registry.
Make it better
Make the register live, not a document: generate most rows from gateway logs and cloud inventories, and ask owners only to confirm. A register that fills itself is far more accurate than one people maintain by hand.
Prove it works
Coverage: of the AI calls seen in gateway and billing data, what share map to a registered use? Target it, publish it, and watch it rise.
Automation
Mostly Rows filled from gateway logs, billing and cloud inventories; changes detected. Stays with people: Owners confirm purpose and data.
What AI is in use that we have not been told about?
The register only lists what people declare. Discovery looks for the rest: tools staff signed up for, features switched on inside software you already own, and scripts calling model APIs with a personal key.
Decide
Which signals you will watch: network and proxy logs, identity provider sign-ins, expense claims, browser extensions, code repositories, cloud accounts.
What you do when you find something: an amnesty and a path to approval works better than blocking and punishing.
What you will not do: read staff content. Discovery looks at which tools and how much, not what was typed.
Build
Keep a list of known AI domains and API endpoints; match it against proxy or DNS logs weekly.
Scan source repositories and notebooks for model SDK imports and API keys.
List OAuth grants and SaaS apps in the identity provider; flag those with AI features.
Review corporate card and expense lines for AI subscriptions.
Send each finding to the owner with three choices: register it, replace it with an approved option, or stop.
Standards and tools
Open standards. OWASP AI Exchange and OWASP LLM Top 10 name the risks of unmanaged use; NIST AI RMF 'Map' function covers knowing context and uses.
Open source. Gitleaks or TruffleHog (find keys in code), Semgrep rules for SDK imports, Zeek or Suricata for DNS and network logs, osquery for browser extensions.
Cloud. Microsoft Defender for Cloud Apps and Purview (AI app discovery), AWS CloudTrail and Config, Google Cloud Asset Inventory, any secure web gateway.
Make it better
Pair discovery with an easy approved route. If getting a safe assistant takes a day, shadow use falls; if it takes three months, it grows.
Prove it works
Count of unregistered uses found per month, and the time from finding one to it being registered, replaced or stopped.
Automation
Fully Weekly scans of proxy logs, repositories, app grants and expenses. Stays with people: Someone triages each finding and offers a safe route.
3. Risk tiering
Which uses need heavy checking and which can move fast?
A short rule that sorts every AI use into a few tiers. The tier decides how much testing, review and monitoring the use must have. Without it, you either check everything (too slow) or nothing (too risky).
Decide
The questions that set the tier. A useful set: how much can the output change a person's life or money? How autonomous is the system? What data does it use? Who sees the output? Can it be reversed?
How many tiers. Three or four is enough: minimal, limited, high, and not allowed.
What each tier requires, written as a table, so a tier is an instruction and not just a label.
Who may overrule a tier, and how that is recorded.
Build
Write the tiering questionnaire as code or a form that returns a tier plus its reasons.
Store the answers, not just the result, so a tier can be re-computed when rules change.
Link each tier to a checklist of required controls from the other capabilities in this guide.
Align the top tier with the legal definitions that apply to you (for example the EU AI Act's high-risk list and banned practices).
Standards and tools
Open standards. EU AI Act (prohibited, high-risk, limited-risk and minimal-risk categories); ISO/IEC 23894 (AI risk management guidance); NIST AI RMF 'Map' and 'Measure'; Australia's Voluntary AI Safety Standard; Singapore's Model AI Governance Framework.
Open source. A decision table in a rules engine (OPA, Drools) or plain YAML plus a validator.
Cloud. No major cloud provides this for you; it is your policy, expressed as data.
Make it better
Tier by autonomy as well as impact. A tool that only suggests and a tool that acts can hold the same data yet need very different controls. Most frameworks tier on impact alone.
Prove it works
Agreement test: give ten real uses to three reviewers independently. If they land on the same tier for at least nine, your questions are clear enough.
Automation
Assisted The tier is computed from the answers and recomputed when rules change. Stays with people: A person answers the questions honestly and can overrule with a recorded reason.
Decide the rules
Who may do what, who answers for it, and what must be checked before a use is allowed.
4. Policy and acceptable use
Who may use which AI, for what, with which data?
The written rules, split by role, function and data class, and the technical settings that make the rules real. A policy nobody can enforce is a wish.
Decide
A short principle-level policy that leadership signs, with detailed standards underneath it that change faster.
A matrix: role (and function) by data class by AI route. For example, customer data may go only to the approved internal assistant.
Which uses are banned outright, which need approval, and which are pre-approved.
What a person does when the policy blocks something legitimate (an exception route with an expiry date).
Build
Express the matrix as policy-as-code so the same rule is applied by the gateway, the data platform and the identity provider.
Put one gateway in front of all model calls to apply the rules, log the calls, and strip or block sensitive data.
Publish a one-page plain-language version that staff will actually read.
Review the policy on a calendar and whenever a new class of tool appears.
Standards and tools
Open standards. ISO/IEC 42001 clause on AI policy; ISO/IEC 38507 (governance implications of AI use); NIST AI RMF 'Govern'.
Open source. Open Policy Agent (Rego) or Cedar for rules; LiteLLM, Envoy AI Gateway or Kong AI Gateway as the single gateway; Microsoft Presidio for detecting personal data.
Cloud. Amazon Bedrock Guardrails, Azure AI Content Safety, Google Cloud Model Armor, Microsoft Purview DLP.
Make it better
Treat policy as code with tests and versions, like software. Each rule has an example that must be blocked and one that must pass, run on every change.
Prove it works
Share of AI calls that pass through the gateway, and the number of policy violations blocked versus found afterwards.
Automation
Mostly Rules enforced by the gateway, data platform and identity provider; tests on every rule change. Stays with people: People write the policy and decide the exceptions.
When an AI use goes wrong, who is answerable, and who had to approve it?
Named people for each AI use and a small body that decides the hard cases. Accountability that is shared across a committee is, in practice, accountability that is nobody's.
Decide
A single accountable owner per use (the business), a technical owner (builds and runs it), and an independent reviewer for higher tiers.
A standing group with legal, security, data, risk and business members that meets on a fixed rhythm and has authority to say no.
Which decisions need which sign-off, by tier, so approvals are not negotiated each time.
Separation of duties: the builder does not approve their own high-risk system.
Build
Add owner, reviewer and approver fields to every register entry and block go-live without them.
Write a RACI (who is responsible, accountable, consulted, informed) for each stage in the AI lifecycle.
Keep a decision log: what was decided, by whom, on what evidence, with what conditions.
Run a review board cadence tied to tiers: fast lane for low risk, scheduled review for high.
Standards and tools
Open standards. ISO/IEC 42001 (roles, responsibilities and authorities); NIST AI RMF 'Govern' (accountability structures); EU AI Act (human oversight and deployer duties).
Open source. Architecture decision records (ADRs) in a repository; a ticketing workflow with required approvers.
Cloud. Approval workflows in ServiceNow, Jira or a GRC platform are the usual home; the design matters more than the tool.
Make it better
Publish the decision log (redacted). Showing how and why you said yes or no is the strongest signal of a real governance practice.
Prove it works
Share of live uses with a named owner and a recorded approval, and the median days from request to decision per tier.
Automation
Human Checks that every live use has an owner, reviewer and approval on file. Stays with people: Who is accountable, and the decisions of the review group.
6. Impact assessment
Before we launch, have we written down who could be harmed and how?
A structured document completed before a use goes live. It covers who is affected, what could go wrong for them, what protects them, and what remains. Think of it as a risk assessment for people, not for the company.
Decide
When one is required: by tier, and whenever the purpose, data or model changes materially.
Who completes it (the owner) and who challenges it (an independent reviewer).
What it must cover: purpose, affected groups, data, fairness and bias, privacy, human oversight, failure modes, and how to appeal a decision.
How it joins existing privacy and security reviews rather than becoming a third separate form.
Build
Create a template with required fields and mark which answers force a higher tier.
Link each answer to evidence, such as test results, so claims are checkable.
Store assessments next to the register entry, versioned.
Re-run it on a schedule and on any significant change.
Standards and tools
Open standards. ISO/IEC 42005 (AI system impact assessment); GDPR Article 35 data protection impact assessment; EU AI Act Article 27 fundamental rights impact assessment for certain deployers; Canada's Algorithmic Impact Assessment tool as a worked model.
Open source. Templates from the OECD and from public-sector bodies; a Markdown or form template in a repository.
Cloud. Microsoft publishes a Responsible AI Impact Assessment template; most clouds provide documentation, not a tool.
Make it better
Generate the evidence section automatically from the test run and monitoring data, and leave people to write only the judgments. It keeps assessments short and honest.
Prove it works
Share of high-tier uses with a current assessment, and the number of changes to a design that the assessment actually caused. Zero changes in a year suggests a rubber stamp.
Automation
Assisted Evidence sections filled from test and monitoring data; reminders on change. Stays with people: Judging harm to people and whether the remaining risk is acceptable.
7. Bought-in and vendor AI
What AI is inside the products we buy, and what are we accepting by using it?
You do not control a vendor's model, but you do control what data it gets, what you ask of the vendor, and whether you can switch. Third-party AI needs its own review because most of your AI exposure is probably here.
Decide
A vendor questionnaire specific to AI: models used, training on your data, subprocessors, data location, change notices, incident duties.
Contract terms: no training on your data without consent, notice of model changes, audit rights, exit and data return.
How you test what you cannot see: sample outputs, a bias check on a data sample, a reliability check on your own cases.
Concentration risk: what happens if the one model provider you depend on changes terms or goes down.
Build
Add a vendor AI section to procurement and security review, and block purchases that skip it.
Register every vendor AI feature as an entry, with the vendor as the owner of the model and you as owner of the use.
Keep a fallback route (a second provider or an open-weight model) for important uses.
Re-review at renewal, and whenever the vendor announces a model change.
Standards and tools
Open standards. ISO/IEC 27001 and SOC 2 reports for the vendor; ISO/IEC 42001 certification as a signal; NIST AI RMF 'Govern' on third-party risk; EU AI Act provider and deployer duties.
Open source. Cloud Security Alliance AI Controls Matrix and its due-diligence questionnaire; the open Shared Assessments SIG content.
Cloud. Vendor trust centres and model cards (AWS AI Service Cards, Azure Transparency Notes, Google model cards).
Make it better
Run your own small test set against each important vendor feature and keep the results. When the vendor updates the model, you will see the difference the day it happens.
Prove it works
Share of vendor AI features with a completed review, and time to detect a vendor model change.
Automation
Assisted Questionnaire routing, trust-page and certificate expiry checks, your own test set run against the vendor feature. Stays with people: Contract terms and the decision to accept the vendor.
Prove it before launch
Evidence, not promises. Data you may use, tests that pass, and a trail an auditor can follow.
8. Data governance for AI
Are we allowed to use this data, is it good enough, and can we trace it?
AI inherits every problem your data has: wrong values, missing consent, personal data in places it should not be. This is the strongest foundation you can have, because most AI harm starts in data.
Decide
Which data classes may reach which AI routes, and the consent and purpose basis for each.
Provenance for training, tuning, retrieval and evaluation data: where it came from, licence, date, who approved.
Quality rules for AI inputs (freshness, completeness, duplicates) and for what you retrieve to answer questions.
Retention, deletion and the right to be forgotten, including data already used to tune a model.
Build
Tag columns and documents with sensitivity and purpose, and have the gateway and retrieval layer respect the tags.
Capture lineage from source to prompt and to model, so any output can be traced back.
Run data quality checks before data is used for tuning or retrieval, and fail closed.
Keep dataset cards and a hash of each training set so a model can be reproduced and audited.
Standards and tools
Open standards. ISO/IEC 5259 (data quality for analytics and machine learning); ISO/IEC 27701 (privacy); GDPR; India's DPDP Act 2023; EU AI Act Article 10 (data and data governance); OpenLineage for lineage; Datasheets for Datasets as documentation practice.
Open source. OpenLineage with Marquez, DataHub or OpenMetadata; Great Expectations or Soda for quality; Presidio for personal data detection; DVC or lakeFS for dataset versions.
Cloud. Unity Catalog, Microsoft Purview, AWS Glue Data Catalog with Lake Formation, Google Dataplex.
Make it better
Make retrieval permission-aware: the assistant can only fetch passages the asking person could open themselves. Most leaks in document assistants come from skipping this.
Prove it works
Share of AI data sources with lineage and sensitivity tags, and a test that an unauthorized user cannot retrieve restricted content through the assistant.
Automation
Mostly Tagging, lineage capture, quality gates, permission-aware retrieval tests. Stays with people: Purpose and consent decisions, and what data is off limits.
What evidence do we have that this system does what we claim, and fails safely?
A catalog of tests chosen by the kind of system and its tier, with results kept as evidence. Different kinds of AI fail differently, so the tests differ: a summariser needs faithfulness checks, an agent needs checks on what it changed.
Decide
Which tests are owed for which kind of system (transform, extractor, retrieval answerer, assistant, workflow, agent, multi-agent) and for classical models (accuracy, stability, fairness).
Pass marks set before running, not after seeing results.
Separation of test data from tuning data, and a frozen test set for regression.
Judging by outcomes in the world, not only by the text of the answer.
Build
Build a regression suite of real cases, plus generated edge cases and attack cases.
Run it in CI so every change to a prompt, model or tool is tested before release.
Test agents in a simulated world with faults injected, and check what changed in the systems they touched.
Store results with the model version and dataset hashes as evidence for the register entry.
Standards and tools
Open standards. NIST AI RMF 'Measure'; NIST AI 600-1 (generative AI profile); OWASP LLM Top 10; MITRE ATLAS; ISO/IEC 42001 verification and validation controls; the AI Assurance Catalog (application type to test obligation).
Open source. promptfoo, DeepEval, Ragas, Inspect (UK AI Security Institute), lm-evaluation-harness, garak and PyRIT (attacks), Fairlearn and AIF360 (fairness), Giskard.
Cloud. Bedrock evaluations, Azure AI Foundry evaluations, Vertex AI evaluation, MLflow evaluate.
Make it better
Check behaviour under test-like and live-like conditions. Systems that do better when they seem to be watched give false comfort, and few teams measure the gap.
Prove it works
Share of owed tests that exist and pass, per tier. Report the obligations you could not test as well; the unmapped list is the honest gap.
Automation
Fully Regression, attack and fairness suites run in CI; results stored with versions. Stays with people: Choosing what to test and where the pass marks sit.
Can we show an auditor, in an afternoon, how we meet each requirement?
A cross-reference from each rule or standard to the control that meets it and the evidence that proves it. Done well, an audit becomes a query, not a project.
Decide
Which frameworks apply to you (law, sector rules, customer contracts, voluntary standards) and which you adopt as a baseline.
One set of controls mapped to many frameworks, not one set per framework.
Evidence rules: what counts, how long it is kept, who can alter it (nobody after the fact).
What you say is not yet covered. A mapping with an honest 'not covered' list is more credible than a full one.
Build
Keep a controls table: control id, description, owner, frameworks and clauses it satisfies, evidence source.
Collect evidence automatically from pipelines (test reports, approvals, access logs) into a store with write-once retention.
Generate an audit pack on demand: pick a use, get its tier, assessment, tests, approvals and monitoring summary.
Test the mapping yearly by asking an outsider to find a requirement and trace it to proof.
Standards and tools
Open standards. ISO/IEC 42001 (certifiable AI management system) and ISO/IEC 42006 (requirements for its auditors); NIST AI RMF; EU AI Act; OWASP and ATLAS crosswalks; OSCAL (NIST's machine-readable control format).
Open source. OSCAL tooling (compliance-trestle), OpenSSF Scorecard-style evidence checks, the AI Assurance Catalog crosswalk files.
Cloud. AWS Audit Manager, Azure Policy and Purview Compliance Manager, Google Assured Workloads; GRC platforms as consumers of the evidence.
Make it better
Treat evidence as code. A control is a test that runs and emits a signed result; the audit pack is a view over those results.
Prove it works
Time to assemble a complete audit pack for a named high-tier use, and the share of controls with automatically collected evidence.
Automation
Mostly Controls run as tests that emit evidence into write-once storage; audit packs generated on demand. Stays with people: The auditor's judgment, and your statement of what is not covered.
What holds while the system is live, and what happens when it goes wrong.
10. Runtime controls
While the system is live, what stops it from doing something it should not?
Checks that run on every request: what goes in, what comes out, and what actions are allowed. Testing shows it worked on your cases; runtime controls hold when the real world sends something you did not expect.
Decide
Where checks sit: on input, on retrieved content, on the model's output, and before any action with side effects.
Which checks are plain rules (cheap, predictable), which are small classifiers, and which need a judge model. Use the simplest that works.
What happens when a check fires: block, ask a person, retry safely, or degrade to a safe answer.
Least privilege for tools: each agent gets only the actions it needs, with limits on amount and rate.
Build
Put controls in a gateway or harness layer outside the model, so changing the model does not remove them.
Require approval for irreversible or high-value actions, and keep a hard kill switch per use.
Make actions idempotent so a retry cannot charge or send twice.
Log every blocked and allowed decision with the rule that fired.
Standards and tools
Open standards. OWASP LLM Top 10 (prompt injection, excessive agency, sensitive information disclosure); MITRE ATLAS; Google SAIF; EU AI Act Articles 14 and 15 (oversight, robustness and cybersecurity).
Open source. Open Policy Agent and Cedar for action rules; NeMo Guardrails, Guardrails AI, Llama Guard for content checks; Presidio for personal data.
Cloud. Amazon Bedrock Guardrails and AgentCore policy, Azure AI Content Safety and Prompt Shields, Google Cloud Model Armor; Microsoft Agent Governance Toolkit.
Make it better
Write each guardrail as a rule plus the test that proves it holds, and keep both together. Then the catalog of controls and the catalog of tests cannot drift apart.
Prove it works
Block rate on a standing set of attack and misuse cases, false-block rate on good cases, and time for the kill switch to take effect.
Automation
Fully Input, output and action checks on every request; budgets and kill switch. Stays with people: Setting limits and handling the cases that are held for a person.
How would we know, this week, that quality or behaviour has slipped?
Continuous measurement after launch. Models, data, prompts, vendors and users all change, so something that passed in March can quietly fail by June.
Decide
What to measure: quality on sampled real traffic, fairness across groups, input drift, cost, latency, refusals, human overrides.
Who looks at each signal and what threshold triggers action.
Sampling and labelling: a small, steady stream of human-reviewed cases is worth more than a big one-time test.
Privacy: monitor patterns and samples, and mask personal data in traces.
Build
Emit traces for every model call using one common schema, with model version, prompt version, tool calls and outcome.
Run the pre-launch test suite on a schedule against the live system (a canary).
Compute drift on inputs and outputs and alert on the shift, not only on errors.
Review a weekly sample of real conversations or decisions with a short rubric.
Standards and tools
Open standards. OpenTelemetry GenAI semantic conventions; NIST AI RMF 'Manage'; EU AI Act Article 72 (post-market monitoring) and Article 12 (logging); SR 11-7 ongoing monitoring for models in banking.
Open source. OpenTelemetry, Langfuse, Arize Phoenix, OpenLLMetry, Evidently (drift), Prometheus and Grafana.
Cloud. CloudWatch and Bedrock invocation logs, Azure Monitor and Application Insights, Cloud Monitoring, Databricks Lakehouse Monitoring, SageMaker Model Monitor.
Make it better
Monitor the controls themselves: track how often each guardrail fires. A guardrail that never fires, or fires on everything, is a silent failure.
Prove it works
Time from a real quality drop to an alert, measured by planting a known regression in a staging copy.
Automation
Fully Traces, drift alerts, scheduled canary tests, cost and guardrail-fire rates. Stays with people: Reviewing a weekly sample and acting on alerts.
When an AI use harms someone or leaks something, what do we do in the first hour, and who do we tell?
AI incidents are different from outages: the system may be 'up' while giving harmful answers. You need a way to recognise them, contain them, learn from them, and report them where the law requires.
Decide
What counts as an AI incident: harmful output, leak, discrimination, unauthorized action, serious near miss.
Severity levels and who is called at each, including legal, communications and the affected business.
Reporting duties: serious incidents to regulators, personal data breaches to privacy authorities, notices to customers.
Containment options prepared in advance: switch off, fall back to a person, roll back a model, restrict a tool.
Build
Extend the existing security incident process with AI-specific playbooks and severity guides.
Give staff and customers an easy way to report a bad AI output, and route it to the owner.
Capture the full trace (inputs, retrieved data, tool calls, versions) for any reported case.
After each incident, add a regression test and update the relevant control; record both in the register entry.
Standards and tools
Open standards. EU AI Act Article 73 (serious incident reporting for high-risk systems); GDPR breach notification; OECD AI incident reporting framework; NIST AI RMF 'Manage'; MITRE ATLAS case studies.
Open source. AI Incident Database (public incidents to learn from); OWASP guidance; incident tooling you already run.
Cloud. Existing cloud security and on-call tooling; the AI-specific part is the playbook and the trace.
Make it better
Run tabletop exercises on AI scenarios (a leak through the assistant, an agent issuing wrong refunds) and publish the lessons. Few organizations have rehearsed these.
Prove it works
Time to contain in a drill, share of incidents that produced a new test, and whether each reporting deadline was met.
Automation
Assisted Detection, trace capture, paging, deadline timers for reporting. Stays with people: Judging severity, containment, and what to tell regulators and customers.
13. Lifecycle: change, re-approval, retirement
When the model, prompt or data changes, is the approval still valid, and when do we turn things off?
Approval is for a particular version in a particular setting. Model providers update models, teams edit prompts, data shifts. Without lifecycle rules, an approved system slowly becomes an unapproved one.
Decide
What counts as a material change that needs re-testing or re-approval, by tier.
How model and prompt versions are pinned, and how upgrades are staged.
When a use is reviewed even if nothing changed (a yearly review is common).
How uses are retired: what happens to data, logs and users.
Build
Version prompts, tools, models and test sets together; store the bundle with the register entry.
Block promotion to production unless the owed tests pass on the exact bundle.
Use staged rollout: shadow mode, small share of traffic, then full, with rollback.
Schedule expiry dates on approvals so unused or unreviewed systems lapse by default.
Standards and tools
Open standards. ISO/IEC 5338 (AI system life cycle processes); ISO/IEC 42001 operation and change controls; EU AI Act on substantial modification; SR 11-7 on model change.
Open source. Git and CI for prompts and configs, MLflow, DVC, feature-flag tools (OpenFeature), model signing with the OpenSSF model-signing project.
Cloud. Model registries and deployment pipelines in the major clouds; provider model-deprecation notices.
Make it better
Give every approval an expiry. Approvals that lapse by default keep the register honest and force a conversation about what is still needed.
Prove it works
Share of production systems whose running version matches an approved bundle, and the count of lapsed approvals found each quarter.
Automation
Mostly Blocks promotion without passing tests on the exact bundle; approvals expire by default. Stays with people: Deciding what counts as a material change.
Be open and capable
People who use AI, or are affected by it, are told and are able.
14. Transparency and documentation
Do people know when they are dealing with AI, and can we explain what it does?
Three audiences need different documents: users need a notice, affected people need an explanation and a way to appeal, and technical reviewers need a full record of how the system works.
Decide
Where users must be told they are talking to AI or reading AI-made content.
What an explanation looks like for a decision, in plain words, and how a person asks for human review.
Technical documentation standard: model card, data card, intended use, limits, known failures, test results.
How AI-made media is labelled and its source recorded.
Build
Generate model and system cards from the register and test results, so they stay current.
Add visible notices and a human-review route to user-facing flows.
Attach provenance data to generated media where your channel supports it.
Publish a public statement of how you use AI and what you will not do, if your sector expects it.
Standards and tools
Open standards. EU AI Act Article 50 (transparency duties) and Annex IV (technical documentation); Model Cards and Datasheets for Datasets; C2PA content credentials; ISO/IEC 42001 documentation controls.
Open source. Hugging Face model card tooling, Google's Model Card Toolkit, C2PA open-source libraries, SHAP and LIME for explanations of classical models.
Cloud. AWS AI Service Cards, Azure Transparency Notes, SageMaker Model Cards, Vertex AI model cards.
Make it better
Write the explanation for the person affected first, then derive the technical record from it. It forces clarity about what the system actually does.
Prove it works
Test the notice and the explanation on real users: can they say, afterwards, that AI was involved and how to get a person?
Automation
Assisted Model and system cards generated from the register and test results. Stays with people: Writing the plain-language explanation and notices.
15. AI literacy and training
Do the people using and approving AI understand what it can and cannot do?
Rules do not work if people do not understand them. Training should differ by role: everyone needs the basics and the do-and-don't list, builders need secure and testable practice, approvers need to read evidence critically.
Decide
Role-based tracks: everyone, builders, approvers and reviewers, leaders.
Short, practical content with real examples from your own incidents and near misses.
How you check it worked: a test or a practical exercise, not attendance.
A regular refresher, because the tools change quickly.
Build
Build a one-hour baseline course with scenarios, and a half-day workshop for approvers.
Add 'how do I check the AI's answer' habits to the course: sources, second checks, when to stop and ask.
Track completion by role and link it to access to approved tools where appropriate.
Run a yearly red-team day where staff try to trick internal assistants.
Standards and tools
Open standards. EU AI Act Article 4 (AI literacy duty for providers and deployers); ISO/IEC 42001 competence controls; NIST AI RMF 'Govern'.
Open source. Open courses (for example Elements of AI, BlueDot Impact's AI safety courses), OWASP training material, Anthropic and other labs' public prompting guides.
Cloud. Cloud training catalogs for builders; internal learning platforms for the rest.
Make it better
Teach by showing your own failures. A short, honest case from your incident log changes behavior more than any slide of rules.
Prove it works
Scenario test scores by role before and after, and a fall in policy violations and shadow-tool findings.
Automation
Assisted Enrolment, completion tracking, scenario tests. Stays with people: Designing the training and running the workshops.
Cost and platform choices that touch every other capability.
17. Cost control for AI
What does each AI use cost, who pays, and what happens when usage spikes?
AI cost is variable and easy to lose track of: tokens, retries, long conversations, and idle GPUs. Cost is also a safety issue, since a runaway agent loop can burn a month's budget in an afternoon.
Decide
Cost per use, per team and per outcome (for example per resolved ticket), not just per model.
Budgets and hard limits per use, with an alert before the limit and a stop at it.
Which tasks deserve a large model and which a small one; routing rules and caching.
How the benefit is measured, so that cost is compared with value.
Build
Tag every model call with the register id so spend rolls up by use.
Set per-use token and spend limits in the gateway, plus a limit on agent steps.
Cache repeated requests, route easy ones to smaller models, and measure the quality you give up.
Report monthly: cost per use, cost per outcome, top movers.
Standards and tools
Open standards. FinOps Foundation framework and the FOCUS billing specification; OpenTelemetry for token and cost attributes.
Open source. LiteLLM and Portkey-style gateways for budgets, OpenCost, Kubecost, Langfuse cost tracking.
Cloud. AWS Cost Explorer with application inference profiles, Azure Cost Management, Google Cloud Billing, Databricks system tables.
Make it better
Report cost per correct outcome, not cost per token. A cheap model that needs three retries and human fixes can be the expensive one.
Prove it works
Share of AI spend attributed to a registered use, and forecast error month to month.
Automation
Fully Per-use cost tagging, budgets, stop limits, monthly reports. Stays with people: Deciding whether the value justifies the spend.
Can the same rules hold when AI runs on several clouds, tools and vendors?
Real organizations use more than one cloud, several model providers and many SaaS tools. If every platform has its own rules, governance fragments. The aim is one set of rules, with a thin adapter for each platform.
Decide
Which controls are universal (policy, register, evidence) and which are implemented per platform (a gateway setting, a cloud guardrail).
A 'paved road': the approved way to build and run AI, easy enough that teams choose it.
How you avoid lock-in for critical uses: ports and adapters, with a second provider tested.
Where data may be processed (region, residency) and how that is enforced.
Build
Define interfaces for model calls, retrieval, tools, logging and policy; implement one adapter per vendor.
Store the rules once as policy-as-code and compile them to each platform's native settings.
Run the same test suite against each adapter on a schedule to catch drift between platforms.
Standardize on a few open formats: OpenTelemetry for traces, OpenLineage for lineage, OSCAL for controls, CycloneDX for AI bills of materials.
Standards and tools
Open standards. OpenTelemetry, OpenLineage, OSCAL, CycloneDX ML-BOM, Model Context Protocol for tool access, ISO/IEC 42001 as the management system that spans platforms.
Open source. Open Policy Agent, Envoy AI Gateway, LiteLLM, Terraform or OpenTofu, Kubernetes, Crossplane.
Cloud. AWS, Azure, Google Cloud and Databricks each ship guardrail, registry and monitoring features. Use them behind your interfaces, not as the interface.
Make it better
Prove portability with a contract test: the same governance test suite must pass on every adapter. If switching provider is only an adapter plus a configuration file, you have control.
Prove it works
Time and effort to move one important use to a second provider with all controls intact, measured in a drill.
Automation
Mostly One rule set compiled to each platform; the same tests run on every adapter. Stays with people: Choosing platforms and accepting lock-in on purpose.
A fair question is whether this can run itself, with automatic checks that tell you whether you meet ISO and other standards at any moment. Mostly yes for the technical controls, and no for the decisions. The table below says which is which, capability by capability.
Fully A machine does it and a person only handles exceptions.
Mostly A machine does the routine work; people confirm or decide the hard parts.
Assisted A machine prepares the evidence; a person makes the call.
Human A person's decision. Machines only check that it was made.
Capability
Automation
What a machine does
What stays with people
1. Inventory (the AI register)
Mostly
Rows filled from gateway logs, billing and cloud inventories; changes detected.
Owners confirm purpose and data.
2. Discovery (finding what nobody registered)
Fully
Weekly scans of proxy logs, repositories, app grants and expenses.
Someone triages each finding and offers a safe route.
3. Risk tiering
Assisted
The tier is computed from the answers and recomputed when rules change.
A person answers the questions honestly and can overrule with a recorded reason.
4. Policy and acceptable use
Mostly
Rules enforced by the gateway, data platform and identity provider; tests on every rule change.
People write the policy and decide the exceptions.
5. Roles and accountability
Human
Checks that every live use has an owner, reviewer and approval on file.
Who is accountable, and the decisions of the review group.
6. Impact assessment
Assisted
Evidence sections filled from test and monitoring data; reminders on change.
Judging harm to people and whether the remaining risk is acceptable.
7. Bought-in and vendor AI
Assisted
Questionnaire routing, trust-page and certificate expiry checks, your own test set run against the vendor feature.
Contract terms and the decision to accept the vendor.
One rule set compiled to each platform; the same tests run on every adapter.
Choosing platforms and accepting lock-in on purpose.
Checking standards automatically
The pattern is the same one used for cloud security compliance. Write each requirement as a control. Make each control a test that runs on a schedule and emits a signed result. Store the results where nobody can alter them. Map each control to the clauses of every standard it satisfies. Then "are we aligned with ISO/IEC 42001?" becomes a query over current results, and an audit pack becomes a report. Open tools for this: Open Policy Agent for rules, OSCAL for the machine-readable control catalog, OpenTelemetry and OpenLineage for evidence, and the AI Assurance Catalog for which tests are owed.
What large organizations do
The big platform companies publish a good deal about their approach. This summary uses only what they state publicly, and does not claim to know their internal systems.
Organization
What it publicly states or sells
Evidence
Google and Google Cloud
Published AI Principles and a Secure AI Framework (SAIF) that sets security practice across the AI life cycle; a Frontier Safety Framework for its most capable models; model cards; and cloud products for the controls in this guide, such as Model Armor (screening prompts and responses), Sensitive Data Protection, and evaluation services. It has also stated ISO/IEC 42001 certification for parts of its AI offering.
Public documents and product pages
Microsoft
A Responsible AI Standard applied through internal review, Transparency Notes for its services, an AI red team, and products that bring the controls together: Purview (data and compliance), Defender (threats), and content safety. It has stated ISO/IEC 42001 certification for Microsoft 365 Copilot.
Public documents and product pages
Amazon Web Services
Bedrock Guardrails, AgentCore policy at the gateway, AI Service Cards, an Audit Manager framework for generative AI, and ISO/IEC 42001 certification announced for several AI services.
Public documents and product pages
Anthropic and other frontier labs
Published scaling policies that tie capability levels to required safeguards, with evaluations that must pass before a model is released; Anthropic has stated ISO/IEC 42001 certification.
Public documents
Large banks and insurers
Decades of model risk management under rules such as SR 11-7: a central model inventory, independent validation before use, and ongoing monitoring. The newest AI work extends this existing machinery.
Regulation and supervisory guidance
What this does and does not tell you
Certification is a snapshot. ISO/IEC 42001 certification is an audit at a point in time. What makes it continuous is evidence collected automatically between audits.
Technical controls automate well; judgment controls do not. Gateways, tests, monitoring, lineage and cost can run themselves. Accountability, harm judgment and exceptions stay with people, and the honest design makes that division explicit.
The firm-wide pieces are younger. Runtime guardrails and evaluation are common in products. A live register that fills itself, tiering that recomputes, and audit packs generated from evidence are newer, and mostly show up in vendor offerings and early adopters, not as settled practice.
Public claims are not audits. What a company publishes shows its intent and its products. It does not show how complete its internal coverage is.
Tools that sell the automation
Kind
Examples and what to check
Compliance automation for security standards
Vanta, Drata, Secureframe and similar tools collect evidence continuously for ISO/IEC 27001 and SOC 2, and have added ISO/IEC 42001 frameworks.
AI governance platforms
Credo AI, IBM watsonx.governance, ServiceNow AI Control Tower, OneTrust and others offer a register, assessments and policy workflows. Check how much is automated evidence and how much is forms.
Cloud-native policy and audit
AWS Config and Audit Manager, Azure Policy and Purview Compliance Manager, Google Security Command Center. Strong for infrastructure; AI-specific controls are still being added.
Open building blocks
Open Policy Agent and Cedar for rules, OSCAL for machine-readable controls and compliance-trestle to work with it, OpenTelemetry and OpenLineage for evidence, plus the AI Assurance Catalog for the test obligations.
The order to build in
Do not build all eighteen at once. A small, honest version of the first group beats a polished version of the last.
Standards and laws change. ISO/IEC 42001 and 42005, the NIST AI Risk Management Framework and its generative AI profile, and the OWASP lists are current at the time of writing. The EU AI Act applies in phases, and its timetable has been under review, so confirm the dates that apply to you. Tool names are examples, not endorsements; check current documentation before you build. If you find something wrong or missing, tell us and we will correct it.