AI Universe · Assurance & operations
Assurance & operations zoom 2
Can we prove it works, and run it? 6 concept buckets. Each one is a job you can learn once; underneath it, who does that job on each platform.
Concept
Evaluation
Proving the system does the right thing: inline, online and offline.
golden setsscenariosLLM judgesrelease gates
Physical: who does this job, where
| Open source | pydantic-evals, DeepEval, Ragas, promptfoo, Inspect |
|---|---|
| Azure | Foundry evaluation |
| AWS | Bedrock evaluations |
| GCP | Vertex Gen AI evaluation |
| Vendors | LangSmith, Braintrust |
Concept
Observability
Seeing every step of every turn, after the fact.
traces and spansmetricslogsdashboards
Physical: who does this job, where
| Open source | OpenTelemetry, Langfuse, Arize Phoenix, Prometheus, Grafana |
|---|---|
| Azure | Application Insights, Foundry tracing |
| AWS | CloudWatch, X-Ray, AgentCore Observability |
| GCP | Cloud Trace, Cloud Monitoring |
| Vendors | LangSmith, Datadog LLM Observability |
Concept
Security and red-teaming
Attacking it before someone else does.
prompt injectiondata leakagetool misuse
Physical: who does this job, where
| Open source | PyRIT, garak, promptfoo red team |
|---|---|
| Azure | AI Red Teaming Agent, Defender for AI |
| AWS | Bedrock Guardrails, GuardDuty |
| GCP | Model Armor, Security Command Center |
| Vendors | Lakera, Protect AI |
Concept
Cost and FinOps
Knowing what each answer costs, and keeping it down.
cost per turnbudgetstoken metering
Physical: who does this job, where
| Open source | your meter, Langfuse cost |
|---|---|
| Azure | Cost Management, APIM token metrics |
| AWS | Cost Explorer, Bedrock invocation logs |
| GCP | Billing reports |
| Vendors | Helicone |
Concept
Governance and regulation
Who owns it, what risk tier, and which rules apply.
risk tiersaudit traillaws and standards
Physical: who does this job, where
| Open source | AAC, AHC (open catalogs) |
|---|---|
| Azure | Purview AI hub, Azure Policy |
| AWS | SageMaker governance |
| GCP | Vertex AI governance |
| Vendors | NIST AI RMF, ISO 42001, EU AI Act, India DPDP Act |
Concept
Release and LLMOps
Changing prompts, models and policy safely, with a way back.
versioningcanary releasesrollbackfeature flags
Physical: who does this job, where
| Open source | Git, GitHub Actions, OpenFeature |
|---|---|
| Azure | App Configuration, Container Apps revisions |
| AWS | AppConfig, CodePipeline |
| GCP | Cloud Deploy |
| Vendors | LaunchDarkly |