AI Universe · Assurance & operations

Assurance & operations zoom 2

Can we prove it works, and run it? 6 concept buckets. Each one is a job you can learn once; underneath it, who does that job on each platform.

Concept

Evaluation

Proving the system does the right thing: inline, online and offline.

golden setsscenariosLLM judgesrelease gates

Physical: who does this job, where

Open sourcepydantic-evals, DeepEval, Ragas, promptfoo, Inspect
AzureFoundry evaluation
AWSBedrock evaluations
GCPVertex Gen AI evaluation
VendorsLangSmith, Braintrust

Concept

Observability

Seeing every step of every turn, after the fact.

traces and spansmetricslogsdashboards

Physical: who does this job, where

Open sourceOpenTelemetry, Langfuse, Arize Phoenix, Prometheus, Grafana
AzureApplication Insights, Foundry tracing
AWSCloudWatch, X-Ray, AgentCore Observability
GCPCloud Trace, Cloud Monitoring
VendorsLangSmith, Datadog LLM Observability

Concept

Security and red-teaming

Attacking it before someone else does.

prompt injectiondata leakagetool misuse

Physical: who does this job, where

Open sourcePyRIT, garak, promptfoo red team
AzureAI Red Teaming Agent, Defender for AI
AWSBedrock Guardrails, GuardDuty
GCPModel Armor, Security Command Center
VendorsLakera, Protect AI

Concept

Cost and FinOps

Knowing what each answer costs, and keeping it down.

cost per turnbudgetstoken metering

Physical: who does this job, where

Open sourceyour meter, Langfuse cost
AzureCost Management, APIM token metrics
AWSCost Explorer, Bedrock invocation logs
GCPBilling reports
VendorsHelicone

Concept

Governance and regulation

Who owns it, what risk tier, and which rules apply.

risk tiersaudit traillaws and standards

Physical: who does this job, where

Open sourceAAC, AHC (open catalogs)
AzurePurview AI hub, Azure Policy
AWSSageMaker governance
GCPVertex AI governance
VendorsNIST AI RMF, ISO 42001, EU AI Act, India DPDP Act

Concept

Release and LLMOps

Changing prompts, models and policy safely, with a way back.

versioningcanary releasesrollbackfeature flags

Physical: who does this job, where

Open sourceGit, GitHub Actions, OpenFeature
AzureApp Configuration, Container Apps revisions
AWSAppConfig, CodePipeline
GCPCloud Deploy
VendorsLaunchDarkly

Zoom in further

← Experience

Compute & infrastructureDataModelModel accessThe AI application (agent harness)ExperienceAssurance & operations