Databricks GenAI Engineer Associate
Section 6 · 12%Evaluation and Monitoring

Evaluate and monitor

MLflow evaluation, scorers and judges, tracing, inference tables, AI Gateway, cost controls and SME feedback. Know which judges need ground truth and which Databricks table holds what.

PracticeS6 practice bank →

19 items, practice or timed exam mode.

Teaching cards9 cards

One-liner, how it works, must-know facts, traps and a quick check for each.

Walk the guide, line by line

This section tests how you know whether an application is good and stays good. The biggest way people lose marks is mixing up judges that need expected answers with judges that do not, and choosing by a general benchmark when the question gives its own metrics. Each row is one line of the official guide. Read across: plain words, what a question looks like, the answer, and the trap.

Guide lineIn plain wordsA question looks likeThe answerThe trap
6.1 Select an LLM choice (size and architecture) based on a set of quantitative evaluation metricsChoosing a model's size and design from numbers means comparing candidates on the same test and picking by the metrics that matter for the use case.Two models were tested on the same set. The larger scores only slightly higher on correctness and costs far more per request.Pick the smaller model unless the quality gap crosses the threshold the business set. A bigger model is only worth it if a metric you care about fails on the smaller one. Include latency and cost next to quality.Choosing on the highest quality score alone ignores that the difference is small and costly.
Runner-up wins when: The larger model wins when the smaller one fails a required quality threshold.
6.2 Select key metrics to monitor for a specific LLM deployment scenarioMonitoring keeps watching a live app. You choose the metrics that reveal the risks of that kind of app, such as answer quality, latency, errors and cost.A RAG app answers customer questions, and the team asks what to watch after launch.Watch groundedness (are answers supported by retrieved text), retrieval relevance, latency, error rate and token cost. These are the signature risks of a retrieval app. Safety and user feedback add to them.Watching only uptime and request counts tells you the app runs, not that it answers well.
6.3 Evaluate agent performance using MLflow scoring and tracingA trace is a record of every step an agent took for one request. A scorer is a function or judge that gives that request a score. MLflow lets you run scorers over traces.An agent sometimes gives wrong answers, and the team needs to see which step went wrong.Turn on MLflow tracing, which records inputs, outputs, latency, token usage and cost for each intermediate step, then run scorers with mlflow.genai.evaluate. Read the failing trace to find the step at fault.Scoring only final answers shows that something failed but not where.
6.4 Use inference logging to assess deployed RAG application performanceInference logging records the requests and responses of a deployed app so you can analyze real behavior afterward.A team wants real questions and answers from a deployed RAG app to review quality.Enable inference logging on the endpoint so requests and responses land in a Unity Catalog Delta table; rows can take up to an hour to appear. Then run quality checks over that table or the traces.Expecting to see a request in the table seconds later wastes time debugging a delay that is normal.
6.5 Use Databricks features to control LLM costsCost controls limit and attribute spending on models: pick the right size, cap usage, and see who spends what.A finance team wants spend per team and a cap on runaway usage of a model endpoint.Use budget policies to attribute serving usage to teams, rate limits through AI Gateway to cap requests, and the usage system table to see token counts. Choose pay-per-token for light use and provisioned throughput for steady, heavy use.Only picking a cheaper model does not give attribution or a hard cap.
Runner-up wins when: A smaller model wins when the question asks to lower the cost per request, not to attribute or cap.
6.6 Use inference tables and Agent Monitoring to track a live LLM endpointInference tables store requests and responses of an endpoint. Agent Monitoring runs scorers on live traces. Together they show what a live endpoint is doing.Operators need both raw traffic records and quality scores for a live agent endpoint.Use inference tables for the raw request and response records, and production monitoring for quality scores on a sampled set of traces; scores attach to each trace as feedback. Raw logs say what happened; scorers say whether it was good.Logs alone show traffic but no quality judgement.
6.7 Identify evaluation judges that require ground truthSome judges compare an answer to a known right answer, called ground truth or expectations. Others judge on their own. You must know which need ground truth.A team has only logged live traffic with no reference answers and asks which judges can run.Judges that need expectations include Correctness, RetrievalSufficiency and ToolCallCorrectness. Judges such as RetrievalRelevance, safety and groundedness work without them. With only live traffic, use the second group.Running Correctness with no expected answers returns nothing useful.
Runner-up wins when: Correctness wins once the team builds a set of reference answers.
6.8 Use AI Gateway (Inference Tables, Usage Tables, and rate limiting) to track an LLM or agent deployed via Agent FrameworkAI Gateway sits in front of serving endpoints and adds usage tracking and limits. It records requests in inference tables, usage in a system table, and enforces rate limits.A platform team wants per-requester token counts and rate limits for a deployed agent.Enable AI Gateway features: inference tables for request records, usage tracking in system.serving.endpoint_usage (token counts, status codes and requester; only account admins can query it) and rate limits. The docs page layout has changed, so check the current gateway page.Assuming any workspace user can query the usage table.
6.9 Use Databricks custom Scorers for evaluating agents and LLMsA custom scorer is a function you write when the built-in judges do not measure your rule, such as 'the reply must include a case number'.A rule says every reply must contain a ticket number in a set format, which no built-in judge checks.Write a custom scorer with the @scorer decorator. Its parameters are optional keywords: inputs, outputs, expectations and trace; declare only what you need. Return a value or feedback, then pass it to evaluation.Using an LLM judge for a rule a few lines of code can check adds cost and noise.
Runner-up wins when: An LLM-based judge wins when the rule is fuzzy, such as 'polite tone'.
6.10 Incorporate SME feedback to improve agent performanceSubject matter experts, SMEs, are people who know the domain. Their labels on real responses become feedback that improves the agent and its test set.Domain experts should rate real agent answers, and the ratings must feed the next evaluation.Create a labeling session over existing traces so experts review them. Labels are stored as assessments on the traces and the originals are untouched. Use the labeled traces as expected answers in the next evaluation set.Collecting ratings in a spreadsheet loses the link to the trace.

Mind map: Which judge can run

People assume every judge can run on any data, but some need a reference answer.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Live traffic, no reference answersJudges without expectationsRun relevance, groundedness and safety. If expected answers exist, correctness wins.
Curated set with expected answersGround-truth judgesRun Correctness and RetrievalSufficiency.
A fixed format ruleCustom code scorerWrite @scorer. If the rule is fuzzy, an LLM judge wins.
Need to know where an agent failedTracesRead the trace. Final-answer scores alone cannot place the fault.

Mind map: Spend and traffic tools

People assume one tool covers cost and quality, but each answers a different question.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Who spent whatUsage system table and budget policiesQuery it; only account admins can read the system table.
Cap requestsRate limits in AI GatewaySet limits. To reduce per-request cost, a smaller model wins.
Was the answer goodProduction monitoring scoresSample traces. For raw records, inference tables win.

AChoose models and metrics

6.1

Pick an LLM from the numbers

Run the same evaluation dataset and scorers against each candidate model, then pick the smallest, cheapest one that clears your quality bar within the latency budget.

If you know this from Azure/Claude: Same as model comparison in Azure Foundry evaluations and CCA-F's "cheapest model that meets the bar". Databricks-specific: each mlflow.genai.evaluate() call is an evaluation run in the experiment, so you compare runs side by side.

How it works

  1. Fix the evaluation dataset and scorers first, so runs are comparable
  2. Run mlflow.genai.evaluate() once per candidate (same data, same scorers)
  3. Compare quality scores, latency and token cost per run
  4. Choose the smallest model that passes every must-have threshold

Must know

  • A bigger model is only worth it if a quality metric you care about fails on the smaller one.
  • If two models tie on quality, the lower latency and cost wins.
  • A fine-tuned or smaller specialist model can beat a large general model on a narrow task; the metrics decide.
  • Each evaluation run stores per-row traces with each judge's feedback, so you can see where a model fails.
Metric patternChoose
Small model passes all quality thresholdsSmall model: lower cost and latency
Small model fails correctness by a wide marginLarger or better-suited model
Models tie on quality; one is twice as fastThe faster one
Best quality model misses the latency SLANot that one; the SLA is a hard limit

Traps

  • Choosing the model with the best single metric while it breaks the latency or cost limit in the stem.
  • Comparing runs that used different datasets or scorers.

Quick check: s6-q01 · s6-q02 · Source: docs: mlflow3/genai/eval-monitor/evaluate-app

6.2

Which metrics to watch for which deployment

Watch two families: operational metrics (latency, errors, request volume, tokens and cost) and quality metrics (scorer results such as safety, groundedness and relevance, plus user feedback).

If you know this from Azure/Claude: Same split as Azure's operational versus quality monitoring. Databricks-specific: operational data comes from inference tables and usage system tables; quality comes from scorers run on traces.

How it works

  1. Name the risk in the scenario: cost overrun, slow answers, hallucination, harmful output
  2. Map it to a metric: token usage, p95 latency, groundedness pass rate, safety pass rate
  3. Pick the source: usage tables, inference tables, or production monitoring scorers
  4. Set thresholds and review trends, not single requests

Must know

  • A RAG app's signature quality risks are groundedness (hallucination) and retrieval relevance.
  • A customer-facing chatbot needs safety tracked on all traffic.
  • Cost-sensitive internal tools: watch tokens per request and requests per user.
  • HTTP status codes and latency are recorded per request in inference tables.

Traps

  • Tracking only latency and uptime for a RAG app: it can be fast and wrong.
  • Tracking BLEU or ROUGE on live traffic: there is no reference answer in production.

Quick check: s6-q03 · s6-q04 · s6-q19 · Source: docs: mlflow3/genai/eval-monitor/production-monitoring · docs: ai-gateway/inference-tables

BEvaluate with MLflow: judges, scorers, traces

6.3

mlflow.genai.evaluate() and the built-in judges

Pass a dataset, your app as predict_fn, and a list of scorers; MLflow runs the app, traces each row and attaches each judge's feedback.

If you know this from Azure/Claude: Like Azure AI evaluators or a CCA-F LLM-as-judge harness. Databricks-specific: the judge class names, and knowing which ones need ground truth in expectations.

How it works

  1. Build a dataset: rows with inputs, plus expectations where a judge needs ground truth
  2. Choose scorers: built-in judges, Guidelines, or custom scorers
  3. mlflow.genai.evaluate(data=..., predict_fn=..., scorers=[...])
  4. Open Evaluation runs in the experiment, or mlflow.search_traces(run_id=...)

Must know

  • Need ground truth (expectations): Correctness, RetrievalSufficiency, ToolCallCorrectness.
  • No ground truth: RelevanceToQuery, RetrievalRelevance, RetrievalGroundedness, Safety, Guidelines, ToolCallEfficiency.
  • ExpectationsGuidelines needs per-row guidelines supplied in expectations, not a reference answer.
  • Judges use a Databricks-hosted LLM by default; override with model="<provider>:/<model>".
JudgeAsksGround truth?
CorrectnessDoes the answer match the expected facts?Yes
RetrievalSufficiencyDoes the retrieved context contain what's needed for the expected facts?Yes
ToolCallCorrectnessWere the right tools called with the right arguments?Yes
RetrievalGroundednessIs the answer supported by the retrieved context?No
RetrievalRelevanceIs the retrieved context relevant to the request?No
RelevanceToQueryDoes the answer address the request?No
SafetyIs the content free of harmful material?No
GuidelinesDoes the answer meet a natural-language rule?No

Traps

  • RetrievalGroundedness checks the answer against retrieved context, so it needs no expected answer. RetrievalSufficiency does need one.
  • Guidelines (one rule for all rows) versus ExpectationsGuidelines (rules per row, in expectations).

Quick check: s6-q05 · s6-q06 · s6-q07 · Source: docs: mlflow3/genai/eval-monitor/concepts/judges/

6.4

Custom scorers: @scorer and make_judge

When no built-in judge fits, write a code scorer with @scorer for deterministic checks, or an LLM judge with make_judge for judgement calls.

If you know this from Azure/Claude: CCA-F's deterministic-first rule: use code when the check is exact (format, length, a required ID), and an LLM judge only for fuzzy qualities. Databricks-specific: the @scorer signature and return types.

How it works

  1. Decide: is the check exact (code) or a judgement (LLM)?
  2. Code: @scorer function taking any of inputs, outputs, expectations, trace
  3. LLM: make_judge(name=..., instructions=..., model=...)
  4. Pass it in scorers=[...] to mlflow.genai.evaluate(), or register it for production monitoring

Must know

  • @scorer parameters are optional keywords: inputs, outputs, expectations, trace.
  • Return types: "yes"/"no", True/False, int or float, a Feedback (value and rationale), or a list of Feedback.
  • With trace, a scorer can inspect spans, for example to count tool calls.
  • Registered production scorers get inputs and outputs from the trace; expectations isn't available there.
from mlflow.genai.scorers import scorer

@scorer
def cites_ticket_id(outputs: str) -> bool:
    return "TCK-" in outputs

Traps

  • Using an LLM judge to check that the answer is valid JSON: a code scorer is cheaper and exact.
  • Class-based scorers aren't supported for production monitoring; use the decorator.

Quick check: s6-q08 · s6-q09 · s6-q18 · Source: docs: mlflow3/genai/eval-monitor/custom-scorer-reference · docs: mlflow3/genai/eval-monitor/align-judges

6.5

Traces: see every step the agent took

A trace records each step of a request (LLM calls, retrieval, tool calls) with inputs, outputs, latency and tokens, and the same traces power debugging, evaluation and monitoring.

If you know this from Azure/Claude: Like Azure AI Foundry tracing with OpenTelemetry. Databricks-specific: MLflow Tracing, with one-line autologging (for example mlflow.langchain.autolog()) and traces stored in the experiment or in Unity Catalog tables.

How it works

  1. Turn on autologging for your framework, or add manual spans where autolog doesn't reach
  2. Set the experiment with mlflow.set_experiment(...)
  3. Inspect traces in the experiment's Traces tab
  4. Feed traces to scorers, datasets and labeling sessions

Must know

  • Traces record inputs, outputs, latency, token usage and cost for each intermediate step.
  • Autologging covers 30+ frameworks with one line.
  • Traces can be stored in Unity Catalog as Delta tables, with access control and SQL access.
  • Scorers that take trace can judge the steps, not just the final answer.

Traps

  • Judging a multi-step agent only on its final answer hides a wrong tool call that happened to work.
  • Real-time tracing from a notebook in a Git folder needs a non-Git experiment set first.

Quick check: s6-q10 · Source: docs: mlflow3/genai/tracing/

CRun it live: logging, monitoring, gateway, cost, feedback

6.6

Inference tables and production monitoring

Inference tables log every request and response of an endpoint to Unity Catalog; production monitoring runs your scorers on a sample of live traces.

If you know this from Azure/Claude: Azure's continuous evaluation on live traffic, in Databricks form. Databricks-specific: agents.deploy() sets up AI Gateway inference tables and tracing automatically; production monitoring (Beta) uses .register() and .start(sampling_config=...).

How it works

  1. Deploy the agent with agents.deploy(): inference tables, tracing and the Review App come with it
  2. Query the inference table for status codes, latency and requesters
  3. Register scorers: Safety().register(name="safety"), then .start(sampling_config=ScorerSamplingConfig(sample_rate=...))
  4. Review results as feedback on traces and in the monitoring dashboards

Must know

  • Inference tables are Delta tables in Unity Catalog; rows can take up to an hour to appear.
  • They record the raw request and response, the status code and the requester.
  • Production monitoring is Beta; at most 20 scorers per experiment.
  • Suggested sample_rate: 1.0 for critical safety checks, about 0.05 to 0.2 for expensive LLM judges.

Traps

  • Monitoring scorers can't use ground truth: there are no expectations on live traffic.
  • Inference tables are not real-time alerts; data lands with a delay.

Quick check: s6-q11 · s6-q12 · s6-q17 · Source: docs: mlflow3/genai/eval-monitor/production-monitoring · docs: generative-ai/agent-framework/deploy-agent · docs: ai-gateway/inference-tables

6.7

AI Gateway: inference tables, usage tables, rate limits

AI Gateway adds governance to a serving endpoint: payload logging to inference tables, usage tracking in system tables, and rate limits per endpoint, user, group or service principal.

If you know this from Azure/Claude: Like Azure API Management in front of Azure OpenAI. Databricks-specific: the table names and the rate-limit rules.

How it works

  1. Enable inference tables to log payloads to Unity Catalog
  2. Enable usage tracking: system.serving.endpoint_usage and system.serving.served_entities
  3. Set rate limits in QPM or TPM for the endpoint, the default user, specific users or groups
  4. Query the tables to attribute usage and spot heavy users

Must know

  • system.serving.endpoint_usage: per-request token counts, status codes and requester. Only account admins can query it.
  • Rate limits: QPM or TPM; endpoint, user (default), user or service principal, group. No limits by default.
  • A user-specific limit overrides a group limit; if QPM and TPM are both set, the stricter applies; the endpoint limit is a global cap.
  • TPM limits can't be applied to custom model or agent endpoints.
  • Rate limiting is free; inference tables and usage tracking are paid features.

Traps

  • Inference tables hold payloads; usage tables hold who used how much. Know which question each answers.
  • Setting only an endpoint limit throttles everyone when one heavy user hits it.

Quick check: s6-q13 · s6-q14 · Source: docs: ai-gateway/configure-ai-gateway-endpoints · docs: ai-gateway/overview-serving-endpoints

6.8

Control LLM cost

Choose the right model and pricing mode, cap usage with rate limits, track usage with system tables, and tag serverless spend with usage policies.

If you know this from Azure/Claude: The levers you know from Azure (PTU versus pay-as-you-go, quotas), with Databricks names. Databricks-specific: Foundation Model APIs pricing modes and serverless usage (budget) policies.

How it works

  1. Right-size the model: smallest that passes evaluation
  2. Pick the pricing mode: pay-per-token to start, provisioned throughput for steady production
  3. Add TPM or QPM rate limits per user or group
  4. Track spend in usage tables; tag it with a serverless usage policy for chargeback

Must know

  • Pay-per-token is the easiest way to start; provisioned throughput is recommended for production, especially high throughput.
  • Provisioned throughput supports fine-tuned and custom models, and comes on-demand or reserved for 1 or 3 months.
  • Priority pay-per-token costs more per token, for latency-sensitive traffic.
  • Serverless usage (budget) policies tag serverless usage for attribution; they don't stop spending.
  • Batch work over a table: ai_query() instead of a loop of real-time calls.

Traps

  • Budget policies as a spending cap: they attribute cost, they don't block it.
  • Provisioned throughput for a spiky prototype with little traffic: you pay for idle capacity.

Quick check: s6-q15 · s6-q16 · Source: docs: machine-learning/foundation-model-apis/ · docs: admin/usage/budget-policies · docs: ai-gateway/configure-ai-gateway-endpoints

6.9

Bring SMEs in: Review App, labeling sessions, judge alignment

Domain experts label traces in the Review App; their labels become ground truth for evaluation datasets and are used to align LLM judges with expert judgement.

If you know this from Azure/Claude: Human-in-the-loop evaluation, which you know in principle. Databricks-specific: labeling sessions, label schemas of type expectation or feedback, and judge.align().

How it works

  1. Define label schemas: type="expectation" for ground truth, type="feedback" for ratings
  2. create_labeling_session(...), then add_traces(...) and share the Review App link
  3. Turn expectation labels into an evaluation dataset
  4. Align a judge: aligned = judge.align(traces); MemAlign is the default optimizer

Must know

  • Labels are stored as assessments on the traces in the labeling session; the original traces are untouched.
  • Reviewers need workspace access, the Databricks SQL access entitlement and CAN_EDIT on the experiment.
  • Judge alignment: at least 10 labelled traces, 50 to 100 better, with about 30% negative examples.
  • The human feedback name must match the judge's name exactly.

Traps

  • Treating raw, inconsistent expert scores as ground truth: agree a rubric first.
  • Letting an unaligned LLM judge overrule experts on domain questions.

Quick check: s6-q09 · s6-q17 · s6-q18 · Source: docs: mlflow3/genai/human-feedback/expert-feedback/label-existing-traces · docs: mlflow3/genai/eval-monitor/align-judges · docs: mlflow3/genai/human-feedback/