| 6.1 Select an LLM choice (size and architecture) based on a set of quantitative evaluation metrics | Choosing a model's size and design from numbers means comparing candidates on the same test and picking by the metrics that matter for the use case. | Two models were tested on the same set. The larger scores only slightly higher on correctness and costs far more per request. | Pick the smaller model unless the quality gap crosses the threshold the business set. A bigger model is only worth it if a metric you care about fails on the smaller one. Include latency and cost next to quality. | Choosing on the highest quality score alone ignores that the difference is small and costly. Runner-up wins when: The larger model wins when the smaller one fails a required quality threshold. |
| 6.2 Select key metrics to monitor for a specific LLM deployment scenario | Monitoring keeps watching a live app. You choose the metrics that reveal the risks of that kind of app, such as answer quality, latency, errors and cost. | A RAG app answers customer questions, and the team asks what to watch after launch. | Watch groundedness (are answers supported by retrieved text), retrieval relevance, latency, error rate and token cost. These are the signature risks of a retrieval app. Safety and user feedback add to them. | Watching only uptime and request counts tells you the app runs, not that it answers well. |
| 6.3 Evaluate agent performance using MLflow scoring and tracing | A trace is a record of every step an agent took for one request. A scorer is a function or judge that gives that request a score. MLflow lets you run scorers over traces. | An agent sometimes gives wrong answers, and the team needs to see which step went wrong. | Turn on MLflow tracing, which records inputs, outputs, latency, token usage and cost for each intermediate step, then run scorers with mlflow.genai.evaluate. Read the failing trace to find the step at fault. | Scoring only final answers shows that something failed but not where. |
| 6.4 Use inference logging to assess deployed RAG application performance | Inference logging records the requests and responses of a deployed app so you can analyze real behavior afterward. | A team wants real questions and answers from a deployed RAG app to review quality. | Enable inference logging on the endpoint so requests and responses land in a Unity Catalog Delta table; rows can take up to an hour to appear. Then run quality checks over that table or the traces. | Expecting to see a request in the table seconds later wastes time debugging a delay that is normal. |
| 6.5 Use Databricks features to control LLM costs | Cost controls limit and attribute spending on models: pick the right size, cap usage, and see who spends what. | A finance team wants spend per team and a cap on runaway usage of a model endpoint. | Use budget policies to attribute serving usage to teams, rate limits through AI Gateway to cap requests, and the usage system table to see token counts. Choose pay-per-token for light use and provisioned throughput for steady, heavy use. | Only picking a cheaper model does not give attribution or a hard cap. Runner-up wins when: A smaller model wins when the question asks to lower the cost per request, not to attribute or cap. |
| 6.6 Use inference tables and Agent Monitoring to track a live LLM endpoint | Inference tables store requests and responses of an endpoint. Agent Monitoring runs scorers on live traces. Together they show what a live endpoint is doing. | Operators need both raw traffic records and quality scores for a live agent endpoint. | Use inference tables for the raw request and response records, and production monitoring for quality scores on a sampled set of traces; scores attach to each trace as feedback. Raw logs say what happened; scorers say whether it was good. | Logs alone show traffic but no quality judgement. |
| 6.7 Identify evaluation judges that require ground truth | Some judges compare an answer to a known right answer, called ground truth or expectations. Others judge on their own. You must know which need ground truth. | A team has only logged live traffic with no reference answers and asks which judges can run. | Judges that need expectations include Correctness, RetrievalSufficiency and ToolCallCorrectness. Judges such as RetrievalRelevance, safety and groundedness work without them. With only live traffic, use the second group. | Running Correctness with no expected answers returns nothing useful. Runner-up wins when: Correctness wins once the team builds a set of reference answers. |
| 6.8 Use AI Gateway (Inference Tables, Usage Tables, and rate limiting) to track an LLM or agent deployed via Agent Framework | AI Gateway sits in front of serving endpoints and adds usage tracking and limits. It records requests in inference tables, usage in a system table, and enforces rate limits. | A platform team wants per-requester token counts and rate limits for a deployed agent. | Enable AI Gateway features: inference tables for request records, usage tracking in system.serving.endpoint_usage (token counts, status codes and requester; only account admins can query it) and rate limits. The docs page layout has changed, so check the current gateway page. | Assuming any workspace user can query the usage table. |
| 6.9 Use Databricks custom Scorers for evaluating agents and LLMs | A custom scorer is a function you write when the built-in judges do not measure your rule, such as 'the reply must include a case number'. | A rule says every reply must contain a ticket number in a set format, which no built-in judge checks. | Write a custom scorer with the @scorer decorator. Its parameters are optional keywords: inputs, outputs, expectations and trace; declare only what you need. Return a value or feedback, then pass it to evaluation. | Using an LLM judge for a rule a few lines of code can check adds cost and noise. Runner-up wins when: An LLM-based judge wins when the rule is fuzzy, such as 'polite tone'. |
| 6.10 Incorporate SME feedback to improve agent performance | Subject matter experts, SMEs, are people who know the domain. Their labels on real responses become feedback that improves the agent and its test set. | Domain experts should rate real agent answers, and the ratings must feed the next evaluation. | Create a labeling session over existing traces so experts review them. Labels are stored as assessments on the traces and the originals are untouched. Use the labeled traces as expected answers in the next evaluation set. | Collecting ratings in a spreadsheet loses the link to the trace. |