| 3.1 Select Langchain/similar tools for use in a Generative AI application | LangChain is a library that connects the parts of a language-model application: models, prompts, retrievers and tools. 'Similar tools' means other frameworks that do the same job, and the question is which one fits the application. | A team wants to wire a Databricks chat model and an embedding model into a retrieval chain and keep it easy to swap parts. | Use LangChain with the Databricks wrappers: ChatDatabricks for the chat model and DatabricksEmbeddings for embeddings. A framework earns its place when you have several parts to connect. For a single model call, the plain client is simpler and has fewer dependencies. | Adding a framework to a one-prompt, one-model call adds a dependency and no benefit. Runner-up wins when: A plain API call wins when the application is one prompt to one model with no retrieval. |
| 3.2 Qualitatively assess responses to identify common issues such as quality and safety | Qualitative assessment means reading real responses and judging them for quality problems (wrong, vague, off-topic) and safety problems (harmful, leaking private data). | A reviewer reads twenty responses and finds some confident answers that are not supported by the retrieved text. | That is a groundedness problem, also called hallucination: the answer goes beyond the evidence. Check by comparing each claim to the retrieved context in the trace. Fix the cause that the trace shows, whether that is missing context, a loose prompt, or no instruction to say 'not found'. | Swapping to a larger model first skips diagnosis; if the right context was never retrieved, a bigger model still has nothing to ground on. |
| 3.3 Select chunking strategy based on model & retrieval evaluation | The chunking choice should be made from measurements, not guesses: you try strategies, run retrieval and answer evaluation, and keep the one that scores best. | Two chunk sizes were tried on the same question set. One scores higher on retrieval relevance but lower on answer correctness. | Decide against the metric tied to the application goal. Relevance measures the retriever, correctness measures the end result, so a gain in one with a loss in the other means chunks are found but lack context to answer. Try a middle size or return fuller text around the matched chunk, then re-measure. | Picking the winner on the one metric that looks better ignores that users only see the final answer. |
| 3.4 Augment a prompt with additional context from a user's input based on key fields, terms, and intents | Augmenting a prompt means adding context taken from the user's request, such as key fields, terms or detected intent, before the model sees it. | A user types an order number and a short question. The prompt needs the order's status and the right policy section. | Extract the key field (the order id) and the intent (a returns question), use them to fetch the order record and the matching policy, and insert both into the prompt template. This keeps the prompt grounded in facts you looked up instead of asking the model to guess. | Pasting the entire policy manual into every prompt costs tokens and buries the relevant section. Runner-up wins when: Passing the raw question straight to retrieval wins when there is no structured field or intent to extract. |
| 3.5 Create a prompt that adjusts an LLM's response from a baseline to a desired output | A baseline response is what the model gives with a simple prompt. You change the prompt step by step to move it toward the response you want, such as a tone, length or format. | Replies are accurate but too long and too casual for a bank's customer messages. | Add specific instructions: the audience, tone, a length limit, and one or two short examples of the desired reply. Examples show the target more reliably than adjectives. Change one thing at a time and compare against the baseline so you know which change helped. | Fine-tuning for a tone problem is slow and costly when a prompt change would do. Runner-up wins when: Fine-tuning wins when the style must hold across many cases and prompt changes can no longer keep it consistent. |
| 3.6 Implement LLM guardrails to prevent negative outcomes | Guardrails are checks that stop bad outcomes, such as unsafe output or leaked personal data, before they reach the user. | A public assistant must block harmful replies and hide personal data, and the team wants it applied to the endpoint without changing app code. | Use guardrails configured on the serving endpoint, which cover safety filtering and PII detection; a blocked request returns a default message. The documented guardrails page is the older AI Gateway page and the feature is Public Preview, so check the current docs. Application-level checks wrap the call when you need custom rules. | Relying on the system prompt to refuse unsafe requests is a request, not a control, and can be bypassed. Runner-up wins when: In-app checks win when the rule is specific to your business and the endpoint filters cannot express it. |
| 3.7 Select the best LLM based on the attributes of the application to be developed | Pick the language model from the attributes of the application: task difficulty, speed, cost, context length and whether it is allowed to run where the data is. | A classification job handles millions of short messages a day and quality on the small model is already acceptable. | Choose the smaller instruct model, because a larger model is only worth paying for when a metric you care about fails on the smaller one. High volume and narrow tasks favour small and fast. | Choosing the largest model 'to be safe' multiplies cost and latency with no gain the stem shows. Runner-up wins when: The larger model wins when the smaller one fails a quality metric, or the task needs long, multi-step reasoning. |
| 3.8 Select an embedding model context length based on source documents, expected queries, and optimization strategy | An embedding model turns text into numbers, and it can only read a fixed number of tokens at once. Its context length must cover the chunks you will embed and the queries users will send. | Chunks are about 3,000 tokens long. One candidate embedding model has a 512-token window and another has 8,192. | Pick the model whose window is larger than the chunk, here the 8,192-token one such as gte-large-en (1024 dimensions). A model with a smaller window cuts the text off and ignores the tail. If you can shorten chunks to fit the small model, that is a valid optimization for speed and cost. | Choosing the small-window model for speed silently drops most of each chunk. Runner-up wins when: The 512-token model wins if chunks can be cut to fit and the stem puts a strong limit on cost or latency. |
| 3.9 Select a model from a model hub or marketplace for a task based on model metadata/model cards | A model hub or marketplace lists many models. The model card, the page of metadata, tells you the task, size, licence, languages and limits so you can judge fit before use. | A team must choose a model to summarize German contracts for commercial use. | Read the model card for supported languages, task, context length and licence terms. Databricks-provided models appear in Unity Catalog under system.ai, and are available to account users by default. A model whose card does not list German, or whose licence forbids commercial use, fails the requirement regardless of benchmark rank. | Picking the model with the best general leaderboard score ignores language and licence. |
| 3.10 Select the best model for a given task based on common metrics generated in experiments | Experiments produce numbers such as accuracy, latency and cost per request. You choose the model by the metrics that match the task, tracked in MLflow. | Three models were run on the same question set and logged. One is the most accurate and also the slowest, and the use case is a live chat with a strict response time. | Weigh the metrics against the stated requirement: accuracy only counts if latency stays within the limit. Pick the most accurate model that meets the time limit. MLflow tracking keeps the runs side by side so the choice can be shown. | Picking the top accuracy score alone ignores the latency requirement that was in the stem. Runner-up wins when: The most accurate model wins in a batch job with no time limit. |
| 3.11 Utilize MLflow and Agent Framework for developing agentic systems | MLflow tracks, traces and evaluates the app. Agent Framework is the Databricks toolkit for authoring and deploying agents; Databricks recommends the MLflow ResponsesAgent interface for writing them. | A team wants an agent they can trace step by step, evaluate, and deploy to a serving endpoint. | Author the agent with ResponsesAgent, turn on MLflow tracing so every step records inputs, outputs, latency and token use, and evaluate with MLflow scorers. Then log, register and deploy it with Agent Framework. | Hand-rolling a class with its own request format loses the built-in tracing, review and deployment support. |
| 3.12 Compare the evaluation and monitoring phases of the Gen AI application life cycle | Evaluation happens before release, on a fixed question set. Monitoring happens after release, on live traffic. Both use the same scorers, but they answer different questions. | A team asks what to run before launch and what to run afterward. | Before launch, run evaluation against a curated dataset, often with expected answers, to decide whether to ship. After launch, production monitoring scores a configurable sample of live traces and attaches results to each trace as feedback, to catch drift. Judges that need expected answers can only run in evaluation, where you have them. | Treating the two as the same leaves live traffic unchecked. |
| 3.13 Enable multi-agent systems to leverage Genie Spaces or conversational API to retrieve data | Genie is the Databricks natural-language interface over governed tables, and a multi-agent system can call it for data answers, either through a supervisor or through the Genie conversation API. The guide calls it Genie Spaces; current docs call it Genie Agents. | A supervisor agent must answer sales-figure questions from governed tables without writing its own SQL. | Give the supervisor a Genie space as a sub-agent, or call it with the Genie conversation API. Genie checks data access with the caller's Unity Catalog permissions, so answers only include data that user may see. | Having the model write raw SQL itself risks wrong queries and bypasses curated business definitions. |