Databricks GenAI Engineer Associate
Section 3 · 30%Application Development

Application development

The biggest section: prompts, guardrails, choosing LLMs and embedding models, MLflow and the Agent Framework, and Genie in multi-agent systems. Concepts carry over from Azure and Claude; spend your time on the Databricks names.

PracticeS3 practice bank →

20 items, practice or timed exam mode.

Teaching cards10 cards

One-liner, how it works, must-know facts, traps and a quick check for each.

Walk the guide, line by line

This is the biggest section, and it tests choices: which framework, which model, which prompt change, which guardrail, and how to judge a response. The most common way to lose marks is choosing by habit (the biggest model, the newest tool) when the question gives a measured attribute or a stated constraint that decides. Each row is one line of the official guide. Read across: plain words, what a question looks like, the answer, and the trap.

Guide lineIn plain wordsA question looks likeThe answerThe trap
3.1 Select Langchain/similar tools for use in a Generative AI applicationLangChain is a library that connects the parts of a language-model application: models, prompts, retrievers and tools. 'Similar tools' means other frameworks that do the same job, and the question is which one fits the application.A team wants to wire a Databricks chat model and an embedding model into a retrieval chain and keep it easy to swap parts.Use LangChain with the Databricks wrappers: ChatDatabricks for the chat model and DatabricksEmbeddings for embeddings. A framework earns its place when you have several parts to connect. For a single model call, the plain client is simpler and has fewer dependencies.Adding a framework to a one-prompt, one-model call adds a dependency and no benefit.
Runner-up wins when: A plain API call wins when the application is one prompt to one model with no retrieval.
3.2 Qualitatively assess responses to identify common issues such as quality and safetyQualitative assessment means reading real responses and judging them for quality problems (wrong, vague, off-topic) and safety problems (harmful, leaking private data).A reviewer reads twenty responses and finds some confident answers that are not supported by the retrieved text.That is a groundedness problem, also called hallucination: the answer goes beyond the evidence. Check by comparing each claim to the retrieved context in the trace. Fix the cause that the trace shows, whether that is missing context, a loose prompt, or no instruction to say 'not found'.Swapping to a larger model first skips diagnosis; if the right context was never retrieved, a bigger model still has nothing to ground on.
3.3 Select chunking strategy based on model & retrieval evaluationThe chunking choice should be made from measurements, not guesses: you try strategies, run retrieval and answer evaluation, and keep the one that scores best.Two chunk sizes were tried on the same question set. One scores higher on retrieval relevance but lower on answer correctness.Decide against the metric tied to the application goal. Relevance measures the retriever, correctness measures the end result, so a gain in one with a loss in the other means chunks are found but lack context to answer. Try a middle size or return fuller text around the matched chunk, then re-measure.Picking the winner on the one metric that looks better ignores that users only see the final answer.
3.4 Augment a prompt with additional context from a user's input based on key fields, terms, and intentsAugmenting a prompt means adding context taken from the user's request, such as key fields, terms or detected intent, before the model sees it.A user types an order number and a short question. The prompt needs the order's status and the right policy section.Extract the key field (the order id) and the intent (a returns question), use them to fetch the order record and the matching policy, and insert both into the prompt template. This keeps the prompt grounded in facts you looked up instead of asking the model to guess.Pasting the entire policy manual into every prompt costs tokens and buries the relevant section.
Runner-up wins when: Passing the raw question straight to retrieval wins when there is no structured field or intent to extract.
3.5 Create a prompt that adjusts an LLM's response from a baseline to a desired outputA baseline response is what the model gives with a simple prompt. You change the prompt step by step to move it toward the response you want, such as a tone, length or format.Replies are accurate but too long and too casual for a bank's customer messages.Add specific instructions: the audience, tone, a length limit, and one or two short examples of the desired reply. Examples show the target more reliably than adjectives. Change one thing at a time and compare against the baseline so you know which change helped.Fine-tuning for a tone problem is slow and costly when a prompt change would do.
Runner-up wins when: Fine-tuning wins when the style must hold across many cases and prompt changes can no longer keep it consistent.
3.6 Implement LLM guardrails to prevent negative outcomesGuardrails are checks that stop bad outcomes, such as unsafe output or leaked personal data, before they reach the user.A public assistant must block harmful replies and hide personal data, and the team wants it applied to the endpoint without changing app code.Use guardrails configured on the serving endpoint, which cover safety filtering and PII detection; a blocked request returns a default message. The documented guardrails page is the older AI Gateway page and the feature is Public Preview, so check the current docs. Application-level checks wrap the call when you need custom rules.Relying on the system prompt to refuse unsafe requests is a request, not a control, and can be bypassed.
Runner-up wins when: In-app checks win when the rule is specific to your business and the endpoint filters cannot express it.
3.7 Select the best LLM based on the attributes of the application to be developedPick the language model from the attributes of the application: task difficulty, speed, cost, context length and whether it is allowed to run where the data is.A classification job handles millions of short messages a day and quality on the small model is already acceptable.Choose the smaller instruct model, because a larger model is only worth paying for when a metric you care about fails on the smaller one. High volume and narrow tasks favour small and fast.Choosing the largest model 'to be safe' multiplies cost and latency with no gain the stem shows.
Runner-up wins when: The larger model wins when the smaller one fails a quality metric, or the task needs long, multi-step reasoning.
3.8 Select an embedding model context length based on source documents, expected queries, and optimization strategyAn embedding model turns text into numbers, and it can only read a fixed number of tokens at once. Its context length must cover the chunks you will embed and the queries users will send.Chunks are about 3,000 tokens long. One candidate embedding model has a 512-token window and another has 8,192.Pick the model whose window is larger than the chunk, here the 8,192-token one such as gte-large-en (1024 dimensions). A model with a smaller window cuts the text off and ignores the tail. If you can shorten chunks to fit the small model, that is a valid optimization for speed and cost.Choosing the small-window model for speed silently drops most of each chunk.
Runner-up wins when: The 512-token model wins if chunks can be cut to fit and the stem puts a strong limit on cost or latency.
3.9 Select a model from a model hub or marketplace for a task based on model metadata/model cardsA model hub or marketplace lists many models. The model card, the page of metadata, tells you the task, size, licence, languages and limits so you can judge fit before use.A team must choose a model to summarize German contracts for commercial use.Read the model card for supported languages, task, context length and licence terms. Databricks-provided models appear in Unity Catalog under system.ai, and are available to account users by default. A model whose card does not list German, or whose licence forbids commercial use, fails the requirement regardless of benchmark rank.Picking the model with the best general leaderboard score ignores language and licence.
3.10 Select the best model for a given task based on common metrics generated in experimentsExperiments produce numbers such as accuracy, latency and cost per request. You choose the model by the metrics that match the task, tracked in MLflow.Three models were run on the same question set and logged. One is the most accurate and also the slowest, and the use case is a live chat with a strict response time.Weigh the metrics against the stated requirement: accuracy only counts if latency stays within the limit. Pick the most accurate model that meets the time limit. MLflow tracking keeps the runs side by side so the choice can be shown.Picking the top accuracy score alone ignores the latency requirement that was in the stem.
Runner-up wins when: The most accurate model wins in a batch job with no time limit.
3.11 Utilize MLflow and Agent Framework for developing agentic systemsMLflow tracks, traces and evaluates the app. Agent Framework is the Databricks toolkit for authoring and deploying agents; Databricks recommends the MLflow ResponsesAgent interface for writing them.A team wants an agent they can trace step by step, evaluate, and deploy to a serving endpoint.Author the agent with ResponsesAgent, turn on MLflow tracing so every step records inputs, outputs, latency and token use, and evaluate with MLflow scorers. Then log, register and deploy it with Agent Framework.Hand-rolling a class with its own request format loses the built-in tracing, review and deployment support.
3.12 Compare the evaluation and monitoring phases of the Gen AI application life cycleEvaluation happens before release, on a fixed question set. Monitoring happens after release, on live traffic. Both use the same scorers, but they answer different questions.A team asks what to run before launch and what to run afterward.Before launch, run evaluation against a curated dataset, often with expected answers, to decide whether to ship. After launch, production monitoring scores a configurable sample of live traces and attaches results to each trace as feedback, to catch drift. Judges that need expected answers can only run in evaluation, where you have them.Treating the two as the same leaves live traffic unchecked.
3.13 Enable multi-agent systems to leverage Genie Spaces or conversational API to retrieve dataGenie is the Databricks natural-language interface over governed tables, and a multi-agent system can call it for data answers, either through a supervisor or through the Genie conversation API. The guide calls it Genie Spaces; current docs call it Genie Agents.A supervisor agent must answer sales-figure questions from governed tables without writing its own SQL.Give the supervisor a Genie space as a sub-agent, or call it with the Genie conversation API. Genie checks data access with the caller's Unity Catalog permissions, so answers only include data that user may see.Having the model write raw SQL itself risks wrong queries and bypasses curated business definitions.

Mind map: Choosing a model or a fix

People assume the larger or more advanced option is the safer pick, but the stated metric or limit in the question decides.

What you observeWhat it points toWhat to do (and when the runner-up wins)
High volume, narrow task, small model already passesSmaller instruct modelUse it. If a quality metric fails on it, the larger model wins.
Chunks longer than the embedding windowText is cut off silentlyPick the larger-window model. If chunks can be shortened and cost is tight, the small model wins.
Most accurate model is too slow for live chatLatency is a hard limitPick the best model within the limit. In an offline batch job with no limit, the most accurate wins.
Right style missing, content correctPrompt problemAdd instructions and examples. If it must hold across many cases and prompts fail, fine-tuning wins.
Confident answers not in retrieved textGroundedness gapFix retrieval or the prompt. Only if retrieval is sound and a metric fails does a bigger model win.

Mind map: Before launch versus after launch

People assume evaluation and monitoring are one activity, but they differ in data and in what you can know.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Fixed question set with expected answersEvaluationRun correctness judges that need ground truth. With no expected answers, use relevance and safety judges.
Live traffic, no expected answersMonitoringScore a sample of traces. If something looks wrong, add it to the evaluation set.
Safety or PII concern on every requestGuardrail on the pathBlock on the endpoint. If the rule is business-specific, an app-level check wins.

AFrameworks and models

3.1

LangChain on Databricks: which class does what

The databricks-langchain package wraps Databricks services as ordinary LangChain parts, so a chain or LangGraph agent can use them directly.

If you know this from Azure/Claude: The chain-and-tools idea is the same as tool use in Claude and agents in Foundry. What's Databricks-specific is the class names and that every part points at a Databricks endpoint, index or Unity Catalog object.

How it works

  1. Pick the LLM: ChatDatabricks(endpoint=...) points at a Model Serving or Foundation Model API endpoint
  2. Add retrieval: DatabricksVectorSearch(index_name=...), then .as_retriever()
  3. Add tools: UCFunctionToolkit turns Unity Catalog functions into agent tools
  4. Add structured data: GenieAgent sends a question to a Genie space

Must know

  • ChatDatabricks is the chat model; DatabricksEmbeddings is the embedding model wrapper.
  • DatabricksVectorSearch works with Delta Sync and Direct Vector Access indexes.
  • UCFunctionToolkit exposes Unity Catalog functions as tools, so governance stays in Unity Catalog.
  • DatabricksMCPServer and DatabricksMultiServerMCPClient connect an agent to Databricks MCP endpoints.
  • Docs now call Vector Search AI Search; the exam guide still says Vector Search.
You needClass
Chat LLM on a serving endpointChatDatabricks
EmbeddingsDatabricksEmbeddings
Retriever over an indexDatabricksVectorSearch(...).as_retriever()
Unity Catalog functions as toolsUCFunctionToolkit
Ask a Genie spaceGenieAgent

Traps

  • DatabricksEmbeddings makes vectors; it doesn't search an index. The retriever is DatabricksVectorSearch.
  • A Unity Catalog function becomes a tool through UCFunctionToolkit, not by pasting its SQL into the prompt.

Quick check: s3-q01 · s3-q02 · Source: docs: large-language-models/langchain · docs: generative-ai/agent-framework/langchain-uc-integration

3.2

Choose the LLM by what the application needs

Start from the application's attributes (task, input size, latency, cost, licence, quality bar), then pick the smallest model that meets them.

If you know this from Azure/Claude: Same reasoning as choosing a model in Foundry or picking Haiku vs Opus. Here the candidates are Foundation Model API endpoints and models in system.ai.

How it works

  1. Name the task: summarise, classify, extract, generate, reason with tools
  2. Size the input: longest prompt plus retrieved context must fit the context window
  3. Set the limits: latency target, cost per request, licence terms
  4. Test two or three candidates on your own evaluation set and compare

Must know

  • High-volume, narrow tasks (classification, extraction) usually suit a smaller instruct model.
  • Foundation Model API chat models differ in context window, for example 128K tokens for Llama 3.3 70B Instruct and GPT OSS 120B.
  • Instruct (chat-tuned) models are for serving; base models are for fine-tuning.
  • A bigger model costs more per token and adds latency; it's only worth it if your evaluation shows the quality gain.

Traps

  • "Most capable model" is rarely right when the stem stresses cost or throughput.
  • An embedding model can't answer questions; it only makes vectors.

Quick check: s3-q03 · s3-q07 · Source: docs: machine-learning/foundation-model-apis/supported-models

3.3

Embedding model context length

The embedding model's window must cover your largest chunk and typical query; anything past the window is cut off before embedding.

If you know this from Azure/Claude: Same trade-off as choosing an Azure OpenAI embedding model: window, dimensions, cost. On Databricks the two pay-per-token embedding endpoints have very different windows.

How it works

  1. Find your largest chunk in tokens (and your longest expected query)
  2. Pick a model whose window covers it; text past the window is ignored
  3. If cost and latency matter more than quality, prefer the smallest window and dimension that still fits
  4. Embed documents and queries with the same model

Must know

  • gte-large-en: 1024 dimensions, 8,192-token window, embeddings not normalised.
  • bge-large-en: 1024 dimensions, 512-token window, normalised embeddings.
  • Qwen3-Embedding-0.6B (Public Preview): up to about 32K tokens, dimensions configurable up to 1024.
  • A bigger window doesn't help if every chunk is already short; it just costs more.
ModelWindowDimensionsNormalised
gte-large-en8,1921024No
bge-large-en5121024Yes
Qwen3-Embedding-0.6B (preview)~32Kup to 1024—

Traps

  • Choosing bge-large-en for 2,000-token chunks silently truncates them to 512 tokens.
  • Mixing models (documents with one, queries with another) puts vectors in different spaces.

Quick check: s3-q04 · s3-q05 · Source: docs: machine-learning/foundation-model-apis/supported-models

3.4

Pick from a model hub, then by experiment metrics

Shortlist with the model card (task, languages, licence, size), then decide with numbers from your own MLflow experiments.

If you know this from Azure/Claude: Same as the Foundry model catalog and its model cards. On Databricks, ready models sit in Unity Catalog under system.ai, and comparisons live in MLflow runs.

How it works

  1. Browse system.ai in Catalog Explorer (or Marketplace) and read each model's card
  2. Drop models whose licence, languages or task don't fit
  3. Evaluate the survivors on the same dataset with mlflow.genai.evaluate()
  4. Pick the model that meets every hard limit, then the best quality among those

Must know

  • Databricks-provided models live in Unity Catalog under system.ai and are available to all account users by default.
  • Databricks recommends instruct versions for deployment and base versions for fine-tuning.
  • Serve a system.ai model from its model page with Serve this model.
  • Marketplace models arrive as read-only catalogs in Unity Catalog.

Traps

  • A leaderboard score or download count doesn't override a licence that forbids your use.
  • Pick the best score only among models that already pass the hard limits (latency, cost).

Quick check: s3-q07 · s3-q08 · Source: docs: generative-ai/pretrained-models · docs: mlflow3/genai/eval-monitor/

BPrompts, quality and guardrails

3.5

Augment and adjust prompts

Pull key fields and the intent out of the user's message, fetch the matching facts, and put them in the prompt; then shape the output with instructions, examples or a schema.

If you know this from Azure/Claude: Straight from Claude and Azure prompt work: context in, clear instructions, few-shot examples, structured outputs. Databricks adds response_format on Foundation Model API chat endpoints.

How it works

  1. Classify intent and extract key fields (order ID, product, date)
  2. Look up the facts for those fields and add them to the prompt as labelled context
  3. Compare the baseline output with the desired one; name the gap (format, tone, length, content)
  4. Close the gap: instructions, a few examples of the exact output, or response_format with a JSON schema

Must know

  • response_format with type: json_schema (and strict: true) or json_object requests structured output.
  • Supported on Foundation Model API pay-per-token and provisioned throughput chat endpoints.
  • For Claude models on Databricks: json_schema only, no streaming, and not combined with tools.
  • Few-shot examples fix format and tone faster than longer instructions.

Traps

  • Lowering temperature makes output steadier, but it doesn't teach a format.
  • Fine-tuning to add fresh per-user facts is the wrong tool; fetch them at request time.

Quick check: s3-q09 · s3-q10 · Source: docs: machine-learning/model-serving/structured-outputs

3.6

Guardrails on serving endpoints

AI Gateway guardrails screen requests and responses on a serving endpoint: a safety filter for harmful content and PII detection that blocks or masks.

If you know this from Azure/Claude: Like Azure content filters and Claude's own safety, but set on the endpoint. New for you: the exact settings and limits.

How it works

  1. Open the endpoint's AI Gateway settings
  2. Turn on Safety for input, output or both
  3. Set PII detection to Block, Mask or None
  4. Test with streaming off if you rely on output guardrails

Must know

  • Guardrails on serving endpoints are Public Preview (legacy AI Gateway page).
  • PII detection: Block rejects the request or response; Mask replaces the sensitive values and lets it through.
  • Output guardrails are not supported for embedding models or for streaming.
  • The newer Unity Gateway governs model and MCP services with service policies (built-in, custom, external), checked on input and output.

Traps

  • Inference tables and rate limits are AI Gateway features too, but they record or throttle; they don't filter content.
  • "Mask" and "Block" aren't the same: Block stops the answer, Mask lets it through with values hidden.

Quick check: s3-q11 · s3-q12 · Source: docs: ai-gateway/configure-ai-gateway-endpoints · docs: ai-gateway/

3.7

Read responses for quality and safety; pick chunking from evaluation

Read real responses and sort what's wrong into quality issues (wrong, ungrounded, incomplete, off-format) and safety issues (harmful, leaks data); then let evaluation numbers, not taste, choose the chunk size.

If you know this from Azure/Claude: Same failure taxonomy as the Azure evaluators (groundedness, relevance, safety) and CCA-F review loops. On Databricks the scores come from MLflow judges such as RetrievalGroundedness and Safety.

How it works

  1. Sample real questions and answers; read them with the retrieved context beside them
  2. Tag each problem: ungrounded, irrelevant, incomplete, unsafe, leaks data, off-format
  3. For retrieval problems, run the same evaluation over two or three chunking set-ups
  4. Keep the set-up with the best retrieval and answer scores at acceptable cost

Must know

  • An answer that sounds right but isn't supported by the retrieved text is a groundedness (hallucination) problem.
  • Sharing another person's data or giving dangerous instructions is a safety issue, however polished the answer.
  • RetrievalSufficiency asks whether the retrieved context holds what's needed; it needs expected facts.
  • If answers span sections, small chunks split them; evaluate larger or structure-aware chunks.

Traps

  • Long or wordy answers are a quality issue, not a safety issue.
  • Choosing chunk size by gut feel; the objective says choose it from evaluation.

Quick check: s3-q13 · s3-q14 · s3-q15 · Source: docs: mlflow3/genai/eval-monitor/concepts/judges/ · docs: generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag

CAgents, evaluation and Genie

3.8

Build agents with MLflow and the Agent Framework

Write the agent in any framework, wrap it in MLflow ResponsesAgent, log it from code, and trace every step.

If you know this from Azure/Claude: Like building a Foundry agent or a Claude Agent SDK loop, but packaged as an MLflow model. New for you: ResponsesAgent and models-from-code.

How it works

  1. Write the agent (LangGraph, OpenAI Agents SDK, plain Python) in agent.py
  2. Wrap it in ResponsesAgent; end the file with mlflow.models.set_model(AGENT)
  3. Log it with mlflow.pyfunc.log_model(python_model="agent.py", ...), register to Unity Catalog, deploy
  4. Turn on tracing (autolog) to see each LLM and tool call

Must know

  • Databricks recommends MLflow ResponsesAgent for authoring agents.
  • ResponsesAgent lets MLflow infer a signature that works with Databricks product features.
  • Models-from-code logs the script path, not a pickled object; it suits GenAI agents with no trained weights.
  • Docs now recommend the Agent Bricks CLI for new agents; the ResponsesAgent path is labelled legacy but still documented.
# agent.py
import mlflow
from mlflow.pyfunc import ResponsesAgent

class SupportAgent(ResponsesAgent):
    def predict(self, request): ...

AGENT = SupportAgent()
mlflow.models.set_model(AGENT)

# driver notebook
mlflow.pyfunc.log_model(name="agent", python_model="agent.py")

Traps

  • Pickling a LangChain or LangGraph object is fragile; log from code instead.
  • A custom PythonModel with a home-made input format loses the automatic agent signature.

Quick check: s3-q16 · s3-q17 · Source: docs: generative-ai/agent-framework/author-agent · mlflow: ml/model/models-from-code/

3.9

Evaluation phase vs monitoring phase

Evaluation scores a fixed dataset before release; monitoring runs the same scorers on a sample of live traces after release.

If you know this from Azure/Claude: Same split as Azure's offline evaluation vs continuous evaluation. On Databricks both use MLflow scorers: mlflow.genai.evaluate() before, registered scorers with .start() after.

How it works

  1. Before release: build an evaluation dataset (with expected answers where you can)
  2. Run mlflow.genai.evaluate() and compare app versions
  3. After release: scorer.register(name=...), then .start(sampling_config=ScorerSamplingConfig(sample_rate=...))
  4. Read assessments on traces and monitoring dashboards; feed bad cases back into the dataset

Must know

  • Production monitoring scores a configurable sample of incoming traces; results attach to each trace as feedback.
  • Suggested sample rates: 0.05–0.2 for expensive judges, 1.0 for critical safety checks.
  • At most 20 active scorers per experiment; allow 15–20 minutes for first results.
  • Judges that need ground truth (Correctness, RetrievalSufficiency, ToolCallCorrectness) suit evaluation datasets with expectations, not unlabelled live traffic.

Traps

  • Re-running evaluate() nightly on a fixed dataset isn't monitoring live traffic.
  • Putting Correctness on live traffic fails: there's no expected answer to compare with.

Quick check: s3-q18 · s3-q19 · Source: docs: mlflow3/genai/eval-monitor/ · docs: mlflow3/genai/eval-monitor/production-monitoring · docs: mlflow3/genai/eval-monitor/concepts/judges/

3.10

Genie in multi-agent systems

Let a Genie space answer questions over structured tables, as one worker agent beside retrieval and tool agents, called through GenieAgent or the Genie Conversation API.

If you know this from Azure/Claude: New for you: Genie is Databricks' natural-language-to-SQL over Unity Catalog tables. The supervisor/worker pattern itself is the same as Claude subagents or Foundry connected agents.

How it works

  1. Create a Genie space over the right tables and a SQL warehouse
  2. Add it as a worker: GenieAgent(genie_space_id=...) or a supervisor's Genie subagent
  3. Use on-behalf-of-user authorization so queries run with each user's permissions
  4. From custom code, call the Conversation API and poll until COMPLETED

Must know

  • Genie checks data access with the caller's Unity Catalog permissions, so answers include only data they can see.
  • The Conversation API returns generated SQL, query results and visualisations; poll every 1–5 seconds until COMPLETED, FAILED or CANCELLED.
  • The API doesn't retry failed requests; add backoff. Start a new conversation per session.
  • Auth: OAuth U2M when a user is present; a service principal (M2M) for headless use.

Traps

  • Embedding table rows into a vector index to answer numeric questions; that's Genie's job.
  • A shared service principal for a user-facing agent returns data some users mustn't see.

Quick check: s3-q06 · s3-q20 · Source: docs: genie/conversation-api · docs: generative-ai/agent-framework/multi-agent-genie · docs: generative-ai/agent-framework/authenticate-on-behalf-of-user