Databricks GenAI Engineer Associate
Section 2 · 14%Data Preparation

Data preparation

Chunking, cleaning and extracting documents, writing chunks to Delta tables in Unity Catalog, and measuring and improving retrieval. The RAG front end you know, with Databricks' tables and functions.

PracticeS2 practice bank →

17 items, practice or timed exam mode.

Teaching cards8 cards

One-liner, how it works, must-know facts, traps and a quick check for each.

Walk the guide, line by line

This section tests everything that happens before a question is asked: how documents are cleaned, cut into chunks, stored, and later found again. The biggest way people lose marks is blaming the language model for a bad answer whose real cause is a bad chunk, a stale document, or missing metadata. Each row is one line of the official guide. Read across: plain words, what a question looks like, the answer, and the trap.

Guide lineIn plain wordsA question looks likeThe answerThe trap
2.1 Apply a chunking strategy for a given document structure and model constraintsA chunk is a piece of a document stored and searched as one unit. Chunk size and cut points should follow how the document is laid out and the limits of the embedding model that turns text into numbers.Product manuals have clear headings, and answers keep missing the sentence that follows a heading because chunks cut in the middle of sections.Use a format-aware strategy that cuts on headings, with a size under the embedding model's input limit and a small overlap. Smaller chunks are precise but lose surrounding context; larger ones carry more context but add noise and cost. Fixed-size cutting is fast but rarely production quality for structured documents.Making every chunk much larger looks like a fix but dilutes the match and can exceed the embedding model's window.
Runner-up wins when: Semantic chunking, which cuts where the topic shifts, wins when the text has no headings or structure to follow.
2.2 Filter extraneous content in source documents that degrades quality of a RAG applicationSource files carry text that is not content: page headers, footers, navigation, legal boilerplate. If it stays in the index, it ends up inside chunks and gets retrieved.Every chunk from a set of PDFs ends with the same copyright footer, and unrelated questions retrieve unrelated chunks.Strip headers, footers and special characters during preprocessing, before chunking. Repeated boilerplate makes different chunks look alike to the embedding model, which hurts ranking. Clean at ingestion because fixing it after retrieval costs money on every request.Telling the model in the prompt to ignore the footer does not help, because the junk chunk has already taken a retrieval slot.
2.3 Choose the appropriate Python package to extract document content from provided source data and formatEach file type needs a tool that can read it. You pick the Python package or Databricks function that matches the source format, such as PDF, Word, slides or scanned images.A folder of mixed PDF and Word files in a Unity Catalog volume must be turned into text, with tables and layout kept.Use ai_parse_document on the file bytes read with read_files(..., format => 'binaryFile'). It accepts PDF, JPG, PNG, TIFF, DOC/DOCX and PPT/PPTX and keeps structure. For purely local work, a format-specific Python package such as a PDF reader or a DOCX reader fits when the format is known.Reading every file as plain text only works for text formats. A PDF opened that way gives garbled bytes.
Runner-up wins when: A Python library wins when the job runs outside SQL, for example in a notebook step that handles one known format.
2.4 Define operations and sequence to write given chunked text into Delta Lake tables in Unity CatalogChunked text must be written to a Delta table in Unity Catalog in a sensible order: read, parse, clean, chunk, assign ids, write. The table is what the search index later reads from.A team has chunks in memory and wants a governed table that a search index can follow as the content changes.Create the table in a Unity Catalog catalog and schema with a stable primary key per chunk, the chunk text, and source metadata, then write the chunks to it. Enable Change Data Feed on the table, because a Delta Sync index follows changes through it; row tracking turns it on automatically for the standard endpoint case. Order the steps so cleaning happens before chunking and ids are set before the write.Writing chunks to files in cloud storage skips Unity Catalog governance and gives the index nothing to sync from.
2.5 Identify needed source documents that provide necessary knowledge and quality for a given RAG applicationA retrieval application can only answer from what it contains. You decide which documents must be in the index so the knowledge is present and trustworthy.A parts-selling assistant answers billing questions well but fails on delivery dates. The customer already supplies account and order ids.Find the missing knowledge: delivery data is per-order and changing, so it belongs in a lookup the app queries by order id, not in documents or the prompt. Static policy text goes in the vector index. Store source, version and date metadata with each chunk so stale and current versions can be told apart.Adding more policy documents to the index looks like more knowledge but cannot supply a per-order date that exists in no document.
Runner-up wins when: A document index wins when the missing knowledge is stable policy text and not a record tied to one order.
2.6 Use tools and metrics to evaluate retrieval performanceRetrieval quality is measured separately from answer quality. Judges check whether the retrieved context was relevant and whether it held enough facts to answer.Users report wrong answers, and the team wants to know whether the retriever or the model is at fault, without having reference answers yet.Run RetrievalRelevance first, which checks if the retrieved chunks relate to the request and needs no ground truth. RetrievalSufficiency asks whether the chunks hold every fact needed and does need a reference answer. Traces in MLflow let you see what was retrieved for each request.Scoring only the final answer cannot tell you which half failed.
Runner-up wins when: RetrievalSufficiency wins once the team has expected answers and the question is about missing facts.
2.7 Design retrieval systems using advanced chunking strategiesAdvanced chunking keeps more context around each piece. For example, the text that is searched can differ from the text that is shown or embedded, so the match is precise but the answer has enough surrounding text.Short chunks match questions well but the model then lacks the surrounding explanation to answer fully.Embed a context-enriched version of the chunk while returning the fuller text for the prompt. The ai_prep_search function produces chunk_to_embed and chunk_to_retrieve for this purpose, with chunk_id, metadata and pages alongside. Hybrid search, which blends vector and keyword matching, helps with exact identifiers.Simply increasing chunk size trades one problem for another by adding noise.
2.8 Explain the role of re-ranking in the information retrieval processRe-ranking takes the first batch of search results and reorders them with a stronger scoring step, so the best few reach the model.The right passage is usually somewhere in the top twenty results but rarely in the top three the prompt can hold.Add re-ranking: it rescores the candidates against the query and puts the best first. It adds latency, typically under a second per query, which is the cost you accept for better ordering. Hybrid search merges vector and keyword results with Reciprocal Rank Fusion, a separate step that widens the candidate set.Raising the number of chunks sent to the model fills the prompt with weaker passages and raises cost.
Runner-up wins when: Skipping re-ranking wins when the stem puts a strict latency limit that the extra step would break.

Mind map: Bad answer: where is the fault

People assume a wrong answer means a weak model, but most wrong answers come from what was retrieved.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Unrelated chunks keep appearing, same footer text in eachBoilerplate left in the chunksClean headers and footers before chunking. If the footer is the only text on a page, dropping the page wins.
Right passage found but answer is thinChunk too small, context lostEmbed an enriched chunk and return fuller text. If chunks are already large and noisy, shrinking wins.
Right passage is in the results but ranked lowOrdering problemAdd re-ranking. If latency is capped tightly, skip it.
Old and new versions of a policy both returnedMissing version metadataStore version and date and filter on them. If old versions should be deleted, removing them wins.
Per-order facts missing from answersKnowledge is a record, not a documentAdd a lookup by order id. If the missing facts are stable text, add them to the index.

AGet the right content in

2.1

Choose the source documents

Retrieval can only return what you indexed: add the documents that hold the missing knowledge, and remove stale or duplicate ones.

If you know this from Azure/Claude: Same lesson as grounding data in Azure AI Search or a Claude RAG bot. The exam frames it as diagnosing which questions fail and which source would answer them.

How it works

  1. Group failing questions by topic
  2. Ask which document would contain the answer; check whether it's indexed
  3. Add authoritative, current sources; drop superseded versions
  4. Structured per-record facts (a delivery date for one order) belong in a table or tool, not in a vector index

Must know

  • Stale and current versions of the same document compete at retrieval time and cause wrong answers.
  • Store document-level metadata (source, version, date) with each chunk so you can filter.
  • Prompting can't create knowledge that no source contains.

Traps

  • Fine-tuning to add missing facts is the wrong fix for a knowledge gap.
  • Indexing everything 'just in case' adds noise and conflicting versions.

Quick check: s2-q07 · Source: docs: generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag

2.2

Extract content: the right package for the format

Match the extractor to the file: PDF text, mixed office files, HTML, or scanned images each have a natural tool, and Databricks adds ai_parse_document in SQL.

If you know this from Azure/Claude: Like choosing between Azure Document Intelligence read and layout models. New for you: ai_parse_document, a SQL AI Function that parses layout, text, tables and figures.

How it works

  1. Digital PDF text: a PDF library such as pypdf
  2. Mixed PDF, Word and PowerPoint with one library: unstructured
  3. HTML pages: BeautifulSoup or lxml
  4. Scanned images: OCR (Tesseract via pytesseract); in SQL, ai_parse_document covers PDFs, images, DOCX and PPTX

Must know

  • ai_parse_document inputs: PDF, JPG/JPEG, PNG, TIFF, DOC/DOCX, PPT/PPTX as binary content.
  • Read files with read_files(..., format => 'binaryFile') and pass the content column.
  • Output is a VARIANT with pages and elements (text, table, figure, title, section header, header/footer...); tables come back as HTML in version 2.0.
  • Limits: 500 pages and 100 MB per document; needs Databricks Runtime 17.3 or above in supported regions.
SourceNatural choice
Digital PDFspypdf
Mixed PDF, DOCX, PPTXunstructured
HTML pagesBeautifulSoup or lxml
Scanned JPG or PNGpytesseract (OCR)
Any of these, from SQL, into Deltaai_parse_document

Traps

  • BeautifulSoup can't read scanned images; OCR is needed.
  • ai_extract returns the fields you ask for, not the whole document's content; use ai_parse_document for full text and tables.

Quick check: s2-q04 · s2-q05 · Source: docs: generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag · docs: sql/language-manual/functions/ai_parse_document

2.3

Filter extraneous content

Strip text that repeats on every page or carries no meaning before chunking, or it floods retrieval with useless matches.

If you know this from Azure/Claude: The same clean-up you'd do before indexing in Azure AI Search. Databricks-specific help: ai_parse_document labels page headers and footers as their own element types, so you can drop them.

How it works

  1. Find repeated text: page headers, footers, legal disclaimers, navigation menus, cookie banners
  2. Remove it after extraction and before chunking
  3. Keep meaningful structure: section titles, tables, figure captions
  4. Re-check retrieval on the questions that failed

Must know

  • Databricks guidance: preprocess extracted text to remove headers, footers and special characters.
  • Repeated boilerplate in every chunk makes unrelated chunks look similar to each other.
  • ai_parse_document element types include header and footer, which makes filtering straightforward.

Traps

  • A reranker can't fix an index full of boilerplate; clean the source first.
  • Don't strip section titles: they are the context that makes chunks retrievable.

Quick check: s2-q03 · s2-q16 · Source: docs: generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag · docs: sql/language-manual/functions/ai_parse_document

BChunk and store

2.4

Chunking for document structure and model limits

Cut along the document's own structure, and keep each chunk inside the embedding model's token window.

If you know this from Azure/Claude: Identical trade-offs to chunking in Azure or a Claude pipeline. Databricks-specific: the embedding endpoints' windows (BGE 512, GTE 8,192 tokens).

How it works

  1. Use structure first: headings in Markdown or HTML, then paragraphs
  2. Cap chunk length at the embedding model's window; longer text is truncated
  3. Add modest overlap so facts at boundaries aren't split
  4. Tune size against retrieval results, not by instinct

Must know

  • Strategies: fixed-size (fast, rarely production-grade), paragraph, format-specific (headers), semantic (topic shifts).
  • Smaller chunks: precise but lose context. Larger: more context, more noise and cost.
  • Overlap preserves continuity across chunk boundaries but adds chunks.
  • databricks-bge-large-en: 512-token window, 1,024 dimensions. databricks-gte-large-en: 8,192 tokens, 1,024 dimensions.
Embedding endpointWindowDimensions
databricks-bge-large-en512 tokens1,024
databricks-gte-large-en8,192 tokens1,024
databricks-qwen3-embedding-0-6b (preview)about 32K tokensup to 1,024

Traps

  • A chunk longer than the embedding window isn't an error; the tail is silently truncated and never searchable.
  • More overlap means more vectors, not fewer.

Quick check: s2-q01 · s2-q02 · s2-q13 · Source: docs: generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag · docs: machine-learning/foundation-model-apis/supported-models

2.5

Advanced chunking for better retrieval

Search with small, precise pieces but give the model enough context: parent-child chunks, context-enriched chunks and metadata.

If you know this from Azure/Claude: The small-to-big and contextual retrieval ideas you know from Claude and Azure. New for you: ai_prep_search (Beta) splits parsed documents into semantic chunks and adds document context to each.

How it works

  1. Parent-child: embed small chunks, return their larger parent section to the LLM
  2. Context-enriched: prepend document title or section path to the text you embed
  3. Store metadata (product, version, date) per chunk to filter at query time
  4. Measure the change with retrieval metrics

Must know

  • ai_prep_search output per chunk: chunk_id, chunk_to_retrieve (raw text), chunk_to_embed (context-enriched), metadata, pages.
  • Flatten to one row per chunk in Delta, then use chunk_to_embed as the embedding column and chunk_id as the primary key.
  • ai_prep_search is Beta and needs Databricks Runtime 18.2 or above.
  • Metadata enables filters, which narrow the search before ranking.

Traps

  • Embedding the enriched text but showing the model only the tiny chunk loses the benefit; return the parent or retrieve text.
  • Semantic chunking isn't free: it costs compute at ingest.

Quick check: s2-q10 · s2-q14 · Source: docs: generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag · docs: sql/language-manual/functions/ai_prep_search

2.6

Write chunks to Delta tables in Unity Catalog

Read, parse, chunk, write one row per chunk with a primary key to a UC Delta table with Change Data Feed on, then point a Delta Sync index at it.

If you know this from Azure/Claude: New for you in its details: Azure indexers pull from blob storage; on Databricks the index syncs from a governed Delta table.

How it works

  1. Read files from a UC volume as binary
  2. Parse to text (ai_parse_document or a Python library) and clean it
  3. Chunk; add chunk_id, source path and metadata columns
  4. Write to catalog.schema.table with delta.enableChangeDataFeed = true; create a Delta Sync index

Must know

  • Standard vector search endpoints need Change Data Feed on the source table; row tracking turns it on automatically.
  • A Delta Sync index keeps itself in step with the source table.
  • Creating an index needs CREATE TABLE on the target schema; the workspace needs Unity Catalog and serverless compute.
  • Use a three-level name: catalog, schema, table.

Traps

  • Writing chunks to a path, not a UC table, means no governance and no Delta Sync.
  • Forgetting a stable primary key per chunk breaks incremental sync.

Quick check: s2-q06 · s2-q17 · Source: docs: ai-search/ai-search · docs: sql/language-manual/functions/ai_prep_search

CRetrieve, measure and re-rank

2.7

Evaluate retrieval with tools and metrics

Measure retrieval separately from generation: are the retrieved chunks relevant, and do they contain everything needed?

If you know this from Azure/Claude: Like Azure's retrieval and groundedness evaluators. Databricks-specific: MLflow judges RetrievalRelevance, RetrievalSufficiency and RetrievalGroundedness.

How it works

  1. Build an evaluation set of questions (with expected facts where you can)
  2. Score retrieved context with RetrievalRelevance (no ground truth needed)
  3. With expected facts, score RetrievalSufficiency
  4. Track ranking metrics such as precision at k and recall at k as you change chunking

Must know

  • RetrievalRelevance: is retrieved context relevant to the request? No ground truth.
  • RetrievalSufficiency: does the context hold all facts needed for the expected answer? Needs ground truth.
  • RetrievalGroundedness: is the response supported by the retrieved context? No ground truth.
  • Precision at k: share of the top k that are relevant. Recall at k: share of all relevant chunks that appear in the top k.

Traps

  • Groundedness checks the answer against context, not whether retrieval found the right context.
  • Sufficiency needs expected facts; without labels, use relevance.

Quick check: s2-q08 · s2-q09 · Source: docs: mlflow3/genai/eval-monitor/concepts/judges/

2.8

Re-ranking and hybrid search

A fast first stage fetches many candidates; a re-ranker scores them more precisely so the best few go to the LLM.

If you know this from Azure/Claude: The same idea as Azure AI Search's semantic ranker. On Databricks: pass DatabricksReranker(columns_to_rerank=[...]) to similarity_search.

How it works

  1. Retrieve broadly (for example 50 candidates) with ANN or hybrid
  2. Re-rank with the reranker over chosen text and metadata columns
  3. Send only the top few to the LLM
  4. Use query_type="hybrid" when exact terms such as codes matter

Must know

  • Reranking adds latency, typically under 1 second per query.
  • Hybrid search combines vector and keyword results with Reciprocal Rank Fusion; hybrid returns at most 200 results by default.
  • Default query type is ANN; FULL_TEXT is Beta.
  • Filters: a dictionary on standard endpoints, a SQL-like string on storage-optimized endpoints.
  • Vector Search is now named AI Search in the docs (formerly Databricks Vector Search); the exam guide uses the old name.

Traps

  • Passing 50 chunks straight to the LLM instead of re-ranking raises cost and buries the answer.
  • When latency is the top constraint, a reranker may be the wrong addition.

Quick check: s2-q11 · s2-q15 · s2-q18 · Source: docs: vector-search/query-vector-search