Home › Key concepts
Key concepts · free
The ideas that are the same everywhere
84 ideas behind data, AI, LLM, ML, software and forward-deployed engineering. Clouds and tools change; these don't. Each is one short page: what it is, why it matters, an example and the common mistake.
Data Engineering
Key concepts in Data Engineering
Then follow the path: Data Engineering · Crawl, Walk, Run
Moving data
ETL vs ELTTwo orders for the same three steps: extract, transform, load.CrawlFull vs incremental loadsReload everything every time, or only what is new since last time.CrawlBatch vs streamingProcess data in scheduled chunks, or event by event as it happens.WalkChange data capture (CDC)Copy only what changed in a database, by reading its change log.WalkIdempotencyRunning the same thing twice gives the same result as running it once.Crawl
Storing data
Warehouse, lake and lakehouseThree ways to store data for analysis, and why most teams now use the third.CrawlFile formats: CSV, JSON, Avro, ParquetHow data is laid out in a file decides how fast and cheap it is to read.CrawlOpen table formats and ACIDDelta Lake, Apache Iceberg and Apache Hudi make files in storage behave like database tables.WalkPartitioning and clusteringArrange data so a query can skip what it does not need.WalkThe small-files problemThousands of tiny files make every query slow.WalkSeparating storage and computeKeep data in one place and rent the engines that read it only when needed.Walk
Shaping data
Medallion architecture (bronze, silver, gold)Clean data in three layers so each step is simple and you can always go back.CrawlDimensional modelling (star schema)Facts in the middle, descriptions around them: tables built for questions.WalkSlowly changing dimensionsHow to record that a customer moved city, without rewriting history.WalkNormalised vs denormalisedStore each fact once, or repeat it to make reading faster.WalkSemantic layer and metricsDefine each business number once, so dashboards and AI give the same answer.Run
Trusting data
Data quality dimensionsSix ways data goes wrong, and a check for each.CrawlData contractsA written agreement between the team that produces data and the teams that use it.WalkSchema evolutionHow a table's columns can change over time without breaking readers.WalkLineageA map of where each table and number came from, and what depends on it.WalkGovernance, access and personal dataWho owns each dataset, who may see it, and how personal data is protected.Walk
Running data
Orchestration and DAGsRun each step in the right order, on time, and retry what fails.CrawlBackfills and reprocessingRe-run a pipeline for past dates, safely.WalkFreshness and SLAsPromise how up to date data will be, then measure it.WalkWhere data costs come fromScanning, shuffling, skew and idle clusters: the four usual suspects.RunVersion control and pull requestsEvery change saved, explained and reviewed before it goes live.CrawlInfrastructure as codeThe whole environment described in files you can review and re-create.Walk
AI Engineering
Key concepts in AI Engineering
Then follow the path: AI Engineering · Crawl, Walk, Run
Working with models
Deterministic firstIf ordinary code can do it, code does it. The model handles only the part that needs judgement.CrawlTokens and the context windowModels read and write in tokens, and can only see so many at once.CrawlStructured outputAsk the model for JSON in a fixed shape, then check it in code.CrawlHallucination and groundingConfident answers that are not true, and how to anchor answers to sources.CrawlTemperature and samplingHow the model picks the next token from its options.Crawl
Tools and agents
Tool callingThe model asks your code to do something; your code decides whether it runs.CrawlThe agent loopModel, tool, result, model again, until done or a limit is hit.WalkThe harnessEverything around the model that makes it safe to use.WalkState and durable waitsRemember where a long task was, survive crashes, and wait days for a human.RunHuman approvalA person says yes before anything that is costly or cannot be undone.Walk
Knowledge
Safety and proof
Checks before and afterCode that inspects what goes into the model and what comes out, and can stop it.WalkPrompt injectionText in a document or tool result that tries to give the model new orders.WalkEvaluationProve it works on cases you wrote down, every time you change it.WalkTracing an AI systemRecord every model call, tool call, token and cost for each turn.WalkLLM cost: tokens, caching and routingKnow the cost per answer, then cut it.Walk
Engineering basics
Version control and pull requestsEvery change saved, explained and reviewed before it goes live.CrawlAPIs and contractsA promised way for programs to talk, which you change carefully.CrawlIdempotencyRunning the same thing twice gives the same result as running it once.CrawlRetries, timeouts and backoffAssume calls fail sometimes, and handle it calmly.WalkPorts and adaptersCode to an interface, so you can swap a vendor without a rewrite.Run
LLM Engineering
Key concepts in LLM Engineering
Then follow the path: LLM Engineering · Crawl, Walk, Run
How models work
How a language model worksIt predicts the next token, again and again.CrawlAttention and the transformerEach word looks at every other word to decide what matters.CrawlTokens and the context windowModels read and write in tokens, and can only see so many at once.CrawlTemperature and samplingHow the model picks the next token from its options.CrawlEmbeddingsTurn text into a list of numbers so similar meanings sit close together.Crawl
Adapting models
Prompt, retrieve or fine-tune?Three ways to adapt a model, in order of cost.WalkPre-training, fine-tuning and alignmentThree stages turn raw text into a helpful assistant.WalkLoRA and efficient fine-tuningFine-tune a big model by training a small add-on.WalkRetrieval-augmented generation (RAG)Find the right passages first, then let the model answer from them, with citations.Walk
Serving models
QuantisationStore a model's numbers with fewer bits so it runs smaller and faster.WalkServing: batching, KV cache, latency vs throughputHow to answer many requests fast and cheap on GPUs.RunCachingKeep a copy of a result so you don't compute it again.WalkLLM cost: tokens, caching and routingKnow the cost per answer, then cut it.Walk
Quality and data
Hallucination and groundingConfident answers that are not true, and how to anchor answers to sources.CrawlEvaluationProve it works on cases you wrote down, every time you change it.WalkBenchmarks and contaminationPublic scores are a starting point; your own tests decide.RunTraining data: collect, clean, dedupe, licenceGood models start with good, legal, de-duplicated data.Run
ML Engineering
Key concepts in ML Engineering
Then follow the path: ML Engineering · Crawl, Walk, Run
Learning basics
Supervised, unsupervised and reinforcement learningLearning from labelled answers, from patterns alone, or from rewards.CrawlTrain, validation and test setsKeep some data back so you can tell if the model really learned.CrawlStart with a baselineBeat a simple rule first, and say by how much.CrawlOverfitting and underfittingMemorising the training data versus not learning enough from it.CrawlData leakageWhen the model is secretly given the answer during training.CrawlChoosing the right metricAccuracy is rarely the number that matters.Walk
Features and data
Feature engineeringTurn raw data into signals a model can use.WalkTraining-serving skewWhen the model sees different data in production than it was trained on.WalkData quality dimensionsSix ways data goes wrong, and a check for each.CrawlMedallion architecture (bronze, silver, gold)Clean data in three layers so each step is simple and you can always go back.Crawl
Production ML
Versioning data, code and modelsBe able to say exactly which data and code made which model.WalkBatch vs online predictionScore everything on a schedule, or one request at a time as it comes.WalkDriftThe world changes and the model quietly gets worse.RunCI/CDEvery change is built, tested and deployed by a pipeline, not by hand.CrawlObservability: logs, metrics, tracesSee what your system is doing from the outside, especially when it goes wrong.Walk
Software Engineering
Key concepts in Software Engineering
Then follow the path: Software Engineering · Crawl, Walk, Run
Working together
Version control and pull requestsEvery change saved, explained and reviewed before it goes live.CrawlThe testing pyramidMany small fast tests, fewer bigger ones, a handful end to end.CrawlCI/CDEvery change is built, tested and deployed by a pipeline, not by hand.CrawlInfrastructure as codeThe whole environment described in files you can review and re-create.Walk
Building services
APIs and contractsA promised way for programs to talk, which you change carefully.CrawlIdempotencyRunning the same thing twice gives the same result as running it once.CrawlRetries, timeouts and backoffAssume calls fail sometimes, and handle it calmly.WalkQueues and asynchronous workHand work to another part of the system to do later, safely.WalkTransactions and consistencyEither all of a change happens, or none of it does.WalkCachingKeep a copy of a result so you don't compute it again.Walk
Running services
Observability: logs, metrics, tracesSee what your system is doing from the outside, especially when it goes wrong.WalkIdentity, secrets and least privilegeEvery person and program gets only the access it needs, and keys never live in code.WalkPorts and adaptersCode to an interface, so you can swap a vendor without a rewrite.Run
Forward Deployed Engineering
Key concepts in Forward Deployed Engineering
Then follow the path: Forward Deployed Engineering · Crawl, Walk, Run
Understand
Discovery: the problem in their wordsUnderstand the customer's real problem before proposing anything.CrawlAgree the measure of success firstDecide with the customer how you will know it worked, before you build.CrawlData map and ownersWhere the customer's data lives, what shape it is in, and who can say yes.CrawlScope by what it will not doWrite down what the system will NOT do. It is the most useful line in the plan.CrawlDeterministic firstIf ordinary code can do it, code does it. The model handles only the part that needs judgement.Crawl
Deliver
Prototype vs productA prototype proves value on real data; a product survives every day after.WalkDeploy in their cloudThe system runs in the customer's account, under their rules, from day one.WalkEvidence, not demosShow quality and cost on one page the customer can read.WalkIdentity, secrets and least privilegeEvery person and program gets only the access it needs, and keys never live in code.Walk
Scale
Hand-over and runbooksLeave a system the customer's team can run without you.RunOne framework, many customersBuild reusable pieces once; bind them to each customer's cloud and rules.RunPorts and adaptersCode to an interface, so you can swap a vendor without a rewrite.RunThe harnessEverything around the model that makes it safe to use.Walk
Glossary
All concepts, A to Z
- Agree the measure of success first: Decide with the customer how you will know it worked, before you build.
- APIs and contracts: A promised way for programs to talk, which you change carefully.
- Attention and the transformer: Each word looks at every other word to decide what matters.
- Backfills and reprocessing: Re-run a pipeline for past dates, safely.
- Batch vs online prediction: Score everything on a schedule, or one request at a time as it comes.
- Batch vs streaming: Process data in scheduled chunks, or event by event as it happens.
- Benchmarks and contamination: Public scores are a starting point; your own tests decide.
- Caching: Keep a copy of a result so you don't compute it again.
- Change data capture (CDC): Copy only what changed in a database, by reading its change log.
- Checks before and after: Code that inspects what goes into the model and what comes out, and can stop it.
- Choosing the right metric: Accuracy is rarely the number that matters.
- Chunking: How you cut documents into pieces decides what search can find.
- CI/CD: Every change is built, tested and deployed by a pipeline, not by hand.
- Data contracts: A written agreement between the team that produces data and the teams that use it.
- Data leakage: When the model is secretly given the answer during training.
- Data map and owners: Where the customer's data lives, what shape it is in, and who can say yes.
- Data quality dimensions: Six ways data goes wrong, and a check for each.
- Deploy in their cloud: The system runs in the customer's account, under their rules, from day one.
- Deterministic first: If ordinary code can do it, code does it. The model handles only the part that needs judgement.
- Dimensional modelling (star schema): Facts in the middle, descriptions around them: tables built for questions.
- Discovery: the problem in their words: Understand the customer's real problem before proposing anything.
- Drift: The world changes and the model quietly gets worse.
- Embeddings: Turn text into a list of numbers so similar meanings sit close together.
- ETL vs ELT: Two orders for the same three steps: extract, transform, load.
- Evaluation: Prove it works on cases you wrote down, every time you change it.
- Evidence, not demos: Show quality and cost on one page the customer can read.
- Feature engineering: Turn raw data into signals a model can use.
- File formats: CSV, JSON, Avro, Parquet: How data is laid out in a file decides how fast and cheap it is to read.
- Freshness and SLAs: Promise how up to date data will be, then measure it.
- Full vs incremental loads: Reload everything every time, or only what is new since last time.
- Governance, access and personal data: Who owns each dataset, who may see it, and how personal data is protected.
- Hallucination and grounding: Confident answers that are not true, and how to anchor answers to sources.
- Hand-over and runbooks: Leave a system the customer's team can run without you.
- How a language model works: It predicts the next token, again and again.
- Human approval: A person says yes before anything that is costly or cannot be undone.
- Idempotency: Running the same thing twice gives the same result as running it once.
- Identity, secrets and least privilege: Every person and program gets only the access it needs, and keys never live in code.
- Infrastructure as code: The whole environment described in files you can review and re-create.
- Lineage: A map of where each table and number came from, and what depends on it.
- LLM cost: tokens, caching and routing: Know the cost per answer, then cut it.
- LoRA and efficient fine-tuning: Fine-tune a big model by training a small add-on.
- Medallion architecture (bronze, silver, gold): Clean data in three layers so each step is simple and you can always go back.
- Normalised vs denormalised: Store each fact once, or repeat it to make reading faster.
- Observability: logs, metrics, traces: See what your system is doing from the outside, especially when it goes wrong.
- One framework, many customers: Build reusable pieces once; bind them to each customer's cloud and rules.
- Open table formats and ACID: Delta Lake, Apache Iceberg and Apache Hudi make files in storage behave like database tables.
- Orchestration and DAGs: Run each step in the right order, on time, and retry what fails.
- Overfitting and underfitting: Memorising the training data versus not learning enough from it.
- Partitioning and clustering: Arrange data so a query can skip what it does not need.
- Ports and adapters: Code to an interface, so you can swap a vendor without a rewrite.
- Pre-training, fine-tuning and alignment: Three stages turn raw text into a helpful assistant.
- Prompt injection: Text in a document or tool result that tries to give the model new orders.
- Prompt, retrieve or fine-tune?: Three ways to adapt a model, in order of cost.
- Prototype vs product: A prototype proves value on real data; a product survives every day after.
- Quantisation: Store a model's numbers with fewer bits so it runs smaller and faster.
- Queues and asynchronous work: Hand work to another part of the system to do later, safely.
- Retries, timeouts and backoff: Assume calls fail sometimes, and handle it calmly.
- Retrieval-augmented generation (RAG): Find the right passages first, then let the model answer from them, with citations.
- Schema evolution: How a table's columns can change over time without breaking readers.
- Scope by what it will not do: Write down what the system will NOT do. It is the most useful line in the plan.
- Semantic layer and metrics: Define each business number once, so dashboards and AI give the same answer.
- Separating storage and compute: Keep data in one place and rent the engines that read it only when needed.
- Serving: batching, KV cache, latency vs throughput: How to answer many requests fast and cheap on GPUs.
- Slowly changing dimensions: How to record that a customer moved city, without rewriting history.
- Start with a baseline: Beat a simple rule first, and say by how much.
- State and durable waits: Remember where a long task was, survive crashes, and wait days for a human.
- Structured output: Ask the model for JSON in a fixed shape, then check it in code.
- Supervised, unsupervised and reinforcement learning: Learning from labelled answers, from patterns alone, or from rewards.
- Temperature and sampling: How the model picks the next token from its options.
- The agent loop: Model, tool, result, model again, until done or a limit is hit.
- The harness: Everything around the model that makes it safe to use.
- The small-files problem: Thousands of tiny files make every query slow.
- The testing pyramid: Many small fast tests, fewer bigger ones, a handful end to end.
- Tokens and the context window: Models read and write in tokens, and can only see so many at once.
- Tool calling: The model asks your code to do something; your code decides whether it runs.
- Tracing an AI system: Record every model call, tool call, token and cost for each turn.
- Train, validation and test sets: Keep some data back so you can tell if the model really learned.
- Training data: collect, clean, dedupe, licence: Good models start with good, legal, de-duplicated data.
- Training-serving skew: When the model sees different data in production than it was trained on.
- Transactions and consistency: Either all of a change happens, or none of it does.
- Version control and pull requests: Every change saved, explained and reviewed before it goes live.
- Versioning data, code and models: Be able to say exactly which data and code made which model.
- Warehouse, lake and lakehouse: Three ways to store data for analysis, and why most teams now use the third.
- Where data costs come from: Scanning, shuffling, skew and idle clusters: the four usual suspects.