<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"><channel><title>Clean Data Engineering · Writing</title><link>https://cleandataengineering.ai/writing/</link>
<description>Short pieces on AI, LLM and data engineering.</description><language>en</language><item><title>Source semantic understanding: a key to data program success</title><link>https://cleandataengineering.ai/writing/source-semantics/</link><guid>https://cleandataengineering.ai/writing/source-semantics/</guid>
<pubDate>Fri, 06 Nov 2026 08:00:00 +0530</pubDate><description>Most &quot;the numbers don&#x27;t match&quot; problems after go-live are not pipeline bugs. They come from not knowing what the source data means and how the source system behaves. This post draws the six questions to ask about every source, who can answer them, where to write the answers down, and why &quot;matches the trusted report&quot; is the only real definition of done.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Shipping agents: versioning prompts, models and tools together</title><link>https://cleandataengineering.ai/writing/shipping-agents/</link><guid>https://cleandataengineering.ai/writing/shipping-agents/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An agent&#x27;s behaviour comes from its prompt, its model, its tools, its checks and its settings, so release them together as one sealed, fingerprinted unit. Pin the model, gate every release on quality, cost and every model it can reach, send a small share of traffic first, rehearse the rollback, keep a kill switch, serve exactly what was tested, and stamp the release id on every trace.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>DBOS vs Temporal: durable agents without the drama</title><link>https://cleandataengineering.ai/writing/dbos-vs-temporal/</link><guid>https://cleandataengineering.ai/writing/dbos-vs-temporal/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Durable execution lets a long agent task survive crashes, restarts and waits of days, with each step done once. DBOS does it as a library that keeps state in the Postgres you already run; Temporal does it with a separate, mature cluster that workers poll. Pick by what you already run and what you need, and if no task waits longer than one request, you may need neither.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Building agents on a moving ship: what has settled, what is still open, and how not to get stuck</title><link>https://cleandataengineering.ai/writing/moving-ship/</link><guid>https://cleandataengineering.ai/writing/moving-ship/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Some parts of the agent stack have settled: OpenTelemetry for traces, Postgres for state, MCP for tools, JSON Schema for typed outputs, OAuth and OpenID Connect for identity, Git and CI for releases, and open table formats for data. Others are still moving, such as frameworks, agent-to-agent protocols, memory, evals, guardrail products, GenAI field names and the models themselves. Pick boring standards for the first group, put the second behind ports with contract tests, and keep the code that protects money in your own hands.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Rust or Python for agents? Probably both</title><link>https://cleandataengineering.ai/writing/rust-or-python/</link><guid>https://cleandataengineering.ai/writing/rust-or-python/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Python wins for the thinking parts of an agent: orchestration, prompts, evals and fast iteration, with the whole AI library ecosystem behind it. Rust wins for the heavy lifting: sandboxes, gateways on the hot path, and tools shipped as one small binary. The model call usually dominates latency, so Rust helps only where the model is not involved.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Choosing your agent loop: LangGraph vs OpenAI Agents SDK vs Claude Agent SDK vs plain code</title><link>https://cleandataengineering.ai/writing/choosing-agent-loop/</link><guid>https://cleandataengineering.ai/writing/choosing-agent-loop/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Every agent runs the same small loop: build the context, ask the model, run the tools it asked for, and go round again. LangGraph, the OpenAI Agents SDK, the Claude Agent SDK and a plain while-loop are four ways to own that loop, each with real gifts and real costs. Pick by the job, then put the loop behind a port so you can change your mind.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Streaming on Databricks without the pain</title><link>https://cleandataengineering.ai/writing/databricks-streaming/</link><guid>https://cleandataengineering.ai/writing/databricks-streaming/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Most teams that ask for streaming need fresh data, not a stream. Start by asking how fresh the data must be; often a batch every 5 to 15 minutes is enough and far simpler. When you do stream, run the same medallion at two speeds, keep your checkpoints, set a watermark, and pick a trigger that matches how often the data actually arrives.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>CI/CD for Databricks with Asset Bundles</title><link>https://cleandataengineering.ai/writing/databricks-asset-bundles/</link><guid>https://cleandataengineering.ai/writing/databricks-asset-bundles/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Most Databricks jobs in production were clicked together in the workspace UI, so nobody can say what changed or rebuild them in another workspace. Databricks Asset Bundles put jobs, pipelines and their settings in one project folder, next to the code, and deploy the same folder to dev, test and prod. Add a pipeline that validates every pull request and deploys on merge as a service principal, and releases stop being a nervous afternoon.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Agent-to-agent protocols (A2A): worth watching, not yet settled</title><link>https://cleandataengineering.ai/writing/a2a-not-settled/</link><guid>https://cleandataengineering.ai/writing/a2a-not-settled/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Agent-to-agent protocols promise that agents built by different teams or companies can find each other and hand over work in one standard way. A2A is the leading one, and it is worth learning, but in October 2026 it is still moving. Don&#x27;t bet your agent on it yet. Put hand-offs behind a port, pass a typed ticket with depth and fan-out limits, keep one trace across agents, and check identity at every hop, so plugging A2A in later is one adapter.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Describe the world, not the test</title><link>https://cleandataengineering.ai/writing/describe-the-world/</link><guid>https://cleandataengineering.ai/writing/describe-the-world/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Writing hundreds of test scripts for an agent doesn&#x27;t scale, because every new rule or fault means editing scripts by hand. Instead, write down the agent&#x27;s world once: its users, systems and data, rules, faults, approvers and time. Scenarios are then generated from that description, including combinations nobody would have thought to script. This post shows what a good world description holds, with a small example, and says honestly what AgentTwin&#x27;s format does today.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>What repeats is broken</title><link>https://cleandataengineering.ai/writing/what-repeats-is-broken/</link><guid>https://cleandataengineering.ai/writing/what-repeats-is-broken/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Every pipeline sees the same data more than once: retries, queues that deliver twice, files sent again, backfills and tired re-runs. A pipeline is idempotent when processing the same input twice gives the same result as processing it once. This post shows, with numbers from real recorded runs, how a naive pipeline doubles revenue and the six patterns that make repeats safe.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Why we wrote our own agent loop</title><link>https://cleandataengineering.ai/writing/why-our-own-loop/</link><guid>https://cleandataengineering.ai/writing/why-our-own-loop/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Our reference agent, a support agent that can move real money, runs on a loop we wrote ourselves in one file of 416 lines. We did it so every budget and check sits at a place a test can reach. We kept the door open by putting the loop behind a port: a Pydantic AI version is written, LangGraph and Claude Agent SDK versions are planned, and all of them must pass the same frozen tests.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Your Databricks bill has a leak: 10 cost mistakes we keep seeing</title><link>https://cleandataengineering.ai/writing/databricks-cost-leaks/</link><guid>https://cleandataengineering.ai/writing/databricks-cost-leaks/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>A Databricks bill rarely jumps because of one big decision. It leaks through ten small holes: the wrong kind of compute, clusters left running, tiny files, one slow task, Python where Spark would do, and no tags to say who spent what. This note shows how to spot each one, the fix in plain words, and a ten-minute weekly check.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Fail fast, own the result: critical thinking for forward-deployed engineers</title><link>https://cleandataengineering.ai/writing/fail-fast-own-the-result/</link><guid>https://cleandataengineering.ai/writing/fail-fast-own-the-result/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Failing fast does not mean failing more. It means finding out you are wrong on day two instead of week seven, by writing down the beliefs your plan stands on and testing the riskiest one in the cheapest way that could prove it wrong. A forward-deployed engineer does this because they own the customer&#x27;s result, not just their own code.</description>
<category>Career</category><category>Builders</category></item><item><title>The LLM gateway: the switch between your agent and every model</title><link>https://cleandataengineering.ai/writing/llm-gateway/</link><guid>https://cleandataengineering.ai/writing/llm-gateway/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An LLM gateway is one service that sits between all your apps and every model provider. It holds the keys, the list of approved models, the limits and budgets per team, the cache, the screening and the cost log, and it switches to another provider when one fails. It is not the provider, and it is not the place for business rules about money or consent; those belong in the agent&#x27;s harness.</description>
<category>LLM Engineering</category><category>Builders</category></item><item><title>Measuring agent ROI: what to count before you scale</title><link>https://cleandataengineering.ai/writing/agent-roi/</link><guid>https://cleandataengineering.ai/writing/agent-roi/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An agent pays off only if the cost of each completed task, counting people as well as the model, falls without errors or complaints rising. Measure the same tasks before and after, include the hidden costs of review, upkeep and on-call, and prove it on a small pilot with a baseline before you scale. One page of numbers is enough to decide.</description>
<category>AI Engineering</category><category>Leaders</category></item><item><title>From data engineer to AI engineer: the 6-month switch</title><link>https://cleandataengineering.ai/writing/data-engineer-to-ai-engineer/</link><guid>https://cleandataengineering.ai/writing/data-engineer-to-ai-engineer/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>A data engineer already knows most of what an AI engineer does, because most of an agent is ordinary software and data plumbing. What is left is six gaps, one a month: model basics, tools and the agent loop, context and retrieval, evals, guardrails, and tracing and cost. Prove it with one project, not a pile of certificates.</description>
<category>Career</category><category>Builders</category></item><item><title>Unity Catalog done right: one governance model for data and AI</title><link>https://cleandataengineering.ai/writing/unity-catalog-done-right/</link><guid>https://cleandataengineering.ai/writing/unity-catalog-done-right/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Unity Catalog is the one place on Databricks that says where every table, model and agent tool lives, who may use it, and where it came from. Done right, that means a simple three-part layout, grants to groups set high up the tree, masks and filters on the table itself, and descriptions good enough for people and AI assistants to read. This post draws each part and lists the mistakes that undo it.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Bronze, Silver, Gold, and where most teams get it wrong</title><link>https://cleandataengineering.ai/writing/medallion-done-right/</link><guid>https://cleandataengineering.ai/writing/medallion-done-right/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Almost every lakehouse says it uses the medallion architecture. Fewer use it well. The layers aren&#x27;t folders; each one is a promise about the data inside it. This post draws the layers as a kitchen, then shows the seven mistakes we see most and the one test that prevents them.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Your agent has the keys: security architecture for autonomous agents</title><link>https://cleandataengineering.ai/writing/agent-security/</link><guid>https://cleandataengineering.ai/writing/agent-security/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An agent that can call tools holds keys to real systems, so the question is whose keys and how many. It should act as the signed-in user, see only the tools that user may use, and meet a second check at the system it calls. Treat everything a tool returns as data, keep secrets out of prompts, and make sure what serves traffic is what you tested.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>The autonomous agent tech stack, one concern at a time</title><link>https://cleandataengineering.ai/writing/agent-tech-stack/</link><guid>https://cleandataengineering.ai/writing/agent-tech-stack/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An autonomous agent has about fifteen jobs to do besides calling a model: the loop, the gateway, tools, context, memory, guardrails, identity, approvals, durability, tracing, evals, cost, release and more. Each job stays the same while the products doing it keep changing. This post gives one picture and one table, with two to four example tools per job across open source and the clouds.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Just use Postgres: the one database your agent needs</title><link>https://cleandataengineering.ai/writing/just-use-postgres/</link><guid>https://cleandataengineering.ai/writing/just-use-postgres/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An agent needs to remember a lot of different things: the conversation, what it already did, what is waiting for a person, and what it decided. One PostgreSQL database can hold all of it, which means one thing to secure, back up and pay for. Move a job out only when you see a clear sign that it has outgrown Postgres.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>The horse and the harness</title><link>https://cleandataengineering.ai/writing/horse-and-harness/</link><guid>https://cleandataengineering.ai/writing/horse-and-harness/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An AI model is a strong, fast horse that doesn&#x27;t know your road and sometimes bolts. What protects your company is the harness around it: clear lanes for what the AI may do alone, what needs a person, and what it must never do, with proof that those lines hold. A better model helps, but it never replaces the reins.</description>
<category>AI Engineering</category><category>Leaders</category></item><item><title>Every guardrail your agent needs, sorted: must-have, should-have, rule or judge</title><link>https://cleandataengineering.ai/writing/guardrails-sorted/</link><guid>https://cleandataengineering.ai/writing/guardrails-sorted/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>&quot;Guardrails&quot; sounds like one product you buy. In a working agent it is about fifty small checks, each with a job, a place and a price. We sorted 57 common ones. 39 are plain rules in code, and only 2 need an LLM judge. Here are the labels, the pictures, and the 15 to build first.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>AgentTwin vs LangWatch, Veris and friends: the simulation landscape</title><link>https://cleandataengineering.ai/writing/simulation-landscape/</link><guid>https://cleandataengineering.ai/writing/simulation-landscape/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Tools that test agents in a simulation each pretend to be a different part of the world. Conversation simulators such as LangWatch Scenario play the customer, red-team kits such as PyRIT play the attacker, Strands ToolSimulator plays the tools, sandboxes such as Veris play the whole environment, τ²-bench is one fixed published world, and AgentTwin builds the world from your agent&#x27;s spec and judges what changed in it. Pick by the risk you fear most, and expect to combine more than one.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Before it ships: offline evals for autonomous agents</title><link>https://cleandataengineering.ai/writing/offline-evals/</link><guid>https://cleandataengineering.ai/writing/offline-evals/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An agent can be wrong without throwing a single error, so you need a test it must pass before every release. Build a golden set of cases with known right answers, play whole conversations against a simulated world, judge each case with code first and an LLM judge only where code can&#x27;t see, compare with the last release slice by slice, and turn every production failure into a new case.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>What one conversation costs: under a tenth of a cent</title><link>https://cleandataengineering.ai/writing/one-conversation-cost/</link><guid>https://cleandataengineering.ai/writing/one-conversation-cost/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>A real conversation with our test claims assistant cost $0.00087 in model fees, and every turn we measured cost between nothing and $0.00093. It is that low because of four design choices, not luck. The model bill is the smallest line in your AI budget; people, review, upkeep and the platform are the real costs, so ask your team for the cost of each completed task.</description>
<category>Cost</category><category>Leaders</category></item><item><title>Should you build or buy your agent? A one-page guide for leaders</title><link>https://cleandataengineering.ai/writing/build-or-buy-agent/</link><guid>https://cleandataengineering.ai/writing/build-or-buy-agent/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Build or buy is rarely an either-or choice for an agent. Six questions decide it: how special the process is, where the data may go, what each task costs to run, what you can switch later, who owns it after launch, and what you must prove to auditors. Most teams end up in the middle: buy the plumbing, and build the rules that protect money and make you different.</description>
<category>AI Engineering</category><category>Leaders</category></item><item><title>Prompt, RAG or fine-tune? Pick in that order</title><link>https://cleandataengineering.ai/writing/prompt-rag-finetune/</link><guid>https://cleandataengineering.ai/writing/prompt-rag-finetune/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>There are three ways to make a language model better at your job. Prompting changes the instructions, RAG hands it the right facts at question time, and fine-tuning trains the model itself. They fix different problems and cost very different amounts, so try them in that order and only climb when your eval set shows the step below can&#x27;t do it.</description>
<category>LLM Engineering</category><category>Builders</category></item><item><title>Swap anything, break nothing: software design patterns for agent harnesses</title><link>https://cleandataengineering.ai/writing/harness-design-patterns/</link><guid>https://cleandataengineering.ai/writing/harness-design-patterns/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Everything around an agent changes every few months: the model, the gateway, the trace backend, the cloud. A handful of old software design patterns keep those changes small. Put each vendor behind a port, pick the adapter in one YAML file, line up the checks so any one can stop a call, wrap calls in retries and traces, and make every adapter pass the same contract tests.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>A flight simulator for your agent: what AgentTwin is, and what it is not</title><link>https://cleandataengineering.ai/writing/agenttwin-flight-simulator/</link><guid>https://cleandataengineering.ai/writing/agenttwin-flight-simulator/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>AgentTwin keeps your AI agent real and makes everything around it pretend: the customer, the systems and data, the approver, the faults and the clock. After each scenario it judges what changed in that pretend world, not what the agent said. It is not a digital twin, a model benchmark, a load test or production monitoring, and this post is honest about which parts are built.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>The AI engineer roadmap: what to learn, in what order, and what to skip</title><link>https://cleandataengineering.ai/writing/ai-engineer-roadmap/</link><guid>https://cleandataengineering.ai/writing/ai-engineer-roadmap/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>AI engineering is a lot to take in, but the order makes it manageable. Learn it in six phases: foundations, building with models, context and knowledge, reliability, running it, and an optional deeper phase on the models themselves. Skip model training, framework-hopping and project-free certificates early on, and finish each phase with one small project.</description>
<category>Career</category><category>Builders</category></item><item><title>Who checks the checker? The oracle problem in agent simulation</title><link>https://cleandataengineering.ai/writing/oracle-problem/</link><guid>https://cleandataengineering.ai/writing/oracle-problem/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>In testing, the &quot;oracle&quot; is whatever decides pass or fail, and for an agent in an open conversation there is no free one. A simulation should decide with code that reads what changed in the world first, then rules on actions, then what was owed but never done, and only last an LLM judge that has been checked against people. Then test the checkers themselves by planting known bugs and seeing which ones they miss.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>How to learn in the AI era: draw the map, grow a T, and teach it back</title><link>https://cleandataengineering.ai/writing/learn-in-the-ai-era/</link><guid>https://cleandataengineering.ai/writing/learn-in-the-ai-era/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>When AI can answer any question, facts are cheap; what&#x27;s scarce is knowing what to learn, how deep to go, and proving you understand it. Draw a map of your field, learn wide at two shallow depths and deep in your core, use AI as a tutor rather than a ghostwriter, and go round the loop of see it, run it, build it, teach it.</description>
<category>Career</category><category>Builders</category></item><item><title>Talk to your data the right way: Genie and metric views on Databricks</title><link>https://cleandataengineering.ai/writing/genie-metric-views/</link><guid>https://cleandataengineering.ai/writing/genie-metric-views/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>A chat window over your data is easy to build and easy to get wrong. The model should not write SQL against your tables; it should pick from numbers you have already defined. On Databricks that means metric views in Unity Catalog as the one definition, with AI/BI Genie and your dashboards both reading from them.</description>
<category>Data + AI</category><category>Builders</category></item><item><title>The agent testing pyramid</title><link>https://cleandataengineering.ai/writing/agent-testing-pyramid/</link><guid>https://cleandataengineering.ai/writing/agent-testing-pyramid/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Most of an agent is ordinary software, so most of its tests are ordinary tests that run in seconds for free. Tests that need a real model are few, and they sit near the top. This post draws the whole testing strategy as a pyramid, from unit tests at the bottom to production monitoring at the top.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Data contracts: stop breaking the team downstream</title><link>https://cleandataengineering.ai/writing/data-contracts/</link><guid>https://cleandataengineering.ai/writing/data-contracts/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Most data breakages start with an upstream change nobody announced. A data contract is a written agreement between the team that produces a dataset and the teams that use it, checked by a machine in three places. This post draws what a contract holds, where it is checked, and which changes break it.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>What a forward-deployed engineer actually does</title><link>https://cleandataengineering.ai/writing/forward-deployed-engineer/</link><guid>https://cleandataengineering.ai/writing/forward-deployed-engineer/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>A forward-deployed engineer works inside a customer&#x27;s team to make a product or AI system actually work in their world. The job runs in six steps: discovery, a scope that does as much as possible without a model, a prototype, deploying in the customer&#x27;s own cloud, proving it with evals and cost numbers, and handing over. It needs engineering plus ten traits, from listening and judgement to ownership of the customer&#x27;s result and the big picture.</description>
<category>Career</category><category>Builders</category></item><item><title>There is no best model, only the best model for this step</title><link>https://cleandataengineering.ai/writing/model-per-step/</link><guid>https://cleandataengineering.ai/writing/model-per-step/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Picking one &quot;best&quot; model for a whole agent is the wrong question. Test candidates on a small golden set of your own tasks, weigh accuracy, cost, speed and reliability for each step, decide where your data is allowed to go, and give each step the cheapest worker that passes, which is often plain code. Then pin the versions and test again whenever a model changes.</description>
<category>LLM Engineering</category><category>Builders</category></item><item><title>Your Databricks cluster scales up and never comes back down</title><link>https://cleandataengineering.ai/writing/cluster-never-scales-down/</link><guid>https://cleandataengineering.ai/writing/cluster-never-scales-down/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Autoscaling on Databricks grows a cluster quickly and shrinks it slowly, and sometimes not at all. The usual reasons are a minimum set too high, clusters that are never quite idle, workers holding data the autoscaler doesn&#x27;t want to throw away, and streaming or shared workloads that keep everything busy. This note explains each one, the fixes, and how to see it in your own cluster events and billing tables.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Your agent passed every test. Then it met a real customer.</title><link>https://cleandataengineering.ai/writing/agent-met-real-customer/</link><guid>https://cleandataengineering.ai/writing/agent-met-real-customer/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Scripted tests send one neat question and read one neat reply. Real customers change their minds, ask two things at once, come back days later and say things in their own words, while systems time out, data goes stale, attackers plant text and approvers go quiet. We ran a good agent and a faulty one through the same five simulated scenarios: read only the replies and both pass everything; read what changed in the world and the faulty one fails all five.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Delta, Iceberg, or both? Open table formats are settled</title><link>https://cleandataengineering.ai/writing/delta-iceberg/</link><guid>https://cleandataengineering.ai/writing/delta-iceberg/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An open table format is Parquet files plus a log that turns a folder into a real table, with transactions, time travel, schema changes and safe concurrent writes. The move to open formats has settled. Delta Lake and Apache Iceberg do the same job with different bookkeeping, and today you can often keep one copy of the data and let readers of both kinds use it.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Stop writing pipelines: one metadata-driven ingestion framework for 500 tables</title><link>https://cleandataengineering.ai/writing/metadata-driven-ingestion/</link><guid>https://cleandataengineering.ai/writing/metadata-driven-ingestion/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Most teams bring data in with one hand-written pipeline per source table, and by table 500 nobody can keep them all working. A metadata-driven framework keeps one row of settings per table in a config store and runs one generic pipeline that reads it. Adding table 501 becomes a reviewed row, not new code.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>Under the hood: why fundamentals are the slow road that wins</title><link>https://cleandataengineering.ai/writing/under-the-hood/</link><guid>https://cleandataengineering.ai/writing/under-the-hood/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Tools and AI make it easy to get results without knowing what happens underneath, and many engineers now skip that part. It is slower to learn what&#x27;s under the hood, but it is what lets you debug, control cost, design well and check an AI&#x27;s answer. Learn one layer below where you work, and open the hood on the tools you use every week.</description>
<category>Career</category><category>Builders</category></item><item><title>Data quality as config, not code</title><link>https://cleandataengineering.ai/writing/data-quality-as-config/</link><guid>https://cleandataengineering.ai/writing/data-quality-as-config/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Most teams have data quality checks, but they are scattered across notebooks, each written a little differently. A metadata-driven framework moves every rule into a YAML file or a rules table, and one engine runs them at each hop. Bad rows go to quarantine with a reason, results land in a table, and a new dataset is a new file, not new code.</description>
<category>Data Engineering</category><category>Builders</category></item><item><title>The model is 10%. The harness is the other 90%.</title><link>https://cleandataengineering.ai/writing/model-is-ten-percent/</link><guid>https://cleandataengineering.ai/writing/model-is-ten-percent/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Most of a working agent is not the model. It is the ordinary software around it, called the harness: what the model sees, what it may touch, what pulls it back, and what writes every step down. This post draws the whole harness on one page.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>One agent or many? When multi-agent is a mistake</title><link>https://cleandataengineering.ai/writing/one-agent-or-many/</link><guid>https://cleandataengineering.ai/writing/one-agent-or-many/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Splitting work across several agents looks tidy on a whiteboard, but every hand-off loses context, every conversation between agents costs money, and every extra agent is one more thing to trace and test. Go multi-agent when permissions, skills, parallel work or sheer size truly demand it. Otherwise, one agent with a router and good tools does the job better.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Most of an agent is ordinary software</title><link>https://cleandataengineering.ai/writing/agent-is-ordinary-software/</link><guid>https://cleandataengineering.ai/writing/agent-is-ordinary-software/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>We counted every line of code in our reference agent and sorted it by job. The code that calls the model is 817 of 24,631 lines, about 3%, and the rest is tests, checks, durable workflows, state, tracing and plumbing. So most of building an agent is ordinary engineering, and the skills you already have apply.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Human in the loop without humans in the way</title><link>https://cleandataengineering.ai/writing/human-in-the-loop/</link><guid>https://cleandataengineering.ai/writing/human-in-the-loop/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>A person should approve the few actions that can&#x27;t be undone, and nothing else. The approval should be a durable wait that holds no server, shows the approver enough to decide in seconds, and has a clear answer for &quot;nobody replied&quot;. Autonomy widens only when people have agreed with the agent on that one decision, measured.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Answer without the model first: 104 ms vs 7 seconds</title><link>https://cleandataengineering.ai/writing/answer-without-the-model/</link><guid>https://cleandataengineering.ai/writing/answer-without-the-model/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Our claims assistant answers some questions without calling the model at all. When a customer names the claim, a few plain rules send the message down a direct path of one read and one approved template: 104 ms and $0, against about 7 seconds through the model loop. This build log shows how the router decides, which checks the direct path still passes, and when not to do it.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Seatbelts for agents: guardrails that actually hold</title><link>https://cleandataengineering.ai/writing/guardrails-that-hold/</link><guid>https://cleandataengineering.ai/writing/guardrails-that-hold/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>A guardrail that only exists in a prompt, or that lets everything through when it breaks, is a seatbelt drawn on your shirt. This post follows one real guardrail, &quot;the user actually asked for this&quot;, through the eight things that make it hold: where it sits, what it reads, how it fails, how it is tested and how it is watched.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>It passed offline. Now watch it live: online evals and monitoring</title><link>https://cleandataengineering.ai/writing/online-evals/</link><guid>https://cleandataengineering.ai/writing/online-evals/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Passing the release gate proves an agent handles the cases you wrote down, not the ones customers bring. After release, quick checks keep running inside every turn, slower scoring re-reads finished turns, and a handful of rates are compared with the agent&#x27;s own normal from last week. Add a test customer that asks known questions on a schedule, join complaints and returns back to the run that caused them, and turn every live failure into a new offline case.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>One harness, five homes: the agent harness on Azure, GCP, AWS, Databricks and open source</title><link>https://cleandataengineering.ai/writing/harness-five-homes/</link><guid>https://cleandataengineering.ai/writing/harness-five-homes/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An agent harness has the same jobs on every platform: a model door, a loop, tools, state, durable waits, identity, guardrails, tracing, evals and hosting. Azure, Google Cloud, AWS, Databricks and open source fill those jobs with different products. Design the harness once, then pick the home by where your data and identity already live and which managed pieces your team trusts.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>OpenTelemetry is here to stay: instrument your agent once, send traces anywhere</title><link>https://cleandataengineering.ai/writing/opentelemetry-for-agents/</link><guid>https://cleandataengineering.ai/writing/opentelemetry-for-agents/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Almost everything in an agent stack changes every few months, but how you send traces out of it has settled: OpenTelemetry. Write your spans once with its library and its standard GenAI field names, put a Collector in between, and the backend becomes one line of config. The standard is settled; some GenAI field names are still moving, so keep them in one file.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Why your agent costs $4 per question: 9 cost mistakes</title><link>https://cleandataengineering.ai/writing/agent-cost-mistakes/</link><guid>https://cleandataengineering.ai/writing/agent-cost-mistakes/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>Teams often find out what their agent costs from the invoice, and the number per question is far higher than anyone guessed. It is rarely the model&#x27;s price. It is the harness around the model resending everything, looping, retrying and never measuring. Here are the nine mistakes behind the bill, a worked example of how they add up, and the questions to check your own agent.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Build only the gaps: AgentTwin assembled from open-source tools</title><link>https://cleandataengineering.ai/writing/agenttwin-assembled/</link><guid>https://cleandataengineering.ai/writing/agenttwin-assembled/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>AgentTwin is an assembler. Where a proven open-source tool already does a job, it plugs that tool in behind a port and builds nothing. It builds only four things no tool did: the world description format, the stand-ins that project that world into the agent&#x27;s systems, the judges that read the world&#x27;s state, and the switch between pretend and real. Of 38 tools we looked at, 12 are in use, 6 were turned down, and every decision has a written reason.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>If you can&#x27;t replay it, you can&#x27;t fix it: tracing autonomous agents</title><link>https://cleandataengineering.ai/writing/tracing-agents/</link><guid>https://cleandataengineering.ai/writing/tracing-agents/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>When an agent gives a bad answer, the final reply tells you almost nothing. You need a trace: one per turn, with a span for every model call, tool call and check, carrying the model, a prompt version, tokens, cost, time and the decision, but not the customer&#x27;s personal data. With that, yesterday&#x27;s bad conversation becomes something you can step through and turn into a test.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Agent memory without the magic</title><link>https://cleandataengineering.ai/writing/agent-memory/</link><guid>https://cleandataengineering.ai/writing/agent-memory/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>An agent&#x27;s memory is not a mysterious brain. It is three plain things: this conversation saved in a checkpoint, a few facts about a person kept across visits, and documents it looks up. This post shows what to store, what to forget, where it lives, and why anything that changes should be read again from the real system instead of remembered.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>MCP is here to stay: give your agent tools the standard way</title><link>https://cleandataengineering.ai/writing/mcp-here-to-stay/</link><guid>https://cleandataengineering.ai/writing/mcp-here-to-stay/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>The Model Context Protocol (MCP) is a standard way for an agent to find and call tools that a separate server offers. It has settled because many agents and tools already speak it, so a tool you write once works in all of them. Wrap your systems as MCP servers, keep the business rules inside the server, and secure it like any other door into your data. For one agent with three small functions, skip it.</description>
<category>AI Engineering</category><category>Builders</category></item><item><title>Context engineering: what goes into the window, and what stays out</title><link>https://cleandataengineering.ai/writing/context-engineering/</link><guid>https://cleandataengineering.ai/writing/context-engineering/</guid>
<pubDate>Sat, 31 Oct 2026 08:00:00 +0530</pubDate><description>On every step, an agent&#x27;s model reads one bundle of text, called the context. What you put in it, in what order, and what you leave out decides how well the agent works and what it costs. This post shows five habits, using a real claims assistant: stable things first, fence what comes back from tools, trim, show only the allowed tools, and pass references instead of whole documents.</description>
<category>AI Engineering</category><category>Builders</category></item></channel></rss>