Home › Paths › AI Engineering › Open source
Learning path · Open source
AI Engineering on Open source
You build products and agents where a model does part of the work, safely.
Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.
Most of an AI system is ordinary software around a small model call: the tools, the checks, the state, the evaluation and the bill. AI engineers build that harness so the product holds up after the demo.
Every topic below is a concept (the job) with the Open source product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: The AI Universe · Agent Harness, drawn simply · The Agent Harness Series · Eval Harness, drawn simply · Vertical agents, by industry · Certification prep on this site.
Beginner · 6 weeks
Crawl: You can build one thing end to end, with help.
Call a model, get checked structured output, and give it a few safe tools.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Open source you use |
|---|---|---|
| Python and APIs | HTTP, JSON, async, tests; read an API's docs | Pythona language, properly Dockerrun an app |
| Calling a model | messages, tokens, temperature, cost per call | Ollama, or a free hosted open modelmodel api |
| Structured output | ask for JSON, validate it, retry on failure | Pydantic + JSON modeprompts and structured output |
| Tools | let the model ask your code to act; check before acting | plain Python loop, no frameworktool calling |
| Deterministic first | if code can do it, code does it; the model handles the rest | Pythona language, properly |
Build on GitHub
Agent harness from scratch
Build the loop, tools, context and checks around a model in plain Python, one small step at a time.
Built on Open source already.
Your first agent, three tools
A support assistant that can look up an order, check a policy and draft a reply, where code checks every tool call first.
You are ready to move on when
- You can say what each call cost.
- A wrong tool call is refused by code, not by hope.
- You have ten written test cases and they pass.
Intermediate · 3 months
Walk: You can build it so it keeps working, without help.
Build the harness: retrieval, state, approvals, tracing and evaluation on every change.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Open source you use |
|---|---|---|
| Retrieval | search by meaning; chunking; citations | pgvector or Qdrantvector search |
| Agent loop and state | steps, budgets, checkpoints, waiting for a human | DBOS or Temporal + your loopagent runtime |
| Checks and guardrails | before and after every model and tool call | rule checks in your own codeguardrails |
| Tracing | every call, token and cent visible | Phoenix or Langfusetracing |
| Evaluation | golden cases, rules, a gate in CI | Eval Harness Framework or promptfooevaluation GitHub Actionsci/cd |
Build on GitHub
Claims assistant on Azure
A motor-insurance claims agent running for real: tools, checks, approvals, durable waits, evaluation and cost.
Built on Azure today. Porting it to Open source is a good first contribution.
LearnLangGraph
A hands-on course in graph-shaped agents: state, branches, human approval and resuming after a crash.
Built on Open source already.
Reference agent
A customer-support agent with no framework: one module per harness layer, the architecture enforced by tests.
Built on Open source already.
You are ready to move on when
- A crash mid-conversation resumes without repeating a payment.
- A release is blocked when evaluation scores drop.
- You can open one trace and explain the whole turn.
Advanced · 6 months +
Run: You can design it, prove it and lead others through it.
Prove an agent before it meets real people, and run many of them at a known cost.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Open source you use |
|---|---|---|
| Simulated worlds | test against simulated customers, systems and faults | Eval Harness Framework or promptfooevaluation |
| Gateway and budgets | many models behind one door; fallbacks; spend limits | LiteLLMmodel gateway |
| Assurance | what must be true before it is trusted; evidence, not claims | Eval Harness Framework or promptfooevaluation rule checks in your own codeguardrails |
| Portability | same agent, new cloud, by swapping adapters | Docker Composeinfrastructure as code DBOS or Temporal + your loopagent runtime |
| Research | long-running agents, self-repair, drift | Phoenix or Langfusetracing |
Build on GitHub
Eval Harness Framework
Rebuild each conversation from the agent's own traces and check it against rules written in plain YAML.
Built on Open source already.
AgentTwin
Simulate the agent's world (customers, systems, faults, time) so a real agent can be tested before it meets real people.
Built on Open source already.
Port the agent to your cloud
Take the reference agent and bind it to AWS or Google Cloud: same code, new adapters, same tests passing.
AgentTwin Lab
Research: an agent runs for weeks in a simulated business while a fixer repairs it. Does the fixer cheat?
Built on Open source already.
You are ready to move on when
- Your agent has passed a simulated week before going live.
- You can move it to another cloud and the same tests pass.
- You have published what you learned.
Optional
Certifications on Open source
Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.
- Claude Certified Architect – Foundations
Free prep we have written: Certifications.
Learn together
Do it with others
Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.