LearnLLMStack

Build a tiny version of every LLM layer, from your own model up to the agent, and connect them.

Active · join meLLM Engineering

7 of 15 tasks done

1 · Purpose

LearnLLMStack builds a small, slow version of each layer between an agent's question and a model's token. It starts at the bottom, by training a tiny language model on a small corpus, then adds the engine, server, provider, gateway, harness, tool server and tracing on top. It is for engineers who use LLMs every day but have never seen what happens below the API. The connections between the layers are the real lesson, so one request should finally travel through code you wrote at every step.

For: Software and data engineers who call LLM APIs and want to understand the provider side: training, inference, hosting and routing.

2 · Business benefits

Fewer cost surprisesEngineers who know why reading a prompt is cheaper than writing an answer make better choices about context size and caching.
Better vendor decisionsA team that has built a gateway and a provider can tell which controls belong to them and which belong to the host.
Realistic fine-tuning plansLearners find out when fine-tuning is worth it and when a prompt or retrieval is enough, before paying for GPU time.
Skills that transferEvery layer speaks the common chat-completions format, so each home-made part can be swapped for a real product with one setting.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Write the table-driven round-trip test cases for the step 2 tokenizer.
  • Draw the token-count comparison table: GPT-2 tokenizer versus ours on the same sentences.
  • Write a load script that sends 1, 4 and 8 requests at once and records time to first token.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Own model and engineHolds the trained weights andruns the predict-pick-appendInference serverTurns the engine into a networkservice with streaming,ProviderPlays the hosting company: APIkeys, rate limits, usage andGatewayPlays the company using themodel: guardrails, budgets andAgent harnessRuns the conversation loop andexecutes tool calls the modelTool serverExposes tools such as files ora database over a simpleTracingFollows one request throughevery layer and records timetokenstool call
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Agent harness → Gateway: chat request · Gateway → Provider: checked request · Provider → Inference server: authorised request · Inference server → Own model and engine: batched prompts · Tracing → Agent harness: spans from every layer

6 · Tech stack, per platform

ComponentOpen stackAzureAWS
Own model and enginePlain Python (step 1 built), then PyTorch on the laptop GPU (next)Azure Machine Learning trainingSageMaker training
Inference serverFastAPI with server-sent eventsAzure Machine Learning online endpointSageMaker endpoint
ProviderFastAPI with a small key and usage storeAzure AI FoundryAmazon Bedrock
GatewayHand-written, compared with LiteLLMAzure API Management AI gatewayAPI Gateway with Bedrock Guardrails
Agent harnessReused from an existing reference agentAzure Container AppsECS Fargate
Tool serverJSON-RPC over stdio by hand, then FastMCPAzure FunctionsAWS Lambda
TracingOpenTelemetryAzure Monitor / Application InsightsAWS X-Ray / CloudWatch

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Design before code 3 of 3 done

TaskStatusSizeSkill
Picture-book design page for every layerdoneMtechnical writing
Decisions, glossary and reuse ledgerdoneSarchitecture
Fine-tuning learning path, nine phasesdoneMML

Layer 1: your own model 2 of 5 done

TaskStatusSizeSkill
Step 1: character bigram model on a children's story corpusdoneSPython
Whole-LLM-on-one-sentence toy and walkthroughdoneSML teaching
Step 2: hand-written byte-pair tokenizer with round-trip tests, including Hindi and emojitodoMPython
Step 3: small GPT-style transformertodoLPyTorch
Step 4: load published GPT-2 weights and match outputs token for tokentodoMPyTorch

Fine-tuning path 2 of 3 done

TaskStatusSizeSkill
Phase 0: setup lesson and notebookdoneSML
'Should you fine-tune at all?' lessondoneStechnical writing
Phase 2: first fine-tune with a chat template, loss masking and LoRAdoingMML

Layers 2 to 7 0 of 4 done

TaskStatusSizeSkill
Inference server with streaming and a load test tabletodoMPython / HTTP
Provider and gateway as separate layerstodoMbackend
Hand-built tool server, then compare with a librarytodoSprotocols
Trace one request across all layerstodoMobservability

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
ML learner-builder14-6Works through the model steps and fine-tuning phases and records results.
Backend engineer12-4Builds the server, provider and gateway layers in the common API format.
Reviewer / explainer11-2Checks the pictures and lessons are correct and plain.

9 · What you'll learn

Level-up questions

10 · Links