LearnLLMStack
Build a tiny version of every LLM layer, from your own model up to the agent, and connect them.
1 · Purpose
LearnLLMStack builds a small, slow version of each layer between an agent's question and a model's token. It starts at the bottom, by training a tiny language model on a small corpus, then adds the engine, server, provider, gateway, harness, tool server and tracing on top. It is for engineers who use LLMs every day but have never seen what happens below the API. The connections between the layers are the real lesson, so one request should finally travel through code you wrote at every step.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Write the table-driven round-trip test cases for the step 2 tokenizer.
- Draw the token-count comparison table: GPT-2 tokenizer versus ours on the same sentences.
- Write a load script that sends 1, 4 and 8 requests at once and records time to first token.
4 · Requirements
What it must do
- Train a language model from scratch in seven steps: counting, tokenizer, transformer, check against published weights, pretraining, chat tuning and tool-call tuning.
- Run the model through a hand-written generation loop with sampling and a key-value cache.
- Serve the model over HTTP in the common chat-completions format, with streaming and a request queue.
- Put a provider layer (keys, rate limits, billing) and a separate gateway layer (guardrails, budgets, routing) in front of the server.
- Connect an existing agent harness and a hand-built tool server, so the model can ask for tools and the harness runs them.
- Trace one request across every layer.
- Teach each step with a picture-book design page, step-by-step walkthrough pages and a fine-tuning learning path.
How well it must do it
- Runs on a laptop: models up to about 124 million parameters, no containers needed.
- Each layer starts with one command and has its own README.
- Each layer has a 'done when' check with a number, such as held-out loss or tokens per second.
- Tests are table-driven.
- Free resources only for heavier work, such as free notebook GPUs for fine-tuning.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS |
|---|---|---|---|
| Own model and engine | Plain Python (step 1 built), then PyTorch on the laptop GPU (next) | Azure Machine Learning training | SageMaker training |
| Inference server | FastAPI with server-sent events | Azure Machine Learning online endpoint | SageMaker endpoint |
| Provider | FastAPI with a small key and usage store | Azure AI Foundry | Amazon Bedrock |
| Gateway | Hand-written, compared with LiteLLM | Azure API Management AI gateway | API Gateway with Bedrock Guardrails |
| Agent harness | Reused from an existing reference agent | Azure Container Apps | ECS Fargate |
| Tool server | JSON-RPC over stdio by hand, then FastMCP | Azure Functions | AWS Lambda |
| Tracing | OpenTelemetry | Azure Monitor / Application Insights | AWS X-Ray / CloudWatch |
7 · What needs to be done
Design before code 3 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Picture-book design page for every layer | done | M | technical writing |
| Decisions, glossary and reuse ledger | done | S | architecture |
| Fine-tuning learning path, nine phases | done | M | ML |
Layer 1: your own model 2 of 5 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Step 1: character bigram model on a children's story corpus | done | S | Python |
| Whole-LLM-on-one-sentence toy and walkthrough | done | S | ML teaching |
| Step 2: hand-written byte-pair tokenizer with round-trip tests, including Hindi and emoji | todo | M | Python |
| Step 3: small GPT-style transformer | todo | L | PyTorch |
| Step 4: load published GPT-2 weights and match outputs token for token | todo | M | PyTorch |
Fine-tuning path 2 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Phase 0: setup lesson and notebook | done | S | ML |
| 'Should you fine-tune at all?' lesson | done | S | technical writing |
| Phase 2: first fine-tune with a chat template, loss masking and LoRA | doing | M | ML |
Layers 2 to 7 0 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Inference server with streaming and a load test table | todo | M | Python / HTTP |
| Provider and gateway as separate layers | todo | M | backend |
| Hand-built tool server, then compare with a library | todo | S | protocols |
| Trace one request across all layers | todo | M | observability |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| ML learner-builder | 1 | 4-6 | Works through the model steps and fine-tuning phases and records results. |
| Backend engineer | 1 | 2-4 | Builds the server, provider and gateway layers in the common API format. |
| Reviewer / explainer | 1 | 1-2 | Checks the pictures and lessons are correct and plain. |
9 · What you'll learn
- What a model actually learns, and why the tokenizer changes cost, especially for Hindi.
- Why prefill is cheaper per token than decode, and why the KV cache grows with the conversation.
- What temperature and top-p really change.
- Why a base model cannot follow instructions until it is chat-tuned.
- Where guardrails belong: in the gateway owned by the user, not the provider.
Level-up questions
- How small can a model be and still teach every layer honestly?
- Should the tokenizer be learned only from practice data, or from your own corpus?
- What does batching gain in throughput, and what does it cost each request in waiting time?
- Which friction points in fine-tuning could become a reusable tool?
10 · Links
- Links come once the work is public.