Spark Cost Agent
A Spark cost analyst agent built as a lab for agent tracing, evaluation and cost per successful task.
1 · Purpose
Spark Cost Agent is a real agent that reads Delta transaction logs and Spark event logs and finds cost anti-patterns such as small files, skew and cartesian joins. The agent is not the point: everything around it is, meaning traces, metrics, evaluations, guardrails and a cost model. It is for engineers who need to run agents in production and prove they work and what they cost. It follows a ten-stage plan and builds the observability by hand so every signal is understood.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Label twenty agent answers by hand for the judge check.
- Add one new test question to the evaluation dataset.
- Write an alert rule for a rising tool-error rate.
4 · Requirements
What it must do
- Answer questions about Spark and Delta cost from real logs, using four domain tools.
- Emit a trace for every task, model call and tool call, using the standard generative-AI attribute names.
- Price real token counts against a reference rate card, so cost stays meaningful on a free tier.
- Capture user feedback and cheap deterministic metrics on every run.
- Build a labelled evaluation dataset from the artifacts and from past incidents.
- Run online checks: guardrails, a sampled model judge, injection detection and quality alerts.
- Report coverage against an open AI assurance catalog.
How well it must do it
- No agent framework: the loop is hand-written so every span is owned code.
- Works without an API key for everything except asking the agent.
- Offline test suite runs with no network.
- Light enough for an 8 GB laptop; hosted free tiers used where local tools are too heavy.
- Model provider switch is a configuration change, never a code change.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS |
|---|---|---|---|
| Domain tools | Python over Delta and Spark JSON logs ✓ | — | — |
| Agent loop | Hand-written loop, OpenAI-compatible client on Groq ✓ | Azure AI Foundry models | Amazon Bedrock models |
| Telemetry emitter | Manual OpenTelemetry, GenAI conventions ✓ | — | — |
| Telemetry seam | OpenTelemetry Collector ✓ | OpenTelemetry Collector on Azure Container Apps | AWS Distro for OpenTelemetry |
| Trace and metric backends | Arize Phoenix + Prometheus, Langfuse and LangSmith free tiers ✓ | Azure Monitor / Application Insights | AWS X-Ray / CloudWatch |
| Cost model | Python rate cards ✓ | — | — |
| Evaluation suite | pytest-style runner + promptfoo ✓ | Azure AI Foundry evaluations | Bedrock model evaluation |
| Online guard | Python guardrails and sampled judge ✓ | Azure AI Content Safety | Bedrock Guardrails |
7 · What needs to be done
Tracing and the seam (stages 1 to 2) 2 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| First real trace on screen | done | S | OpenTelemetry |
| Collector as the neutral seam with redaction and sampling | done | M | OpenTelemetry |
| Add a hosted system of record as a second exporter | doing | S | observability |
Cost and metrics (stages 4 to 5) 2 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Shadow pricing and cost per successful task | done | M | FinOps |
| Deterministic metrics and feedback capture | done | S | Python |
Evaluation (stages 6 to 9) 2 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Labelled dataset from artifacts and incidents | done | M | evaluation |
| Online guardrails, sampled judge and alerts | done | M | evaluation |
| Offline evaluation as a merge gate in CI | doing | M | CI |
| Validate the judge against human labels | doing | M | statistics |
Optimise (stage 10) 0 of 1 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Caching, model routing and context trimming against cost per success | todo | L | LLM engineering |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| AI engineer | 1 | 4-6 | Builds the next stage and keeps the plan honest. |
| Observability engineer | 1 | 2-3 | Owns the collector, backends and alerts. |
| Evaluator | 1 | 1-2 | Labels answers by hand so the judge can be checked. |
9 · What you'll learn
- How to instrument an agent by hand with standard trace names.
- Why a collector between app and backends keeps you vendor-neutral.
- How to compute cost per successful task, not just cost per call.
- Offline versus online evaluation, and when a model judge can be trusted.
- Why counting only added files in a Delta log gives the wrong answer.
Level-up questions
- How much agreement with humans does a judge need before you trust it?
- What sampling rate catches quality drops without paying for every check?
- Which optimisation cuts cost per success most: caching, routing or trimming?
10 · Links
- Links come once the work is public.