Spark Cost Agent

A Spark cost analyst agent built as a lab for agent tracing, evaluation and cost per successful task.

Active · join meAI Engineering

6 of 10 tasks done

1 · Purpose

Spark Cost Agent is a real agent that reads Delta transaction logs and Spark event logs and finds cost anti-patterns such as small files, skew and cartesian joins. The agent is not the point: everything around it is, meaning traces, metrics, evaluations, guardrails and a cost model. It is for engineers who need to run agents in production and prove they work and what they cost. It follows a ten-stage plan and builds the observability by hand so every signal is understood.

For: AI and platform engineers responsible for running, evaluating and costing agents.

2 · Business benefits

Know what each answer costsCost per successful task shows whether an agent is worth its spend, not just total tokens.
Catch quality drops earlyOnline evaluation, guardrails and alerts flag bad answers before users report them.
No vendor lock-inAll telemetry goes through one neutral collector, so backends can be added or swapped without code changes.
Safer releasesAn offline evaluation gate with a pass-rate threshold stops a worse version from merging.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Label twenty agent answers by hand for the judge check.
  • Add one new test question to the evaluation dataset.
  • Write an alert rule for a rising tool-error rate.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Domain toolsRead Delta logs and Spark eventlogs and return findings.Agent loopCalls the model, runs tools andassembles the answer.Telemetry emitterWrites spans and metrics withstandard names.Telemetry seamReceives all telemetry,redacts, samples and fans outTrace and metric backendsStore and display traces,metrics and scores.Cost modelPrices token usage and computescost per successful task.Evaluation suiteOffline datasets, pass-rategate, pairwise preference andOnline guardGuardrails, sampled judging,feedback and alerts on livetask, model anspans and metrredacted, sampled
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Agent loop → Domain tools: tool call · Cost model → Trace and metric backends: shadow cost · Evaluation suite → Agent loop: test questions · Online guard → Agent loop: checks on live answers

6 · Tech stack, per platform

ComponentOpen stackAzureAWS
Domain toolsPython over Delta and Spark JSON logs ✓——
Agent loopHand-written loop, OpenAI-compatible client on Groq ✓Azure AI Foundry modelsAmazon Bedrock models
Telemetry emitterManual OpenTelemetry, GenAI conventions ✓——
Telemetry seamOpenTelemetry Collector ✓OpenTelemetry Collector on Azure Container AppsAWS Distro for OpenTelemetry
Trace and metric backendsArize Phoenix + Prometheus, Langfuse and LangSmith free tiers ✓Azure Monitor / Application InsightsAWS X-Ray / CloudWatch
Cost modelPython rate cards ✓——
Evaluation suitepytest-style runner + promptfoo ✓Azure AI Foundry evaluationsBedrock model evaluation
Online guardPython guardrails and sampled judge ✓Azure AI Content SafetyBedrock Guardrails

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Tracing and the seam (stages 1 to 2) 2 of 3 done

TaskStatusSizeSkill
First real trace on screendoneSOpenTelemetry
Collector as the neutral seam with redaction and samplingdoneMOpenTelemetry
Add a hosted system of record as a second exporterdoingSobservability

Cost and metrics (stages 4 to 5) 2 of 2 done

TaskStatusSizeSkill
Shadow pricing and cost per successful taskdoneMFinOps
Deterministic metrics and feedback capturedoneSPython

Evaluation (stages 6 to 9) 2 of 4 done

TaskStatusSizeSkill
Labelled dataset from artifacts and incidentsdoneMevaluation
Online guardrails, sampled judge and alertsdoneMevaluation
Offline evaluation as a merge gate in CIdoingMCI
Validate the judge against human labelsdoingMstatistics

Optimise (stage 10) 0 of 1 done

TaskStatusSizeSkill
Caching, model routing and context trimming against cost per successtodoLLLM engineering

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
AI engineer14-6Builds the next stage and keeps the plan honest.
Observability engineer12-3Owns the collector, backends and alerts.
Evaluator11-2Labels answers by hand so the judge can be checked.

9 · What you'll learn

Level-up questions

10 · Links