AskSherlock

An open-source Genie-style chat over a ClickHouse cost store, where the model never writes SQL.

Starting · join earlyCostForward-deployed ready

2 of 10 tasks done

1 · Purpose

AskSherlock lets a person ask a cost question in plain English and get the number, a chart and the SQL that produced it. It works over a ClickHouse cost store, as an open-source alternative to Databricks AI/BI Genie. The model only turns the question into a small validated intent from a fixed vocabulary; a semantic layer writes the SQL. It runs on a two-week time box, and the public write-up of what worked is the main deliverable.

For: Data and platform teams who want trustworthy natural-language questions over their own warehouse.

2 · Business benefits

Answers match the dashboardChat and dashboards share one set of metric definitions, so they cannot disagree.
Fewer confident wrong answersThe model cannot invent a metric or a join, which removes a whole class of errors.
Auditable resultsThe SQL is shown with every answer, so anyone can check it.
Low running costUses open-source parts and a free open-weight model on data already in the warehouse.

3 · Status and how to join

Starting · join early. Early days: the shape is still being decided, so joiners now have the most say. Lead: Basant Choudhary.

Good first tasks this week:

  • Write ten gold questions with their expected intent objects.
  • Draft the JSON Schema for the intent object.
  • Write the value dictionary generator for one dimension.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Question inputTakes the user's question.Intent generatorMaps the question to astructured intent using a smallIntent validatorChecks the intent against theschema and the vocabulary.Value resolverMaps user words to realdimension values.Semantic layerHolds metric, dimension andjoin definitions and compilesCost storeThe warehouse that runs thequery.Answer viewShows the number, the chart andthe SQL.Evaluation suiteGold questions with expectedintents and answers.questionintent objectvalid intentresolved intentcompiled SQLresult rows
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Evaluation suite → Intent generator: gold questions

6 · Tech stack, per platform

ComponentOpen stackDatabricksAzureAWS
Question inputWeb panel in an existing Next.js app———
Intent generatorOpen-weight model on Groq—Azure AI Foundry modelsAmazon Bedrock models
Intent validatorJSON Schema———
Value resolverGenerated YAML value dictionaries———
Semantic layerCube Core or a small hand-written compiler, authored in OSI shapeUnity Catalog Metric Views——
Cost storeClickHouse—ClickHouse on an Azure VMClickHouse on EC2
Answer viewNext.js chartsAI/BI Genie (the reference being replaced)——
Evaluation suiteTable-driven tests + Langfuse———

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Research and decisions 2 of 2 done

TaskStatusSizeSkill
Open-source landscape and architecturedoneMresearch
Six decision records: intent not SQL, no graph in v1, value dictionaries, model formatdoneMarchitecture

Phase 0 to 1: questions and semantic layer 0 of 4 done

TaskStatusSizeSkill
Seed gold question setdoingSanalytics
Model measures and dimensions over the main cost tabletodoMsemantic modelling
One-day spike: semantic engine versus hand-written compilertodoMSQL
Value dictionary generatortodoSPython / SQL

Phase 2 to 3: intents and evaluation 0 of 2 done

TaskStatusSizeSkill
Intent schema, validator, model call and refusal pathtodoMLLM engineering
Evaluation harness of about 120 cases in CItodoMevaluation

Phase 4 to 5: UI and write-up 0 of 2 done

TaskStatusSizeSkill
Answer panel with chart and SQLtodoMfrontend
Public write-up of resultstodoMtechnical writing

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Semantic modeller13-4Defines metrics, dimensions and joins.
LLM engineer12-3Builds intent generation, validation and refusal.
Evaluator11-2Writes gold questions and checks answers.

9 · What you'll learn

Level-up questions

10 · Links