Conversation UI on Data

A specification for chat-with-your-data where the model picks certified metrics and never writes SQL.

Active · join meData Engineering

5 of 8 tasks done

1 · Purpose

This is a specification, not an implementation: the right architecture for asking questions of data in plain English. It covers two settings, data governed in Unity Catalog on Databricks, and data in PostgreSQL or ClickHouse on a virtual machine. Its core rule is that the model selects a certified metric and dimensions, and a deterministic engine writes the query. It is for teams planning a conversational analytics product who want answers that match their dashboards.

For: Data architects, analytics engineers and product teams planning natural-language analytics.

2 · Business benefits

Trustworthy numbersChat and dashboards read the same semantic layer, so they always give the same answer.
Less reworkWriting the metric contract first stops gold tables being built in the wrong shape.
Clear tool choiceThe spec explains which semantic layer fits each setting and why.
Lower risk at launchShipping dashboards before chat exposes definition bugs to a forgiving audience first.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Write twenty real business questions as metric, dimensions and grain.
  • Review one track document and list any claim without a source.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Clean data layerConformed, tested data from thesource systems.Atomic gold modelStar-schema facts at the finestgrain any metric needs.Semantic layerCertified metrics, dimensions,joins and synonyms.Pre-aggregation cacheSpeeds up slow queries belowthe semantic layer.DashboardsFirst consumer of the semanticlayer.Conversation UIMaps questions to metrics andshows answers with provenance.Feedback loopTurns refusals and failuresinto backlog.conformed rowsfacts and dimerefusals and f
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Pre-aggregation cache → Semantic layer: fast results · Semantic layer → Dashboards: certified metrics · Semantic layer → Conversation UI: compiled queries

6 · Tech stack, per platform

ComponentDatabricksOpen stackAzureAWS
Clean data layerDelta tables in Unity Catalogdbt Core on PostgreSQL or ClickHouse——
Atomic gold modelDelta star schema in Unity Catalogdbt Core star schemaAzure Databricks or Microsoft FabricAmazon Redshift
Semantic layerUnity Catalog Metric ViewsCube CorePower BI semantic model—
Pre-aggregation cache—Cube pre-aggregations——
DashboardsAI/BI dashboardsApache Superset—Amazon QuickSight
Conversation UIAI/BI GenieOpen-weight model calling Cube over MCPFabric data agentAmazon Q in QuickSight

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Specification 5 of 5 done

TaskStatusSizeSkill
Requirements in testable termsdoneMrequirements
Reference architecture and four correctionsdoneMarchitecture
Databricks track and self-hosted trackdoneMdata platforms
Semantic layer options compareddoneManalytics engineering
Decisions, open questions and risksdoneSarchitecture

Prove it on real data 0 of 3 done

TaskStatusSizeSkill
Twenty real questions as metric, dimensions and graintodoManalytics
Golden question set with human-checked answerstodoMevaluation
Apply the spec in a build (see AskSherlock)doingLdata engineering

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Data architect12-3Owns the spec and its decisions.
Analytics engineer12-3Tests the spec against a real model and questions.
Reviewer11Challenges claims and checks sources.

9 · What you'll learn

Level-up questions

10 · Links