Ontology and Knowledge Graph

Build an ontology and knowledge graph from scratch over code, data, specs and AI assets.

Active · join meData Engineering

4 of 9 tasks done

1 · Purpose

This project builds an ontology and a knowledge graph by hand, over a codebase and the data platform it touches. It parses Python, PySpark, SQL, databases and spec documents into one shared vocabulary, then reasons over it and checks rules. It is for engineers who want to understand how such a graph is designed, rather than learn one vendor's model. The payoff is impact analysis in one query: change a function and see every table, screen and team affected.

For: Data engineers, platform engineers and architects interested in lineage, metadata and agent context.

2 · Business benefits

Safer changesImpact analysis shows every downstream table, screen and team before a change ships.
Find hidden relationshipsValue-overlap mining finds joins nobody declared, which documentation misses.
Lower compliance riskRules flag personal data with no owner or unmasked samples.
Better agent answersA graph gives a coding agent precise context instead of reading the whole repository.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Write one planned lesson, such as the RDF crash course.
  • Add a competency question and the query that answers it.
  • Write the git history extractor for code churn.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

SourcesSample repository, samplewarehouse and spec documents.ExtractorsTurn code, SQL, databases andspecs into graph statements.VocabularyOne core model plus code, data,AI and spec lenses.Graph storeHolds all statements withprovenance.ReasonerAdds inferred edges such astransitive lineage.Rule checkerValidates what must be true,such as owned personal data.Question layerAnswers competency questions,later as task-shaped agentfiles and scheshared termsfactsinferred facts
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Extractors → Graph store: statements · Graph store → Question layer: queries

6 · Tech stack, per platform

ComponentOpen stackAzureAWSDatabricks
SourcesSQLite stand-in warehouse + sample repo ✓Microsoft Purview scansAWS Glue Data CatalogUnity Catalog lineage system tables
ExtractorsPython ast, sqlglot, SQLAlchemy, markdown parsing ✓———
VocabularyRDF / OWL in Turtle files ✓———
Graph storerdflib ✓Azure Cosmos DB for Apache GremlinAmazon Neptune—
Reasonerowlrl ✓———
Rule checkerpyshacl ✓———
Question layerSPARQL ✓; MCP tools planned———

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Phase 1: build from scratch 4 of 7 done

TaskStatusSizeSkill
Core ontology and four lensesdoneMontology design
Python, SQL, database and spec extractorsdoneLPython
Reasoning, rule checks and question queriesdoneMRDF / SPARQL
Lessons 00, 05, 10 and 11doneMtechnical writing
TypeScript and web-app extractortodoMTypeScript parsing
Git history extractor for code churntodoSPython
Incremental re-extraction with named graphstodoMRDF

Phase 2: compare with existing tools 0 of 1 done

TaskStatusSizeSkill
Point an open metadata platform at the same repo and compare modelstodoLdata governance

Phase 3: agent interface 0 of 1 done

TaskStatusSizeSkill
Expose task-shaped tools over the tool protocoltodoMagents

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Ontology designer12-3Owns the vocabulary and competency questions.
Extractor developer12-4Writes new extractors for more languages and sources.
Lesson writer11-2Turns planned lessons into plain prose.

9 · What you'll learn

Level-up questions

10 · Links