Ontology and Knowledge Graph
Build an ontology and knowledge graph from scratch over code, data, specs and AI assets.
1 · Purpose
This project builds an ontology and a knowledge graph by hand, over a codebase and the data platform it touches. It parses Python, PySpark, SQL, databases and spec documents into one shared vocabulary, then reasons over it and checks rules. It is for engineers who want to understand how such a graph is designed, rather than learn one vendor's model. The payoff is impact analysis in one query: change a function and see every table, screen and team affected.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Write one planned lesson, such as the RDF crash course.
- Add a competency question and the query that answers it.
- Write the git history extractor for code churn.
4 · Requirements
What it must do
- Parse Python and PySpark into functions, calls and table reads and writes.
- Parse SQL into table and column lineage, with the transforming expression.
- Reflect database schemas and profile columns with personal data masked.
- Find relationships from declared keys, observed joins and value overlap.
- Parse requirements, acceptance criteria, decision records and glossary terms from markdown.
- Reason transitive lineage and personal-data propagation, and validate rules.
- Answer a written list of competency questions with queries.
How well it must do it
- Pure Python with no containers, servers or paid APIs.
- Extractors are disposable; the vocabulary is durable and reviewed like code.
- Every class and property exists because a competency question needs it.
- One build command runs the whole demo end to end.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS | Databricks |
|---|---|---|---|---|
| Sources | SQLite stand-in warehouse + sample repo ✓ | Microsoft Purview scans | AWS Glue Data Catalog | Unity Catalog lineage system tables |
| Extractors | Python ast, sqlglot, SQLAlchemy, markdown parsing ✓ | — | — | — |
| Vocabulary | RDF / OWL in Turtle files ✓ | — | — | — |
| Graph store | rdflib ✓ | Azure Cosmos DB for Apache Gremlin | Amazon Neptune | — |
| Reasoner | owlrl ✓ | — | — | — |
| Rule checker | pyshacl ✓ | — | — | — |
| Question layer | SPARQL ✓; MCP tools planned | — | — | — |
7 · What needs to be done
Phase 1: build from scratch 4 of 7 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Core ontology and four lenses | done | M | ontology design |
| Python, SQL, database and spec extractors | done | L | Python |
| Reasoning, rule checks and question queries | done | M | RDF / SPARQL |
| Lessons 00, 05, 10 and 11 | done | M | technical writing |
| TypeScript and web-app extractor | todo | M | TypeScript parsing |
| Git history extractor for code churn | todo | S | Python |
| Incremental re-extraction with named graphs | todo | M | RDF |
Phase 2: compare with existing tools 0 of 1 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Point an open metadata platform at the same repo and compare models | todo | L | data governance |
Phase 3: agent interface 0 of 1 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Expose task-shaped tools over the tool protocol | todo | M | agents |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Ontology designer | 1 | 2-3 | Owns the vocabulary and competency questions. |
| Extractor developer | 1 | 2-4 | Writes new extractors for more languages and sources. |
| Lesson writer | 1 | 1-2 | Turns planned lessons into plain prose. |
9 · What you'll learn
- What an ontology is, and how it differs from a knowledge graph.
- How to decide between a class, a property and an instance.
- Why PySpark defeats simple static analysis and how to track it anyway.
- Reasoning versus validation, and the open-world assumption.
- How to expose a graph to an agent as tools, not a raw query endpoint.
Level-up questions
- Which questions justify a graph over a plain catalog?
- When should you stop materialising inferred edges?
- How does a hand-made model compare with an off-the-shelf metadata platform?
10 · Links
- Links come once the work is public.