cleandataengineering.aiData Platform, from the big picture to the code
You are here: Concerns

Data PlatformZoom level 0

Every concern, one page each

A platform is more than its layers. Some jobs cut across all of them: checking quality, keeping history, controlling access, recording what happened. We call these concerns and number them X1 to X24. Each concern gets one page, and every page has the same shape, so you learn the shape once.

The map

RunX1 MetadataX2 OrchestrationX15 ReplayX16 CI/CDX17 TestingX18 ResilienceX23 LayoutX24 StreamingTrustX3 ContractsOpen: X4 Data qualityX4 Data qualityopen ›X5 ProfilingX6 ReconciliationSeeX7 LineageX8 ObservabilityX9 FreshnessX10 CostX19 PerformanceGuardX11 AccessX12 PrivacyX13 GovernanceX14 AuditSelfX20 Self-healingX21 AgentsX22 Simulation
Solid boxes are the five families. Dashed boxes have no page yet. Boxes with a tab open their page.

How to read a concern page

  1. Why. One plain paragraph on what goes wrong without this concern.
  2. The picture. Where the concern happens along the data's path. Every box opens the page for that part.
  3. The places. A table: each place the concern is checked, what is checked, and what a failure does.
  4. Request and response. For each major function: what goes in, what comes out, and who calls it.
  5. One real run. Inputs and outputs printed by the code on a small example. Nothing here is typed by hand.
  6. Code and tests. The code cut from the real file, the settings that drive it, the tests, and what is not built.

All 24

Family#ConcernRules proven by a testPage
RunX1Metadata3 of 7 rules testedPage to come
RunX2Orchestration3 of 3 rules testedPage to come
RunX15Replay13 of 15 rules testedPage to come
RunX16CI/CD0 of 6 rules testedPage to come
RunX17Testing5 of 5 rules testedPage to come
RunX18Resilience2 of 4 rules testedPage to come
RunX23Layoutno rules in the catalogPage to come
RunX24Streamingno rules in the catalogPage to come
TrustX3Contracts4 of 10 rules testedPage to come
TrustX4Data quality8 of 12 rules testedOpen the page
TrustX5Profiling0 of 1 rules testedPage to come
TrustX6Reconciliation6 of 7 rules testedPage to come
SeeX7Lineage1 of 2 rules testedPage to come
SeeX8Observability2 of 2 rules testedPage to come
SeeX9Freshness2 of 5 rules testedPage to come
SeeX10Cost0 of 3 rules testedPage to come
SeeX19Performance1 of 2 rules testedPage to come
GuardX11Access3 of 4 rules testedPage to come
GuardX12Privacy2 of 3 rules testedPage to come
GuardX13Governance3 of 6 rules testedPage to come
GuardX14Audit3 of 5 rules testedPage to come
SelfX20Self-healing0 of 1 rules testedPage to come
SelfX21Agents0 of 1 rules testedPage to come
SelfX22Simulation1 of 1 rules testedPage to come

Look inside