From warehouse to data contracts

Fri, 02 Oct 2026 00:00:00 +0000

From warehouse to data contracts Link to heading

A new approach to our data platform

~10 minutes · then questions

Note: [0:15] Set the frame. One idea for the whole talk: the way we get data in doesn’t scale with a central team, so we’re changing who does the work and giving them a pattern to follow.

A very short history Link to heading

Data warehouse - structured, central, trusted. But one team models everything, and every new source waits in their queue.
Data lake - store anything, cheaply. But no schema, no transactions: the “data swamp”.
Lakehouse - open table formats (Iceberg, Delta, Hudi) bring warehouse guarantees to cheap lake storage.
Data mesh and data contracts - owners publish their own data under an agreed contract.

Note: [1:00] Each step fixed one problem and left another. The last one is about who does the work, and that’s where this talk lands. Dates if asked: Iceberg started at Netflix in 2017 (Apache top-level 2020). Data mesh article 2019. “Lakehouse” popularised 2020-21 (CIDR 2021 paper). Data contracts take shape from 2021; Open Data Contract Standard 2023. Mesh is about who owns data. Lakehouse is about where it lives. Contracts are how owners agree on it. They’re complementary, not successive.

Slides on jayeffsee

From warehouse to data contracts

From warehouse to data contracts Link to heading

A very short history Link to heading