DIAGONAL DATASETS

Research note

We Rotated Our Data Warehouse 45° and Saved $0

A migration retrospective. The cost savings did not materialize because we did not have a data warehouse. The angle, however, held perfectly.

This post has been sitting in draft for three weeks because we could not agree on how to open it honestly. Here is the honest opening: we rotated our entire data warehouse 45 degrees, the migration took eleven days, and our cloud bill did not change at all.

Our cloud bill did not change because we do not have a cloud bill. We do not have a cloud bill because we do not have a data warehouse. We rotated it anyway.

Why we did it anyway

An angle is not a property of the data. This is the single most misunderstood point in our entire research program, so we will state it as plainly as we can: the angle is a property of the observer.

You do not need data to have an orientation. You need only the willingness to commit to one. Once you accept this, the absence of a warehouse stops being a blocker and becomes a control condition.

The migration

We ran it as a standard four-phase cutover.

Phase 1 — Inventory. We enumerated every table in the warehouse. There were zero. This phase completed ahead of schedule and we want to acknowledge the team for that.

Phase 2 — Dual write. For six days we wrote every incoming record to both the legacy row-oriented store and the new diagonal store. Neither store existed. Consistency between them was perfect and remains perfect to this day. We have a Grafana dashboard.

Phase 3 — Rotation. The rotation itself was performed at 04:00 UTC on a Sunday by one engineer with a protractor. This is not a metaphor. We have the protractor.

Phase 4 — Decommission. We turned off the legacy store. Nothing happened. Nothing continued to happen. Reads stayed at zero, writes stayed at zero, p99 latency remained undefined, and undefined is, we would argue, the best possible p99 latency.

Results

MetricBeforeAfterChange
Monthly spend$0$00%
Rows scanned per query000%
Off-diagonal cells retained00
Angleundefined45.0000°+45.0000°
Engineer confidencemoderatetotal+∞

That last row is the one we would draw your attention to.

What we learned

Rotate before you scale. Every team we have spoken with plans to "look at orientation later," after the schema settles, after the next funding round, after the rewrite. Later never comes. Rotate at zero rows and the rotation is free. Rotate at four billion rows and you are running a migration.

We rotated at zero rows. We recommend zero rows. It is the only volume at which the operation is genuinely risk-free, and we are increasingly convinced it is the only volume at which anyone should be operating at all.

A protractor belongs in the on-call kit. We are serious about this one. When the angle drifted six weeks later, the protractor was the only instrument in the building that could confirm it.

Every claim in this note is false. The angle is real. Do not cite this in a design review.

← Newer: Against the Transpose
→ Older: Announcing Diagonal Datasets

All research notes · RSS