Measurement · TPC-D45 suite
Benchmarks
Before the numbers, the disclosure, because burying it would make us the thing we are satirizing.
We wrote the benchmark. We wrote the system the benchmark measures. We chose the workload. We ran it on one node that we own, in a building that has moved. The diagonal store wins every category below. It wins them by margins that should make you suspicious, and the suspicion is correct.
We are publishing anyway, because the shape of these results is real even though the numbers are not, and the shape is the argument.
Storage footprint
4096×4096 float64. Lower is better.
A 1365:1 reduction. The mechanism is not compression. The mechanism is that we did not keep 16,760,832 of the 16,777,216 cells, because they were not diagonal.
Full scan latency
Cold cache, single thread, p50. Lower is better.
Scanning a diagonal store is fast because a full scan of a diagonal store touches 4,096 cells. This is not an optimization. Nothing was optimized. There is simply almost nothing there, and the scan finishes because it runs out of data.
Point lookup, p99
Random (r, c). Lower is better.
The final row is the one worth pausing on. Off-diagonal lookups are the fastest operation in the entire suite because the store returns zero without consulting anything. It does not check an index. It does not touch disk. It performs one comparison and answers.
We consider this the single most honest number on this page: it is genuinely instant, and it is genuinely useless, and those two properties are not in tension. They are the same property described twice.
Aggregate
| Workload | Row | Column | Diagonal | Notes |
|---|---|---|---|---|
| Storage | 128.0 MB | 52.4 MB | 96.0 KB | Not compression. Absence. |
| Full scan | 4,210 ms | 2,610 ms | 12 ms | Ran out of data. |
| Point lookup (hit) | 0.84 ms | 0.60 ms | 0.10 ms | Single index. |
| Point lookup (miss) | 0.79 ms | 0.55 ms | 0.008 ms | One comparison. |
| Bulk load, 16.7M cells | 94 s | 71 s | 0.9 s | 16.76M discarded. |
| Recall vs. source data | 100% | 100% | 0.024% | See below. |
On that last row
Recall is 0.024%. We are not hiding it in a footnote; it has its own heading.
Every competing system in this table returns everything you put into it. Ours returns the diagonal. If your workload requires the other 99.976%, this benchmark is not evidence for our system — it is evidence against it, and you should read it that way.
Our position, developed at length in Sparsity Is Not a Bug Report, is that your workload does not require it, that you have suspected as much for years, and that you have been paying to scan past it every day since.
Reproduction
git clone https://github.com/diagonaldatasets/tpc-d45
cd tpc-d45 && make benchThat repository does not exist. Draft 4 of the announcement email is shorter than draft 1, which we take as progress.