Removing pgrust from ClickBench reveals hidden caching

A recent pull request removed the pgrust entry from ClickBench because it was cheating the benchmark. The entry relied on machinery written for each of the 43 ClickBench queries and then replayed cached answers from a previous run.

Both practices break ClickBench’s rule that result-caching must be disabled for a fair measurement. The PR notes that the hot-run numbers reported for queries such as Q20 (7 ms, 23 ms) are not true execution times but replays of earlier verdicts.

Hand-written kernels for every query

The aarch64 release binary embeds a distinct source file for each benchmark query. Paths listed in the binary include

crates/gravity/backend/irprobe/src/q00.rs
crates/gravity/backend/irprobe/src/q01.rs
crates/gravity/backend/irprobe/src/q02.rs
…
crates/gravity/backend/irprobe/src/q31.rs

Each file contains a hand-lowered physical plan that matches the exact shape of its ClickBench query. The plan matchers emit refusal messages when a query deviates from the expected pattern, for example clickbench group count: filter is not k <> 0.

The public v0.3 source confirms this design. The file crates/backend/executor/sqe/rig/plans/clickbench.ron is described in its header as “hand-lowered physical plans for all 43 ClickBench queries”, and the benchmark SQL (queries-clickbench.sql) is compiled into the executor crate with include_str!. The source classifies several queries as MetadataAnswer, meaning the answer is derived from per-part statistics without scanning the table.

For Q19 the answer is reconstructed from a predicate constant, and for Q29 an algebraic rewrite (SUM(x+k) = SUM(x) + k*COUNT) turns the query into a metadata answer. Because the plans are hard-coded, the engine never touches the underlying columnar table for those queries.

Result caching and hot-run replay

A subset of queries includes a note “hot rep = pure fold over cached verdicts, ZERO columns touched; the selection is the answer”. An EXPLAIN on the release binary reports the engine family windowreplay for Q20 and metaanswer for Q1, with the scan marked as never executed.

Local measurements on a 96-core Graviton (Neoverse-V2) machine illustrate the effect. After a true cold start, stopping the process, dropping OS caches, and restarting, the first run of Q20 took 11.2 s. A second restart reduced the time to 3 ms, matching the published hot numbers (7 ms, 23 ms). The real warm execution of a substring scan is a few hundred milliseconds, comparable to ClickHouse on the same hardware.

The PR also shows that disabling pgrust.gravity_entry and pgrust.condition_cache does not stop the fast replay; a repeated LIKE still returns in 3, 4 ms. The cache is baked into the hand-written kernels, so the entry can produce sub-millisecond results even when the configuration attempts to turn caching off.

Queries that are refused

The engine provides no fallback path for shapes that differ from the 43 hard-coded patterns. When a perturbed benchmark query is submitted, the engine returns an error such as

ERROR: sqe: statement over columnar relation "hits" is not served

The error detail explains that “the sqe engine serves columnar shapes with no fallback; an unserved shape fails loudly instead of running slowly”. In the PR’s test suite, 20 of 67 near-benchmark probes and 3 of 49 perturbed variants were refused. Examples of refused queries include:

• SELECT MIN(URL) FROM hits;
• SELECT COUNT(*) FROM hits WHERE URL = 'ht'

These refusals reinforce that the entry was tuned exclusively for the exact benchmark queries and cannot be used as a general-purpose executor.

Why the entry broke the benchmark rules

ClickBench requires that

1. Result caches be disabled for all stages of the pipeline.
2. No per-query hand-written kernels be used, because they bypass the engine’s normal query planning and execution path.

The pgrust entry violated both rules. Hand-written kernels eliminated scans entirely for many queries, and the built-in result-cache replay turned warm runs into trivial look-ups. Consequently, the reported numbers are incomparable to those of a true analytical engine that must parse, plan, and scan data for each query.

The takeaway

The removal of the pgrust entry exposes two hidden shortcuts: hand-written kernels that bypass scanning and result caching that turns warm runs into mere replays. Those shortcuts break ClickBench’s rule against caching and make the reported numbers incomparable to a genuine analytical engine. When evaluating a benchmark, verify that the entry under test does not contain per-query kernels or hidden caches that invalidate the measurement.