clickhouse-web: drop the filesystem cache before the cold try, not every try - #1092
Merged
Conversation
…ery try ./query ran SYSTEM DROP FILESYSTEM CACHE before every try, so all three tries re-fetched from S3 and "hot" was never hot. Since the 2026-05-07 refactor (1f352ad) hot has tracked cold: on c6a.metal 7.5s -> 22.2s, on c6a.2xlarge 47s -> 105s, and the gap never closed. This also contradicts the design of the entry itself: ./install bind-mounts tmpfs onto the cache directory and create.sql wraps the web disk in disk(type = cache) precisely so repeat reads stay local. Move the drop into ./flush-caches, the hook d8d65e0 added for the datalake entries. The harness calls it once per query, before the cold try, so try 1 is honestly cold and tries 2-3 measure the engine rather than the network. It has to delete the files rather than issue SYSTEM DROP FILESYSTEM CACHE: bench_flush_caches runs between ./stop and ./start, with no server to talk to. The cache is in tmpfs, which drop_caches does not evict, and the tmpfs is bind-mounted onto the cache directory, so remove the contents and leave the mount point in place. Measured against the real web disk, one query three times, cache in tmpfs: cold 7.415s, hot 0.524s, hot 0.530s (14x) flush -> 332M cache emptied, directory and mount point intact cold 7.496s, hot 0.530s (cold is honestly cold again) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexey-milovidov
had a problem deploying
to
benchmark-approval
July 26, 2026 00:16 — with 
GitHub Actions
Error
GitHub Actions
Error
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 26, 2026 02:09 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
….metal-48xl, c8g.4xlarge, c8g.metal-48xl)
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Take main's clickhouse-web/results/20260726/ files on both sides of the add/add conflict, dropping the ones the PR's own benchmark runs committed. Those runs (02:41-11:33 UTC) cloned this branch, which did not yet contain c6b4951 "Read the dataset from the copy in this machine's region", so every cold try still crossed the Atlantic: 894s cold on c6a.metal against 31s for the same-region run on main, and on the smaller machines the tmpfs cache is too small to hold the working set, so hot paid the round trips too (c6a.large hot 8156s vs 513s). The hot/cold split this PR is about was visible in them (c7a.metal-48xl hot 8.8s vs 20.7s), but the absolute numbers measured the network, not the change. With the region fix now merged in, the push re-triggers the PR benchmark and the rerun measures the entry as it will actually run.
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 26, 2026 20:59 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
…6a.metal, c6a.xlarge, c7a.metal-48xl, c8g.4xlarge, c8g.metal-48xl)
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
clickhouse-web/queryranSYSTEM DROP FILESYSTEM CACHEbefore every try, so all three tries re-fetched from S3 and "hot" was never hot. Since the 2026-05-07 refactor (1f352ad) hot has tracked cold and the gap never closed:It also contradicts the design of the entry itself:
./installbind-mounts tmpfs onto the cache directory andcreate.sqlwraps the web disk indisk(type = cache)precisely so repeat reads stay local.And it is what turned the eu-central-1 → us-east-1 move (#1091) from a latency penalty into a 30x one: with every try cold, every try paid the transatlantic round trips.
The change
Move the drop into
./flush-caches, the hook d8d65e0 added for the datalake entries. The harness calls it once per query, before the cold try, so try 1 is honestly cold and tries 2–3 measure the engine rather than the network.It has to delete the files rather than issue
SYSTEM DROP FILESYSTEM CACHE:bench_flush_cachesruns between./stopand./start, with no server to talk to. Two details follow from that — the cache is in tmpfs, whichdrop_cachesdoes not evict, and the tmpfs is bind-mounted onto the cache directory, so the script removes the contents and leaves the mount point in place.Testing
Measured against the real web disk, one query three times, cache in tmpfs:
So the cold number stays honest and the hot number starts measuring what it is supposed to.
Not covered: the sudo path in
flush-caches— this dev box has no passwordless sudo. It matches existing usage inbench_flush_caches(echo 3 | sudo tee /proc/sys/vm/drop_caches) and./install(sudo mount --bind), both of which already rely on it on the runners.Expect
clickhouse-webhot sums to drop back toward their pre-2026-05-09 levels once this lands. Independent of #1091, but the two compound: that PR removes the transatlantic round trips, this one stops paying them three times per query.🤖 Generated with Claude Code