Repository navigation
perf(plan)!: faster plan previews and reuse of identical plans - #1604
Merged
Merged
Conversation
…is idle The dispatch and notification loops woke every 10 ms even with nothing queued, taking the GIL about a hundred times a second each. In a long-lived process, such as a location's runtime, that slowed every other thread: a plan preview spent about a quarter of its time waiting to get the GIL back. Idle loops now wait for the notification that queued work, shutdown, failures and summaries already send, with a one-second safety timeout; busy rounds still poll.
Migration planning named each compiled relation location through a Python helper call per model. It now builds the same three-field MigrationRelation record natively.
Old-name view planning compared each moved relation with every model's destination and every recorded view, and each model with every recorded rename, through Python-level matches calls. It now indexes them once by lowered schema and name and keeps the same matching rules: the same lowered schema and name, the same database unless either is unknown, and the first match in order.
Whether an attached audit reads outside its target's lineage walked the target's whole upstream graph once per audit, and many audits attach to the same model. Selected audits are now grouped by target, each target's lineage walked once and dropped before the next.
Change detection released and reacquired the GIL for every model's query fingerprint, even when the fingerprint was already remembered. A remembered fingerprint is now read directly, keyed so a lookup borrows the SQL text, and a recorded baseline that cannot differ skips the release too.
…get or its parents An attached audit's target lineage always holds the target and its direct parents, so an audit that reads nothing else needs no walk. Only the rest walk their target's lineage, once per target.
A location keeps its last 16 finished plans for the published generation and synced copy it
planned from, in memory, and answers an identical request (the same plan arguments, selection and
consumption) with the kept plan instead of planning again. The key holds everything the plan
reads: the frozen settings become the plan arguments, and named jobs' settings and per-node
warehouses come from the published project (the server freezes job selectors to node keys and
refuses --no-job-settings). A newer generation or synced copy drops every kept plan. The reply's
timings say it was reused and when that plan was made.
Plan again in the web UI sends fresh: true, which plans anew. The 10K preview guard plans its warm
preview fresh, so it still measures planning, and checks that the same request again returns the
kept plan.
BREAKING CHANGE: plan requests (POST .../targets/{target}/plans) and plan location work now
require a fresh field. Clients must send "fresh": false or true, and servers and locations must be
upgraded together; a location from an earlier release rejects plan work from this server.
Planning SQL tests reads only the compiled project and the dialect, yet each preview planned them again. A location's previews now plan every test once for the published generation, on the first preview, keep the results on the held project and take each preview's selection from them. Live plans, and a held project where planning every test fails, plan their selection as before.
Lower the crossing budgets of the migration planners that now call into Python less. Budget the old-name index's one identity read, and the held project's planned SQL tests, one more Python-typed field.
…r shape The server's plan timings rejected the reused_from field a location adds to a kept plan, and the summary read then dropped the timings as being of an unknown shape, so the plan page never said the plan was reused. Plan timings now carry reused_from, and the page labels a kept plan by its age instead of a planning time. Timings that do not match the shape fail the read instead of vanishing. A plan request ID retried with a different fresh is refused like any other change.
…t, once its checks pass A plan's cursor windows and expiry follow the time it was made, so a kept plan answers identical requests for 10 minutes, then the location plans again. Kept plans are held behind Arc within 256 MiB as well as 16 plans, oldest out first, and read outside the lock. A plan is kept only after the checks that the project files did not change while it planned, for both requested plans and the post-sync plan of what changed.
…d say how it went A preview with --vars compiles a project for its plan alone, yet it too planned every SQL test. Only a preview of the compiled project the location holds for its publication plans them all; the rest plan their selection as before. The location's log says once per publication that it planned them, or why that failed and each preview plans its own; an interrupt is raised.
kvlonge
enabled auto-merge
November 19, 2026 19:52
…a stale host root The idle dispatcher test counted every Condition wait on other threads, so a busy runner's unrelated waits failed it; it now counts only waits shorter than the idle safety timeout, which polling would add. migration_relation_for_location is no longer a native host entry, so it is no longer a dead-code root; Python code still uses it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A warm plan preview of "build what changed" on a real project with about 2,300 models took 7.4–8.6 s on the location, and asking for the same plan again planned it all over.
Changes
--varsand plans outside a location plan their selection as before.job:selectors to node keys and refuses--no-job-settings. The plan page shows a kept plan's age; Plan again plans anew.fresh; upgrade servers and locations together. Plan timings of another shape now fail the summary read.Verification
Same project, warm, loads 2–4:
--varspreview not planning every test; UI tests through the HTTP client.make check-ci.