Repository navigation
feat(location): resume dynamic models whose build died, with claims that keep a takeover paused - #1609
Closed
kvlonge wants to merge 15 commits into
Closed
feat(location): resume dynamic models whose build died, with claims that keep a takeover paused#1609kvlonge wants to merge 15 commits into
kvlonge wants to merge 15 commits into
Conversation
…tput The dbt event stream only depends on sqlbuild-core and sqlbuild-python-base, so it moves out of the bindings crate. That crate drops to 50,864 code lines and 92 helper buckets, and both caps come down.
…entry resume_holds resumes still-suspended dynamic models and closes their holds; a build's takeover and the new resume_expired_holds both use it. The sync entry skips models another run holds live and does nothing without a pause table. A build now reads the state of models whose holds it took over again after recording its pause, so a resume in between is paused again.
A runtime request, dynamic_resume, resumes the holds silent past [dynamic] heartbeat_grace in each synced state location over its own writable session, closed before the reply, and records the resume as location-sync:<location id>. The supervisor sends it once after a completed sync that did not halt, before the reader closes; each resume prints one line, and a failure only warns. A sync reads metadata without waking a Snowflake warehouse and skips state tables whose change token did not move, so an unconditional read of the pause table would wake it on every safety sync. A pass that leaves no hold open returns the pause table's change token; the supervisor keeps it per location and sends it back, and while it is unchanged the table is not read. Adapters without change tokens read the table on every pass.
…ee of branches The resume entry is a native host root like the others, with its stub; its entry module hands back the helper's result. The sync-resume tests choose their adapters and tokens by lookup, and a stand-in adapter resumes a model mid-takeover as a sync would.
…the sync reader back The sync's resume takes the sync state and its reader by value and returns them, as the metadata reads do; a reader whose resume timed out is dropped, which ends its process. The bindings pass the adapter and connection settings to the native host entry, which opens and closes the session on the configured Python edge and reads [dynamic] heartbeat_grace through a discovery entry. The served-location test is parametrized, and the sqlbuild-python cap comes down to 51,000 (50,945).
…ent silent The dynamic models page, the heartbeat_grace row and the server's sync details now say a completed sync resumes the models of a build silent past [dynamic] heartbeat_grace, its only write, and leaves live holds and models suspended outside SQLBuild alone.
…s' new registration module
…takeover race A refused ALTER no longer stops the pass: each hold's model is resumed on its own, only the holds whose models resumed are closed, and each failure is returned with its relation. A build taking over expired holds still stops on the first failure, after closing the ones that resumed. A sync that read the holds just before a build took one over can still resume that model after the build paused it. The docs no longer claim otherwise, and a strict xfail test reproduces the order until the fix is decided.
…namic resume Finishing a completed sync now races the expired-hold resume against the stop signal, as reads and deliveries already do, so a slow resume no longer delays shutdown.
…ublishes again Each publication rebuilt the metadata sync and dropped its quiet tokens, so the next sync read the pause table again and resumed a suspended warehouse. The tokens depend only on the pause table, not the generation, so the new sync now inherits them.
…nowflake only, one location at a time The sync only checked the schemas its models write to, but builds keep the pause table in the target's schema, so a project with every model in custom schemas never resumed anything. The location runtime now adds the target's schema unless it is preserved or unset. Only adapters that can build dynamic models resume; on DuckDB and BigQuery the sync opens no warehouse session for it. The served DuckDB test now checks that. A location whose resume fails is reported and the pass carries on with the others. The reply names each location with its quiet token and lists every failure, and the location logs each one.
The strict xfail asserted the unsafe resume itself, so no fix could ever turn it into an XPASS. It now checks only that the model is still suspended, after two of the taking-over build's heartbeats, so a fix in the sync or a re-suspend from the heartbeat both show up.
…uild taking it over keeps it paused A sync that read an expired hold just before a build took it over could resume the model after the build had checked its state, leaving it running under the build's hold. The sync now records a resuming claim on each model, reads the holds again, and skips any model a run holds live. A claim is its own run, closed by a resumed row for that run. A build that finds an open claim on a model it paused waits for the claim to close, then reads the model's state again and suspends it if the sync resumed it. Each side writes before it reads, so whichever comes second sees the other. A claim older than the time a location's reader gets to resume holds (now shared from sqlbuild-core as DYNAMIC_RESUME_SECONDS) counts as abandoned, so a crashed sync delays a build by at most that. The race test is a normal test now, and the docs state the guarantee again.
…d holds A build taking over resumed expired holds on models it does not hold without a claim, so a second build taking over the same hold could find the model resumed after its state check. That resume now goes through the same claimed path as a location sync, leaving one resume path with one guarantee.
…esume claim A build waited on open claims before it started its heartbeat. With a grace shorter than the wait, its own hold looked expired, so another resumer could resume the model and close the hold; the build then suspended the model again and, holding nothing, never resumed it. The heartbeat now starts before the wait. The docs list the remaining gap: a resume that outlasts its 2-minute claim.
Contributor
Author
|
Held for a batch PR: this branch is merged into the local integration branch and will land with the next batch. |
github-merge-queue
Bot
removed this pull request from the merge queue due to a manual request
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes CHI-1569.
Why
When a build dies holding dynamic models suspended, they stay suspended until another build takes the hold over. A location's sync should recover them without ever restarting a model another build holds.
Changes
heartbeat_grace, recorded aslocation-sync:<location>. Live holds and hand-suspended models are left alone.RESUMINGclaim closes the takeover race: the resumer claims, re-reads the holds, and skips models held live. A build, with its heartbeat running, waits (2 min max) for open claims on its models, then re-suspends any a claim resumed.RESUMINGrow type). No schema or control-store change.Verification
make check-cipasses on the head.