Skip to content

feat(location): resume dynamic models whose build died, with claims that keep a takeover paused - #1609

Closed
kvlonge wants to merge 15 commits into
mainfrom
feat/dynamic-sync-resume
Closed

kvlonge wants to merge 15 commits into
mainfrom
feat/dynamic-sync-resume

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Closes CHI-1569.

Why

When a build dies holding dynamic models suspended, they stay suspended until another build takes the hold over. A location's sync should recover them without ever restarting a model another build holds.

Changes

  • After each completed sync, the location resumes models whose holding run stopped heart-beating past heartbeat_grace, recorded as location-sync:<location>. Live holds and hand-suspended models are left alone.
  • A RESUMING claim closes the takeover race: the resumer claims, re-reads the holds, and skips models held live. A build, with its heartbeat running, waits (2 min max) for open claims on its models, then re-suspends any a claim resumed.
  • Pause table: appended rows only (new RESUMING row type). No schema or control-store change.
  • Snowflake only; also looks in the target's default schema; a quiet sync reads nothing, even after a republish.
  • Failures isolated per hold and location; prompt shutdown.
  • Docs: dynamic models (guarantee, limitations) and server pages.

Verification

  • Two reviews with follow-ups; tests for every unsafe interleaving, each shown to fail first.
  • make check-ci passes on the head.
  • Live Snowflake check on a dev schema: only the expired hold resumed, rows written in order, a quiet pass read nothing, and a build waited on a claim then re-suspended. About 0.04 credits; cleaned up.

…tput

The dbt event stream only depends on sqlbuild-core and
sqlbuild-python-base, so it moves out of the bindings crate. That crate
drops to 50,864 code lines and 92 helper buckets, and both caps come
down.
…entry

resume_holds resumes still-suspended dynamic models and closes their holds; a build's takeover and
the new resume_expired_holds both use it. The sync entry skips models another run holds live and
does nothing without a pause table. A build now reads the state of models whose holds it took
over again after recording its pause, so a resume in between is paused again.
A runtime request, dynamic_resume, resumes the holds silent past [dynamic] heartbeat_grace in each
synced state location over its own writable session, closed before the reply, and records the
resume as location-sync:<location id>. The supervisor sends it once after a completed sync that did
not halt, before the reader closes; each resume prints one line, and a failure only warns.

A sync reads metadata without waking a Snowflake warehouse and skips state tables whose change
token did not move, so an unconditional read of the pause table would wake it on every safety
sync. A pass that leaves no hold open returns the pause table's change token; the supervisor keeps
it per location and sends it back, and while it is unchanged the table is not read. Adapters
without change tokens read the table on every pass.
…ee of branches

The resume entry is a native host root like the others, with its stub; its entry module hands back
the helper's result. The sync-resume tests choose their adapters and tokens by lookup, and a stand-in
adapter resumes a model mid-takeover as a sync would.
…the sync reader back

The sync's resume takes the sync state and its reader by value and returns them, as the metadata
reads do; a reader whose resume timed out is dropped, which ends its process. The bindings pass
the adapter and connection settings to the native host entry, which opens and closes the session
on the configured Python edge and reads [dynamic] heartbeat_grace through a discovery entry. The
served-location test is parametrized, and the sqlbuild-python cap comes down to 51,000 (50,945).
…ent silent

The dynamic models page, the heartbeat_grace row and the server's sync details now say a completed
sync resumes the models of a build silent past [dynamic] heartbeat_grace, its only write, and leaves
live holds and models suspended outside SQLBuild alone.
…takeover race

A refused ALTER no longer stops the pass: each hold's model is resumed on its own, only the holds
whose models resumed are closed, and each failure is returned with its relation. A build taking
over expired holds still stops on the first failure, after closing the ones that resumed.

A sync that read the holds just before a build took one over can still resume that model after
the build paused it. The docs no longer claim otherwise, and a strict xfail test reproduces the
order until the fix is decided.
…namic resume

Finishing a completed sync now races the expired-hold resume against the stop signal, as reads
and deliveries already do, so a slow resume no longer delays shutdown.
…ublishes again

Each publication rebuilt the metadata sync and dropped its quiet tokens, so the next sync read
the pause table again and resumed a suspended warehouse. The tokens depend only on the pause
table, not the generation, so the new sync now inherits them.
…nowflake only, one location at a time

The sync only checked the schemas its models write to, but builds keep the pause table in the
target's schema, so a project with every model in custom schemas never resumed anything. The
location runtime now adds the target's schema unless it is preserved or unset.

Only adapters that can build dynamic models resume; on DuckDB and BigQuery the sync opens no
warehouse session for it. The served DuckDB test now checks that.

A location whose resume fails is reported and the pass carries on with the others. The reply
names each location with its quiet token and lists every failure, and the location logs each
one.
The strict xfail asserted the unsafe resume itself, so no fix could ever turn it into an XPASS.
It now checks only that the model is still suspended, after two of the taking-over build's
heartbeats, so a fix in the sync or a re-suspend from the heartbeat both show up.
…uild taking it over keeps it paused

A sync that read an expired hold just before a build took it over could resume the model after the
build had checked its state, leaving it running under the build's hold.

The sync now records a resuming claim on each model, reads the holds again, and skips any model a
run holds live. A claim is its own run, closed by a resumed row for that run. A build that finds an
open claim on a model it paused waits for the claim to close, then reads the model's state again and
suspends it if the sync resumed it. Each side writes before it reads, so whichever comes second sees
the other.

A claim older than the time a location's reader gets to resume holds (now shared from sqlbuild-core
as DYNAMIC_RESUME_SECONDS) counts as abandoned, so a crashed sync delays a build by at most that.

The race test is a normal test now, and the docs state the guarantee again.
…d holds

A build taking over resumed expired holds on models it does not hold without a claim, so a second
build taking over the same hold could find the model resumed after its state check. That resume
now goes through the same claimed path as a location sync, leaving one resume path with one
guarantee.
…esume claim

A build waited on open claims before it started its heartbeat. With a grace shorter than the wait,
its own hold looked expired, so another resumer could resume the model and close the hold; the
build then suspended the model again and, holding nothing, never resumed it. The heartbeat now
starts before the wait.

The docs list the remaining gap: a resume that outlasts its 2-minute claim.
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 20, 2026
@kvlonge

kvlonge commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Held for a batch PR: this branch is merged into the local integration branch and will land with the next batch.

@kvlonge kvlonge closed this Nov 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a manual request Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant