Skip to content

feat: maintained models on Snowflake as dynamic tables, paused safely during builds - #1591

Merged
chio-labs-delivery[bot] merged 17 commits into
mainfrom
feat/maintained-models
Nov 19, 2026
Merged

chio-labs-delivery[bot] merged 17 commits into
mainfrom
feat/maintained-models

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

Teams want tables the warehouse keeps fresh between builds (Snowflake dynamic tables), without refreshes from half-built inputs or two builds fighting over a pause.

Changes

  • materialized maintained with refresh (lag …), refresh_mode (incremental by default), refresh_warehouse and refresh_on_build. Offline errors A013 to A016.
  • Plan: create; recreate (full recompute) on a query or mode change; alter in place when the live lag or warehouse drifts; else keep, noting a full recompute or a pause.
  • Build: CREATE OR REPLACE DYNAMIC TABLE, ALTER and REFRESH. A table or view the model was before is moved aside and restored if the create fails. Tables a maintained model reads, directly or through views, are built with change tracking.
  • Pause lease: a build suspends the models it touches, records append-only events with a heartbeat in _sqlbuild_maintained_pauses, and resumes them afterwards, also after a failure. A live hold fails with A718, and of two builds pausing at once only the first recorded wins. An expired hold is taken over after [maintained] heartbeat_grace, which must exceed the 60s heartbeat. --ignore-maintained-grace needs allow_ignore_grace (A719). Adds target key pause_maintained.
  • Docs state both accepted behaviours: a build's resume undoes a manual suspend made during it, and dynamic tables outside the project that read a paused model pause too.

Verification

  • Stage change tracking was live-tested in Snowflake first and built in slice 6.
  • Live sqb build on a Snowflake trial: create, lag change, a manual suspend kept with a warning, and switching a table to maintained. The switch ran twice: a failed create restored the table, then a create succeeded.
  • Targeted Rust, pytest and E2E suites, vitest, svelte-check; make check-ci on the head.
  • DuckShop can't hold these cases: it runs on DuckDB, where maintained models are an A013 error.
  • Expired holds are recovered by the next build, not by a location's sync.

…checks

Work in progress. Adds materialized maintained with refresh (lag ...),
refresh_mode, refresh_warehouse and refresh_on_build, validated offline.
Coded errors: A013 maintained on an adapter that cannot build it, A014
materialized_view on Snowflake, A015 a lag under Snowflake's 1m minimum,
A016 a maintained model nothing would ever refresh.

Validated: sqlbuild-model-config and maintained_checks unit tests, the
maintained-model compile E2E (5 cases), native model config, unknown key
and compiler differential suites, fensu check. Not yet built: plan,
build execution, pause lease, project settings, docs.
Work in progress. [maintained] heartbeat_grace (fixed duration, default
30m) and allow_ignore_grace (default false), read with
load_maintained_config and validated on every project load, and the
[targets.<name>] pause_maintained flag on project and local targets.

Validated: sqlbuild-config tests (45, new cases fail without the loader),
native stub check, target config unit tests. Known gap: fensu FPRSL105
until the build reads load_maintained_config.
Work in progress. A maintained model is recreated on a first run, a
rename, a full refresh or any query, function, config or schema change,
and kept otherwise. Plan text and the UI word a recreate as a full
recompute with no replay policy, and an alter as an in-place change of
refresh settings. AlterMaintained is defined but not yet chosen: it needs
the live lag and warehouse read.

Validated: planner strategy (14) and cli_text tests, plan-text vitest
(new case fails without the change), 895 planner, manifest and compile
pytest cases, native stub check.
Work in progress. refresh_mode joins the version identity keys, so a
change plans as a config change and recreates the model. Absent keys stay
out of the identity payload, so other models keep their identity. Lag and
warehouse stay out of identity: they will be compared with the live
dynamic table and altered in place.

Validated: 993 planner unit and integration tests; the refresh_mode case
fails without the key.
Work in progress. A kept maintained model now reads its listed target lag
and warehouse through SHOW DYNAMIC TABLES: a drifted lag or declared
warehouse plans an in-place alter, and a name that holds no dynamic table
plans a create. Every maintained plan entry carries its rendered
target_lag. A kept maintained model whose upstream the plan replaces is
marked reinitialised_by that upstream, and sqb plan and the UI say it
recomputes in full.

The read is an optional adapter method; adapters without dynamic tables
list nothing, and a preview plan leaves the action as planned.

Validated: maintained_refresh crate tests, cli_text tests, the Snowflake
SHOW DYNAMIC TABLES read, a DuckDB-backed replan integration test (keep,
alter, create, reinitialise), the strict adapter contract, the plan text
UI tests, and the planner, plan, compile, adapter and run suites.
Work in progress. A build now creates a maintained model as a dynamic
table with CREATE OR REPLACE, an explicit REFRESH_MODE (incremental by
default, so a query that cannot refresh incrementally fails at create)
and INITIALIZE = ON_CREATE, so the table is readable at once. It runs on
refresh_warehouse, or else the session's warehouse; with neither, the
build fails with A717 before any statement. A drifted lag or declared
warehouse is set in place with ALTER, and a kept model runs no refresh
statement. A table or view the model was before is dropped and replaced;
a relation the warehouse lists as a dynamic table is never dropped.

Plan entries carry the model's target_lag, refresh_mode and
refresh_warehouse. The create, alter, suspend, resume and refresh SQL
are optional adapter methods; adapters without dynamic tables raise
A716. The view and maintained executors share one lifecycle for hooks,
final audits and the fingerprint.

Validated: Snowflake SQL rendering, the strict adapter contract, the
maintained executor on DuckDB (create, declared warehouse, replace a
former view, alter, keep, no warehouse, never drop a dynamic table), the
view executor unchanged, and the run, scenario, executor and adapter
suites.
Work in progress. A staged build swaps a freshly created table into
place; on Snowflake, a swapped-in table built without change tracking
makes the next refresh of every dynamic table reading it fail with
"Change tracking is not enabled". The planner now marks every model a
maintained model reads, selected or not, with change_tracking, and the
table, type-enforcement and microbatch full-refresh paths create those
tables with CHANGE_TRACKING = TRUE. A live Snowflake run confirmed that
a stage built this way refreshes the reader after the swap.

Each reader's materialization is read once per plan, only for models
downstream of planned entries. Type enforcement now takes the table
settings as one mapping.

Not covered: a snapshot target that is rebuilt in full.

Validated: Snowflake CTAS rendering, the planner integration test
(tracked, untracked, and a reader outside the selection), the table
executor on DuckDB (staged, type enforcement, untracked control, and a
fails-without-fix check), and the executor, planner and Snowflake
adapter suites.
Work in progress. A build now pauses the maintained models it touches
before its first write and resumes them in a finally step: the models it
creates, and those reading anything it writes, upstream first. Only
models the build found active are paused and resumed; one a person had
suspended stays suspended, even when the build recreates it.

Each pause and resume is appended to _sqlbuild_maintained_pauses in the
target schema with a deterministic event ID, and a heartbeat is written
every 60 s on its own connection, timed by the warehouse clock. A build
that finds a hold whose heartbeat has been silent past
[maintained] heartbeat_grace takes it over and resumes it; a live hold
stops the build before any change (A718) and names the run holding it.
sqb build --ignore-maintained-grace takes over at once, and only when
[maintained] allow_ignore_grace = true (otherwise A719). The flag is not
offered to ad-hoc server runs.

A kept or altered maintained model is refreshed by hand when it sets
refresh_on_build, when a model built later reads it, or on a
pause_maintained target, where models stay suspended and no hold is
recorded. Plan entries now list each model's maintained readers, which
also decide change tracking, in place of the change_tracking flag.

Not covered: the location's warehouse sync does not resume expired
holds; the next build does.

Validated: lease integration tests on DuckDB with a recording adapter
(first build, unchanged and rebuilt upstreams, a person's suspend kept
across a recreate, refresh for a later reader and refresh_on_build,
pause_maintained, expiry takeover, the flag, a live-hold refusal and the
heartbeat), the planner and Snowflake adapter tests, and the executor,
planner, adapter, CLI and janitor suites.
Work in progress. sqb plan now says "paused: scheduled refreshes are
suspended" under a kept maintained model that the warehouse lists as
suspended, and plan JSON marks it with "paused": true; the plan page
shows the same line. The plan never resumes anything.

The live read that already decides whether a kept model's settings
drifted now also returns its scheduling state: the optional adapter
method maintained_listing returns the lag, refresh warehouse and
scheduling state from one SHOW DYNAMIC TABLES, replacing the two
methods it merges. The pause lease reads the same listing.

Validated: the planner integration test (a suspended kept model is
reported, active ones are not), plan text in Rust and in the UI, the
Snowflake listing unit tests, and the executor, planner and adapter
suites.
- Hold warnings and takeovers reach the build's output.
- A build whose hold another build took over leaves those models paused.
- [maintained] heartbeat_grace is at least 3m, three heartbeats.
- Tables a maintained model reads through views are tracked and paused.
- Switching a table or view to maintained restores it if the create fails.
@kvlonge
kvlonge disabled auto-merge November 19, 2026 02:46
@kvlonge
kvlonge marked this pull request as draft November 19, 2026 02:47
…of a failed restore

- Only a heartbeat_grace no longer than the 60s heartbeat interval is rejected.
- After recording its pauses a build reads them back; the earliest live hold on a
  model wins, and a later build closes its hold and stops with A718 before
  suspending anything.
- When moving a former table or view back fails, the error reports both the
  create failure and the restore failure.
A grace just over the 60s interval could otherwise expire a running build,
because its first beat landed one interval plus a connect after its pauses.
@kvlonge
kvlonge marked this pull request as ready for review November 19, 2026 04:26
@kvlonge
kvlonge enabled auto-merge November 19, 2026 04:26
@kvlonge
kvlonge added this pull request to the merge queue Nov 19, 2026
The target-unknown-key failure golden lists every allowed target key;
maintained models add pause_maintained.
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 19, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 7733deb Nov 19, 2026
35 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/maintained-models branch November 19, 2026 06:08
kvlonge added a commit that referenced this pull request Nov 19, 2026
…ained resume

Work in progress. Rebased onto main after #1591.

- Docs: server.mdx (a sync writes only to resume maintained models; the
  resume lines under Follow a sync), materializations and the
  heartbeat_grace row; skill references regenerated.
- The resume logic moved into sqlbuild-python-front, and the writable
  session reuses the history-session edge, to keep sqlbuild-python under
  its line cap.
- Supervisor helpers take and return ownership; tests have no branches;
  dead-code root for the name-table entry.

Verified: maintained-pause Python tests 30/30 (race test still fails on
main's hold code); cargo check for sqlbuild-cli (with tests) and
sqlbuild-python.
Unverified: fensu (last run timed out), the 7-case CLI test and the E2E
since this refactor, the crate cap re-measure.
Still to do: fensu, ruff, clippy, native rebuild and E2E, make test,
review, check-ci.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant