From c0d386a55a2a57295a51778632b9d1e13bebb09f Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Thu, 19 Nov 2026 07:02:40 +0000 Subject: [PATCH] docs: give maintained models their own page Move maintained models from the materializations page to their own page next to Incremental and Snapshots: settings, what the plan shows, what a build does, and what happens when a build dies. The materializations page keeps a short section like View and Table. Drop the error-code table; A718 and A719 are named where their situations are explained. Repoint links to the old anchors. --- .../sqlbuild/references/docs/CONTENTS.md | 1 + .../sqlbuild/references/docs/cli/build.md | 2 +- .../docs/concepts/adapters/snowflake.md | 2 +- .../docs/concepts/models/maintained.md | 82 +++++++++++++++++++ .../docs/concepts/models/materializations.md | 64 +-------------- .../docs/concepts/project-configuration.md | 4 +- website/src/content/docs/docs/cli/build.mdx | 2 +- .../docs/docs/concepts/adapters/snowflake.mdx | 2 +- .../docs/docs/concepts/models/maintained.mdx | 79 ++++++++++++++++++ .../docs/concepts/models/materializations.mdx | 64 +-------------- .../docs/concepts/project-configuration.mdx | 4 +- website/src/sidebar.json | 1 + 12 files changed, 177 insertions(+), 130 deletions(-) create mode 100644 src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/maintained.md create mode 100644 website/src/content/docs/docs/concepts/models/maintained.mdx diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md index d68a377f7b..484e052615 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md @@ -24,6 +24,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in - [Seeds](concepts/seeds.md) (`concepts/seeds`) - Load static CSV data into your pipeline as tables. - [Incremental Models](concepts/incremental.md) (`concepts/incremental`) - Cursor-based incremental strategies, microbatch execution, and backfill policies. - [Snapshots (SCD Type 2)](concepts/snapshots.md) (`concepts/snapshots`) - Preserve row history over time using SCD Type 2 semantics with timestamp or check-based change detection. +- [Maintained Models](concepts/models/maintained.md) (`concepts/models/maintained`) - Let the warehouse keep a model up to date on a schedule, paused safely while builds run. - [Functions](concepts/functions.md) (`concepts/functions`) - User-defined functions and table functions managed as part of your project. - [Macros](concepts/macros.md) (`concepts/macros`) - Reuse SQL with typed SQL macros, or generate it with Python macros when it needs logic. - [SQL Macros](concepts/macros/sql-macros.md) (`concepts/macros/sql-macros`) - Write typed, logic-free SQL fragments that the compiler checks and sees inside. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md index f006832b56..cc013924c9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md @@ -66,7 +66,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--defer-sources-to` | Read managed source data from another target | | `--plan-from` | Read warehouse state `live`, or from the target's `synced_state` checked against the warehouse first; overrides the target's `plan_from`. See [Planning from synced state](../concepts/project-configuration.md#planning-from-synced-state) | | `--fail-fast` | Stop on first failure and skip remaining nodes | -| `--ignore-maintained-grace` | Take over [maintained models](../concepts/models/materializations.md#during-a-build) another build holds, without waiting for `[maintained] heartbeat_grace`. Needs `allow_ignore_grace = true`, else fails with `A719` | +| `--ignore-maintained-grace` | Take over [maintained models](../concepts/models/maintained.md#when-a-build-dies) another build holds, without waiting for `[maintained] heartbeat_grace`. Needs `allow_ignore_grace = true`, else fails with `A719` | | `--concurrency` | Number of worker connections (default: 1) | | `--test-timeout ` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](../concepts/testing-reference.md#time-limits) | | `--verbose`, `-v` | Show lifecycle SQL inline after each model | diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md index 3074e3168e..fa4c878d51 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/snowflake.md @@ -168,7 +168,7 @@ state table. Models are not affected: their table type follows the model's `tabl ## Maintained models -A [maintained model](../models/materializations.md#maintained) is a Snowflake dynamic +A [maintained model](../models/maintained.md) is a Snowflake dynamic table. `materialized materialized_view` fails with `A014`; use `materialized maintained`. The build role needs these grants: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/maintained.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/maintained.md new file mode 100644 index 0000000000..777c4de3c5 --- /dev/null +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/maintained.md @@ -0,0 +1,82 @@ + + +# Maintained Models + +> Let the warehouse keep a model up to date on a schedule, paused safely while builds run. + +Online: https://sqlbuild.com/docs/concepts/models/maintained/ + +A maintained model is one the warehouse keeps up to date on a schedule you set, between builds. A +build creates it, changes it when its definition changes, and otherwise leaves the refreshing to +the warehouse; on Snowflake it is a dynamic table. + +```sql +MODEL ( + description "One row per customer per order date", + materialized maintained, + refresh (lag 1h), +); + +SELECT customer_id, CAST(ordered_at AS DATE) AS order_date, SUM(amount) AS revenue +FROM __ref("stg_orders") +GROUP BY customer_id, CAST(ordered_at AS DATE) +``` + +Only Snowflake supports maintained models. See [Snowflake](../adapters/snowflake.md#maintained-models) +for the grants and change tracking they need. + +## Settings + +| Field | Default | Description | +|-------|---------|-------------| +| `refresh (lag )` | none | How far behind its inputs the model may fall: a fixed duration such as `5m`, `1h` or `1d`, at least `1m` on Snowflake and no shorter than the lag of a maintained model it reads. Without it, the model refreshes when a maintained model that reads it refreshes | +| `refresh_mode` | `incremental` | `incremental`, `full`, `auto` or `adaptive`. Snowflake rejects `incremental` for a query it can't refresh incrementally; use `full` for that query | +| `refresh_warehouse` | the connection's warehouse | Warehouse the scheduled refreshes run on | +| `refresh_on_build` | `false` | Refresh the model in every build that selects it, not only on its schedule | + +## What the plan shows + +| Change | Plan | What the build does | +|--------|------|---------------------| +| First build, or the dynamic table is missing | `create` | Creates it and fills it | +| Query or `refresh_mode` changed | `recreate, full recompute` | Replaces it; every row is computed again | +| `refresh` lag or `refresh_warehouse` differs from the warehouse | `change refresh settings in place` | Alters the schedule; rows stay | +| Nothing | `maintained` | Leaves it alone | + +A model the plan keeps can carry one more line: + +- `recomputes in full: stg_orders is rebuilt`: the build replaces a model it reads, so its next refresh computes every row again. +- `paused: scheduled refreshes are suspended`: Snowflake lists it as suspended, so it is not refreshing. + +## During a build + +A build pauses the maintained models it changes, and those that read a model it rebuilds, directly +or through views, so they never refresh from half-built inputs. It resumes them when it ends, +whether it succeeds or fails. + +- Dynamic tables outside the project that read a paused model pause too, until the build resumes it. +- A model someone suspended before the build stays suspended, and the build warns about it. +- A suspend made by someone else during the build is undone when the build resumes the model. +- A kept model refreshes during the build when a model built later in the same build reads it. +- `refresh_on_build` refreshes the model in every build that selects it. +- A target with `pause_maintained = true` leaves its maintained models suspended between builds; see [Targets](../project-configuration.md#targets). + +## When a build dies + +- **One build at a time.** A build that needs a maintained model another build holds fails with + `A718`, naming that run. Wait for it to finish. +- **A stopped build leaves its models suspended.** A build killed before it resumes them, for + example by `SIGKILL`, can't resume them itself. +- **The next build resumes them.** The next build that pauses maintained models on the target + resumes them once the stopped build has been silent for `[maintained] heartbeat_grace`, 30 + minutes by default; see [Maintained](../project-configuration.md#maintained). +- **Taking over at once.** Run `sqb build --ignore-maintained-grace` to take the models over + without waiting. It needs `[maintained] allow_ignore_grace = true`; without it the flag fails + with `A719`. You can also resume the models in Snowflake by hand. + +## Limitations + +- Writing out the default `refresh_mode incremental` recreates the model once. +- A snapshot rebuilt in full loses change tracking; run `ALTER TABLE SET CHANGE_TRACKING = TRUE` before a maintained model reads it again. +- Unit tests and scenarios treat a maintained model as a table. +- On a target with `schema = "preserve"`, builds see each other's pauses only when their maintained models share a schema. diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md index a6c8c4d203..5ca09cd193 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/materializations.md @@ -86,75 +86,17 @@ See [Snapshots](../snapshots.md) for timestamp and check strategies, historical ## Maintained -The warehouse keeps a maintained model up to date on a schedule you set, between builds. On Snowflake it is a dynamic table; a build creates it, changes it when its definition changes, and otherwise leaves the refreshing to Snowflake. +The warehouse keeps the model up to date on a schedule you set, and builds pause it while they change its inputs: ```sql -MODEL ( - description "One row per customer per order date", - materialized maintained, - refresh (lag 1h), -); +MODEL (description "One row per customer per order date", materialized maintained, refresh (lag 1h)); SELECT customer_id, CAST(ordered_at AS DATE) AS order_date, SUM(amount) AS revenue FROM __ref("stg_orders") GROUP BY customer_id, CAST(ordered_at AS DATE) ``` -| Field | Default | Description | -|-------|---------|-------------| -| `refresh (lag )` | none | How far behind its inputs the model may fall: a fixed duration such as `5m`, `1h` or `1d`, at least `1m` on Snowflake and no shorter than the lag of a maintained model it reads. Without it, the model refreshes when a maintained model that reads it refreshes | -| `refresh_mode` | `incremental` | `incremental`, `full`, `auto` or `adaptive`. Snowflake rejects `incremental` for a query it can't refresh incrementally; use `full` for that query | -| `refresh_warehouse` | the connection's warehouse | Warehouse the scheduled refreshes run on | -| `refresh_on_build` | `false` | Refresh the model in every build that selects it, not only on its schedule | - -Only Snowflake supports maintained models. See [Snowflake](../adapters/snowflake.md#maintained-models) for the grants and change tracking they need. - -### What the plan shows - -| Change | Plan | What the build does | -|--------|------|---------------------| -| First build, or the dynamic table is missing | `create` | Creates it and fills it | -| Query or `refresh_mode` changed | `recreate, full recompute` | Replaces it; every row is computed again | -| `refresh` lag or `refresh_warehouse` differs from the warehouse | `change refresh settings in place` | Alters the schedule; rows stay | -| Nothing | `maintained` | Leaves it alone | - -A model the plan keeps can carry one more line: - -- `recomputes in full: stg_orders is rebuilt`: the build replaces a model it reads, so its next refresh computes every row again. -- `paused: scheduled refreshes are suspended`: Snowflake lists it as suspended, so it is not refreshing. - -### During a build - -A build pauses the maintained models it changes, and those that read a model it rebuilds, directly or through views, so they never refresh from half-built inputs. It resumes them when it ends, whether it succeeds or fails. - -- Dynamic tables outside the project that read a paused model pause too, until the build resumes it. One that someone had suspended before stays suspended. -- A model that was already suspended when the build started stays suspended, and the build warns about it. -- If someone suspends a model during the build, the build's resume undoes that suspend. -- A kept model is refreshed during the build when a model built later in the same build reads it, so that model reads current rows. `refresh_on_build` refreshes it in every build. -- A target with `pause_maintained = true` leaves its maintained models suspended between builds and refreshes them only in builds; see [Targets](../project-configuration.md#targets). - -Two builds can't pause the same models at once. A build that needs a model another build holds fails with `A718`, naming that run. When two builds pause a model at the same moment, the one that recorded its pause first keeps it, and the other fails with `A718` without suspending or resuming anything. A build that stops without resuming, for example because its process was killed, leaves its models suspended. The next build that pauses maintained models on the target resumes them once that build has been silent for `[maintained] heartbeat_grace` (see [Maintained](../project-configuration.md#maintained)). Until then, resume them in Snowflake by hand, or run `sqb build --ignore-maintained-grace` where the project allows it. - -### Error codes - -| Code | When | Fix | -|------|------|-----| -| `A013` | `materialized maintained` on an adapter other than Snowflake | Use `table` or `incremental` on that adapter | -| `A014` | `materialized materialized_view` on Snowflake | Use `materialized maintained` | -| `A015` | A `refresh` lag under `1m` on Snowflake | Set a lag of at least `1m` | -| `A016` | A model with no `refresh` that no maintained model reads, so it would never refresh | Add `refresh (lag 1h)` or another lag | -| `A716` | The adapter can't build a maintained model | Use an adapter that supports them | -| `A717` | No warehouse to refresh with | Set `refresh_warehouse`, or a `warehouse` on the connection | -| `A718` | Another build holds the models this build needs | Wait for it to finish, or for its hold to pass `heartbeat_grace` | -| `A719` | `--ignore-maintained-grace` without `allow_ignore_grace = true` | Set `[maintained] allow_ignore_grace = true`, or wait for the grace to pass | - -### Limitations - -- Writing out the default `refresh_mode incremental` changes the model's definition, so the next build recreates it once. -- A custom materialization named `maintained` is not used; the built-in one takes the name. -- Unit tests and scenarios treat a maintained model as a table. -- On a target with `schema = "preserve"`, builds see each other's pauses only when their maintained models share a schema. -- A snapshot rebuilt in full is created without change tracking. Before a maintained model reads it again, run `ALTER TABLE SET CHANGE_TRACKING = TRUE`. +See [Maintained models](maintained.md) for its settings, what the plan shows and what happens when a build dies. ## Custom diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md index 38ca3a22f6..9a40443f25 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md @@ -190,7 +190,7 @@ A target is a named build context - the database and schema you build into, plus | `concurrency` | Worker limit for commands run against this target, replacing `[settings] concurrency` (see below) | | `ignore_audit_failures` | Whether builds on this target let audit failures through, replacing `[settings] ignore_audit_failures`; see [Ignoring audit failures](audits.md#ignoring-audit-failures) | | `plan_from` | Where `sqb plan` and `sqb build` read warehouse state: `live` (default) or `synced_state`; see [Planning from synced state](#planning-from-synced-state) | -| `pause_maintained` | `true` keeps this target's [maintained models](models/materializations.md#maintained) suspended between builds; each build refreshes the ones it selects. Default `false` | +| `pause_maintained` | `true` keeps this target's [maintained models](models/maintained.md) suspended between builds; each build refreshes the ones it selects. Default `false` | ```toml [connections.warehouse] @@ -699,7 +699,7 @@ old_name_views = "30d" ## Maintained -How builds pause [maintained models](models/materializations.md#maintained) while they run: +How builds pause [maintained models](models/maintained.md) while they run: ```toml [maintained] diff --git a/website/src/content/docs/docs/cli/build.mdx b/website/src/content/docs/docs/cli/build.mdx index 31fed956bc..2a69160633 100644 --- a/website/src/content/docs/docs/cli/build.mdx +++ b/website/src/content/docs/docs/cli/build.mdx @@ -49,7 +49,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--defer-sources-to` | Read managed source data from another target | | `--plan-from` | Read warehouse state `live`, or from the target's `synced_state` checked against the warehouse first; overrides the target's `plan_from`. See [Planning from synced state](/docs/concepts/project-configuration#planning-from-synced-state) | | `--fail-fast` | Stop on first failure and skip remaining nodes | -| `--ignore-maintained-grace` | Take over [maintained models](/docs/concepts/models/materializations#during-a-build) another build holds, without waiting for `[maintained] heartbeat_grace`. Needs `allow_ignore_grace = true`, else fails with `A719` | +| `--ignore-maintained-grace` | Take over [maintained models](/docs/concepts/models/maintained#when-a-build-dies) another build holds, without waiting for `[maintained] heartbeat_grace`. Needs `allow_ignore_grace = true`, else fails with `A719` | | `--concurrency` | Number of worker connections (default: 1) | | `--test-timeout ` | Time limit for each unit test the build runs, such as `30s`, or `disabled`; see [Time limits](/docs/concepts/testing-reference#time-limits) | | `--verbose`, `-v` | Show lifecycle SQL inline after each model | diff --git a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx index 8dddc56ef3..f229f05121 100644 --- a/website/src/content/docs/docs/concepts/adapters/snowflake.mdx +++ b/website/src/content/docs/docs/concepts/adapters/snowflake.mdx @@ -152,7 +152,7 @@ state table. Models are not affected: their table type follows the model's `tabl ## Maintained models -A [maintained model](/docs/concepts/models/materializations#maintained) is a Snowflake dynamic +A [maintained model](/docs/concepts/models/maintained) is a Snowflake dynamic table. `materialized materialized_view` fails with `A014`; use `materialized maintained`. The build role needs these grants: diff --git a/website/src/content/docs/docs/concepts/models/maintained.mdx b/website/src/content/docs/docs/concepts/models/maintained.mdx new file mode 100644 index 0000000000..e825a6db5e --- /dev/null +++ b/website/src/content/docs/docs/concepts/models/maintained.mdx @@ -0,0 +1,79 @@ +--- +title: "Maintained Models" +description: "Let the warehouse keep a model up to date on a schedule, paused safely while builds run." +--- + +A maintained model is one the warehouse keeps up to date on a schedule you set, between builds. A +build creates it, changes it when its definition changes, and otherwise leaves the refreshing to +the warehouse; on Snowflake it is a dynamic table. + +```sql +MODEL ( + description "One row per customer per order date", + materialized maintained, + refresh (lag 1h), +); + +SELECT customer_id, CAST(ordered_at AS DATE) AS order_date, SUM(amount) AS revenue +FROM __ref("stg_orders") +GROUP BY customer_id, CAST(ordered_at AS DATE) +``` + +Only Snowflake supports maintained models. See [Snowflake](/docs/concepts/adapters/snowflake#maintained-models) +for the grants and change tracking they need. + +## Settings + +| Field | Default | Description | +|-------|---------|-------------| +| `refresh (lag )` | none | How far behind its inputs the model may fall: a fixed duration such as `5m`, `1h` or `1d`, at least `1m` on Snowflake and no shorter than the lag of a maintained model it reads. Without it, the model refreshes when a maintained model that reads it refreshes | +| `refresh_mode` | `incremental` | `incremental`, `full`, `auto` or `adaptive`. Snowflake rejects `incremental` for a query it can't refresh incrementally; use `full` for that query | +| `refresh_warehouse` | the connection's warehouse | Warehouse the scheduled refreshes run on | +| `refresh_on_build` | `false` | Refresh the model in every build that selects it, not only on its schedule | + +## What the plan shows + +| Change | Plan | What the build does | +|--------|------|---------------------| +| First build, or the dynamic table is missing | `create` | Creates it and fills it | +| Query or `refresh_mode` changed | `recreate, full recompute` | Replaces it; every row is computed again | +| `refresh` lag or `refresh_warehouse` differs from the warehouse | `change refresh settings in place` | Alters the schedule; rows stay | +| Nothing | `maintained` | Leaves it alone | + +A model the plan keeps can carry one more line: + +- `recomputes in full: stg_orders is rebuilt`: the build replaces a model it reads, so its next refresh computes every row again. +- `paused: scheduled refreshes are suspended`: Snowflake lists it as suspended, so it is not refreshing. + +## During a build + +A build pauses the maintained models it changes, and those that read a model it rebuilds, directly +or through views, so they never refresh from half-built inputs. It resumes them when it ends, +whether it succeeds or fails. + +- Dynamic tables outside the project that read a paused model pause too, until the build resumes it. +- A model someone suspended before the build stays suspended, and the build warns about it. +- A suspend made by someone else during the build is undone when the build resumes the model. +- A kept model refreshes during the build when a model built later in the same build reads it. +- `refresh_on_build` refreshes the model in every build that selects it. +- A target with `pause_maintained = true` leaves its maintained models suspended between builds; see [Targets](/docs/concepts/project-configuration#targets). + +## When a build dies + +- **One build at a time.** A build that needs a maintained model another build holds fails with + `A718`, naming that run. Wait for it to finish. +- **A stopped build leaves its models suspended.** A build killed before it resumes them, for + example by `SIGKILL`, can't resume them itself. +- **The next build resumes them.** The next build that pauses maintained models on the target + resumes them once the stopped build has been silent for `[maintained] heartbeat_grace`, 30 + minutes by default; see [Maintained](/docs/concepts/project-configuration#maintained). +- **Taking over at once.** Run `sqb build --ignore-maintained-grace` to take the models over + without waiting. It needs `[maintained] allow_ignore_grace = true`; without it the flag fails + with `A719`. You can also resume the models in Snowflake by hand. + +## Limitations + +- Writing out the default `refresh_mode incremental` recreates the model once. +- A snapshot rebuilt in full loses change tracking; run `ALTER TABLE SET CHANGE_TRACKING = TRUE` before a maintained model reads it again. +- Unit tests and scenarios treat a maintained model as a table. +- On a target with `schema = "preserve"`, builds see each other's pauses only when their maintained models share a schema. diff --git a/website/src/content/docs/docs/concepts/models/materializations.mdx b/website/src/content/docs/docs/concepts/models/materializations.mdx index 9fa53d5d07..827b8616ed 100644 --- a/website/src/content/docs/docs/concepts/models/materializations.mdx +++ b/website/src/content/docs/docs/concepts/models/materializations.mdx @@ -73,75 +73,17 @@ See [Snapshots](/docs/concepts/snapshots) for timestamp and check strategies, hi ## Maintained -The warehouse keeps a maintained model up to date on a schedule you set, between builds. On Snowflake it is a dynamic table; a build creates it, changes it when its definition changes, and otherwise leaves the refreshing to Snowflake. +The warehouse keeps the model up to date on a schedule you set, and builds pause it while they change its inputs: ```sql -MODEL ( - description "One row per customer per order date", - materialized maintained, - refresh (lag 1h), -); +MODEL (description "One row per customer per order date", materialized maintained, refresh (lag 1h)); SELECT customer_id, CAST(ordered_at AS DATE) AS order_date, SUM(amount) AS revenue FROM __ref("stg_orders") GROUP BY customer_id, CAST(ordered_at AS DATE) ``` -| Field | Default | Description | -|-------|---------|-------------| -| `refresh (lag )` | none | How far behind its inputs the model may fall: a fixed duration such as `5m`, `1h` or `1d`, at least `1m` on Snowflake and no shorter than the lag of a maintained model it reads. Without it, the model refreshes when a maintained model that reads it refreshes | -| `refresh_mode` | `incremental` | `incremental`, `full`, `auto` or `adaptive`. Snowflake rejects `incremental` for a query it can't refresh incrementally; use `full` for that query | -| `refresh_warehouse` | the connection's warehouse | Warehouse the scheduled refreshes run on | -| `refresh_on_build` | `false` | Refresh the model in every build that selects it, not only on its schedule | - -Only Snowflake supports maintained models. See [Snowflake](/docs/concepts/adapters/snowflake#maintained-models) for the grants and change tracking they need. - -### What the plan shows - -| Change | Plan | What the build does | -|--------|------|---------------------| -| First build, or the dynamic table is missing | `create` | Creates it and fills it | -| Query or `refresh_mode` changed | `recreate, full recompute` | Replaces it; every row is computed again | -| `refresh` lag or `refresh_warehouse` differs from the warehouse | `change refresh settings in place` | Alters the schedule; rows stay | -| Nothing | `maintained` | Leaves it alone | - -A model the plan keeps can carry one more line: - -- `recomputes in full: stg_orders is rebuilt`: the build replaces a model it reads, so its next refresh computes every row again. -- `paused: scheduled refreshes are suspended`: Snowflake lists it as suspended, so it is not refreshing. - -### During a build - -A build pauses the maintained models it changes, and those that read a model it rebuilds, directly or through views, so they never refresh from half-built inputs. It resumes them when it ends, whether it succeeds or fails. - -- Dynamic tables outside the project that read a paused model pause too, until the build resumes it. One that someone had suspended before stays suspended. -- A model that was already suspended when the build started stays suspended, and the build warns about it. -- If someone suspends a model during the build, the build's resume undoes that suspend. -- A kept model is refreshed during the build when a model built later in the same build reads it, so that model reads current rows. `refresh_on_build` refreshes it in every build. -- A target with `pause_maintained = true` leaves its maintained models suspended between builds and refreshes them only in builds; see [Targets](/docs/concepts/project-configuration#targets). - -Two builds can't pause the same models at once. A build that needs a model another build holds fails with `A718`, naming that run. When two builds pause a model at the same moment, the one that recorded its pause first keeps it, and the other fails with `A718` without suspending or resuming anything. A build that stops without resuming, for example because its process was killed, leaves its models suspended. The next build that pauses maintained models on the target resumes them once that build has been silent for `[maintained] heartbeat_grace` (see [Maintained](/docs/concepts/project-configuration#maintained)). Until then, resume them in Snowflake by hand, or run `sqb build --ignore-maintained-grace` where the project allows it. - -### Error codes - -| Code | When | Fix | -|------|------|-----| -| `A013` | `materialized maintained` on an adapter other than Snowflake | Use `table` or `incremental` on that adapter | -| `A014` | `materialized materialized_view` on Snowflake | Use `materialized maintained` | -| `A015` | A `refresh` lag under `1m` on Snowflake | Set a lag of at least `1m` | -| `A016` | A model with no `refresh` that no maintained model reads, so it would never refresh | Add `refresh (lag 1h)` or another lag | -| `A716` | The adapter can't build a maintained model | Use an adapter that supports them | -| `A717` | No warehouse to refresh with | Set `refresh_warehouse`, or a `warehouse` on the connection | -| `A718` | Another build holds the models this build needs | Wait for it to finish, or for its hold to pass `heartbeat_grace` | -| `A719` | `--ignore-maintained-grace` without `allow_ignore_grace = true` | Set `[maintained] allow_ignore_grace = true`, or wait for the grace to pass | - -### Limitations - -- Writing out the default `refresh_mode incremental` changes the model's definition, so the next build recreates it once. -- A custom materialization named `maintained` is not used; the built-in one takes the name. -- Unit tests and scenarios treat a maintained model as a table. -- On a target with `schema = "preserve"`, builds see each other's pauses only when their maintained models share a schema. -- A snapshot rebuilt in full is created without change tracking. Before a maintained model reads it again, run `ALTER TABLE SET CHANGE_TRACKING = TRUE`. +See [Maintained models](/docs/concepts/models/maintained) for its settings, what the plan shows and what happens when a build dies. ## Custom diff --git a/website/src/content/docs/docs/concepts/project-configuration.mdx b/website/src/content/docs/docs/concepts/project-configuration.mdx index 2d11732db2..52fb28a2c0 100644 --- a/website/src/content/docs/docs/concepts/project-configuration.mdx +++ b/website/src/content/docs/docs/concepts/project-configuration.mdx @@ -163,7 +163,7 @@ A target is a named build context - the database and schema you build into, plus | `concurrency` | Worker limit for commands run against this target, replacing `[settings] concurrency` (see below) | | `ignore_audit_failures` | Whether builds on this target let audit failures through, replacing `[settings] ignore_audit_failures`; see [Ignoring audit failures](/docs/concepts/audits#ignoring-audit-failures) | | `plan_from` | Where `sqb plan` and `sqb build` read warehouse state: `live` (default) or `synced_state`; see [Planning from synced state](#planning-from-synced-state) | -| `pause_maintained` | `true` keeps this target's [maintained models](/docs/concepts/models/materializations#maintained) suspended between builds; each build refreshes the ones it selects. Default `false` | +| `pause_maintained` | `true` keeps this target's [maintained models](/docs/concepts/models/maintained) suspended between builds; each build refreshes the ones it selects. Default `false` | ```toml [connections.warehouse] @@ -672,7 +672,7 @@ old_name_views = "30d" ## Maintained -How builds pause [maintained models](/docs/concepts/models/materializations#maintained) while they run: +How builds pause [maintained models](/docs/concepts/models/maintained) while they run: ```toml [maintained] diff --git a/website/src/sidebar.json b/website/src/sidebar.json index ea80e3d9da..0be8c87a80 100644 --- a/website/src/sidebar.json +++ b/website/src/sidebar.json @@ -31,6 +31,7 @@ "docs/concepts/seeds", "docs/concepts/incremental", "docs/concepts/snapshots", + "docs/concepts/models/maintained", "docs/concepts/functions", { "label": "Macros",