Skip to content

feat(state): record each run in every schema it writes to - #1600

Merged
kvlonge merged 9 commits into
mainfrom
feat/run-records-per-schema
Nov 19, 2026
Merged

kvlonge merged 9 commits into
mainfrom
feat/run-records-per-schema

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Why

A run appended its _sqlbuild_runs rows to the target's default schema, creating it if missing, so a role that builds only in other schemas needed rights on a schema it never builds in. Separately, a build that hit its execution deadline lost its finished run record: the row was written under the expired deadline and refused.

Changes

  • Each build and sqb seed writes its started and finished rows, under one run ID, to every schema it writes: its models' and seeds' that run, its SQL functions' and its loaded sources'. Readers merge the rows by run ID.
  • A run that writes no relation, such as only Python tasks or checks, records in the target's default schema, as audit results do.
  • Schema names compare as the dialect resolves unquoted names, so one physical schema spelled two ways is written once. Each phase opens at most four connections.
  • Writers run with deadlines suspended in the caller's context, so a build stopped by its deadline still records its finished row and run-record statements stay in the build's events and recall.
  • Ctrl+C stops the writers promptly: a stop flag ends each writer before its next table and running statements are cancelled. Cut-short writes are not reported as failed.
  • A failed write warns naming that schema's table and never fails the run.
  • Run records and the janitor share one target-location reader.

Verification

  • Integration tests: runs across two schemas, a custom schema only, case spellings, a skipped model, a function, a source load, a build that raises, a task-only run, a seed run, an unwritable schema, a real SIGINT while an insert hangs, and run-record inserts in the build's recall.
  • Worker run-record case, Snowflake state-DDL unit tests, make check-ci; one review plus follow-up.

A run appended its started and finished rows to _sqlbuild_runs in the
target's default schema, creating that schema when missing. A build that
writes only to other schemas then needed rights on a schema it never
builds in, and could create a schema solely to hold state.

Each run now writes both rows, with the same run ID, to every schema it
writes to: the schemas of the models and seeds it runs (plan action not
skip), its SQL functions and the sources it loads; for `sqb seed`, its
seeds' schemas. The set is fixed from the plan before the first node
runs, so each schema gets the pair. The writes run concurrently. As
before, the started row sets each schema's runs table up (creating the
schema, which the build would create anyway) and the finished row
inserts directly, setting the table up again only when an ordinary error
stops that insert; an interrupt still stops the run. A failed write warns
naming that schema's table and never fails the run; the other schemas
still record it. A run whose nodes write no relation records nothing.
Readers merge the rows by run ID.
…e default schema

Review follow-up to recording runs per schema.

One physical schema spelled two ways, such as marts and MARTS on
Snowflake or with and without the target's database, was written twice.
Schemas now compare as the dialect resolves unquoted names (its native
profile) with an unnamed database filled in from the target's, so each
runs table gets one pair.

Each phase opened a connection per schema, and a second on a first run.
It now opens at most four, each writing its share of schemas in turn,
and the create-and-retry reuses its connection: one login per phase for
one schema, four for five or twenty. A failed schema still warns alone.
Every writer runs on its own thread, outside the calling thread's
context, so a build stopped by its execution deadline still records its
finished row in each schema. An interrupt raised in a writer, or a
Ctrl+C while the writers run, still stops the run once they finish.

A run that writes no relation, such as only Python tasks or checks, now
records in the target's default schema, resolved as audit results
resolve theirs, instead of nowhere. Runs that write a relation still
write only their own schemas.

Tests cover case spellings, a skipped (disabled) model's schema, a
function's schema, a build that raises across two schemas, and a
task-only run; the killed-build test selects its crashing task alone
again. The docs no longer claim the sync merges runs yet.
…em on Ctrl+C

Review follow-up to recording runs per schema.

Each run-record writer thread started from an empty context, so its
statements left the build's event and recall stream. Writers now take
a copy of the caller's context and run with every deadline suspended,
through the existing execution_deadlines_suspended scope, now a native
host entry. A build stopped by its execution deadline still records its
finished row in each schema, and its run-record statements are recalled
with the build again.

Ctrl+C no longer waits for every writer to finish its share. A caught
signal sets a stop flag that each writer checks before its next table
and before a retry, and cancels the running statements as SIGTERM does.
A further signal while the writers stop cancels again instead of staying
pending. Writes cut short that way are not reported as failed, and the
interrupt escapes once the writers stop.

run_recorder now takes the command's pipeline and picks the build's or
seed's schemas itself, which keeps the Python bindings crate within its
code-line cap.

Tests send a real SIGINT while a run-record insert hangs, and check that
the build stops within seconds, raises KeyboardInterrupt and warns of no
failed write; and that a build's run-record inserts are among its own
recalled statements.
Run records and janitor planning both read the active target's database
or schema as a location, naming none for preserve. They now share
native_host's target_location.
@kvlonge
kvlonge enabled auto-merge November 19, 2026 17:20
Native run-record writers resolve execution_deadlines_suspended through
the sqlbuild._native_host name table, so it is a dead-code root like the
other host targets.
…schema

# Conflicts:
#	tests/integration/src/sqlbuild/cli/commands/main/_test_types.py
#	tests/integration/src/sqlbuild/cli/commands/main/helpers.py
@kvlonge
kvlonge added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit eb91cc8 Nov 19, 2026
35 checks passed
@kvlonge
kvlonge deleted the feat/run-records-per-schema branch November 19, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant