Skip to content

feat(jobs)!: apply named jobs' settings to the nodes they select - #1597

Merged
chio-labs-delivery[bot] merged 23 commits into
mainfrom
feat/jobs-phase1
Nov 19, 2026
Merged

chio-labs-delivery[bot] merged 23 commits into
mainfrom
feat/jobs-phase1

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Why

A job's settings applied only when the server ran it, and every setting reached the whole build. A build naming jobs, --select job:nightly_orders_job, should behave as the job does, give each job's settings only to its own nodes, and say so when two named jobs disagree.

Changes

  • One table lists every sqb build option as a job setting with its scope: node (full_refresh, run_tests, ignore_audit_failures, warehouse), build-wide, or a per-job concurrency limit. One resolver merges named jobs below the command line. Conflicts fail with S027 and help generated from the table.
  • Node settings cover everything the job selects: functions are recreated, singular audits are ignored only when every node they read ignores failures.
  • Each job's nodes run on its warehouse, switching only on change. A warehouse every named job shares is the whole build's. A failed switch fails only that node. Dynamic models refresh on the target's build warehouse.
  • Group jobs must agree on vars and changed. A job setting ignore_dynamic_grace needs [dynamic] allow_ignore_grace at compile.

Breaking changes

  • [on_failure], retries and retry_delay_seconds are removed from job and group files. Rename [on_failure] to [retry], set attempts = retries + 1, and rename retry_delay_seconds to delay_seconds. Old names fail to compile with that hint.
  • Server job runs now build --select job:<name> with the job's build-wide settings and warehouse as flags; node settings apply per node inside the build.

Verification

  • Table tests check every build option against the parser, its scope's merge and conflict rules, and a consumer for every node setting.
  • CLI and worker parity tests show a job run matching its settings as global flags.
  • Offline Snowflake tests cover warehouse switching and dynamic refresh warehouses.
  • fensu, cargo and targeted pytest, the 10K preview guard, make check-ci; review plus follow-ups.

A job named in a selection, as job:<name> or a bare job name, gives the
nodes it selects its settings; naming a group names its jobs. A node
takes a job's settings only when a --select token names the job, selects
the node and the job's own selection holds it, so an intersection narrows
them and + around a job never widens them.

run_tests, ignore_audit_failures, full_refresh and [retry] apply per
node, and two named jobs giving one node different values conflict.
concurrency becomes a limit on the job's own nodes. Every other setting
is build-wide and must be the same in every named job, and a group's jobs
must agree on vars and changed.
…1533)

sqb build and sqb plan now take the settings of the jobs their selection
names. Build-wide settings, such as vars and changed, apply where the
command line leaves them unset, as the same flags typed there would;
every named job must agree on them. Node-level settings apply to the
named job's own nodes: full_refresh rebuilds just those models,
run_tests = false drops their SQL tests, and ignore_audit_failures
decides their audits. concurrency limits how many of the job's nodes run
at once inside the build, and the scheduler starts another ready node
while one job is full. --no-job-settings ignores job settings, and a
group whose jobs set different vars or changed fails to compile.
…ags (CHI-1533)

Apply a named job's build-wide settings as the flags the command line
leaves out, decided from the parser's record of which options it wrote,
so abbreviated and global options count, and parse the command line
once more. A job conflict, invalid configuration or an unreadable file
found while doing so fails again, and is reported, when the command
runs; the planner now also checks that named jobs agree on build-wide
settings, so sqb plan reports the conflict too.

Skip the work when a selection cannot name a job or the project has no
orchestration folder, test the scheduler's per-job dispatch, the
integration of settings, conflicts and per-job full refresh through the
CLI, and document job settings on the jobs, selectors, concurrency,
build, plan and configuration pages.
job: accepts glob patterns, which selected the matching jobs' nodes but
applied none of their settings. A pattern now names every job, and every
group's jobs, it matches, as it already does for selection.
…scheduler (CHI-1533)

A queued microbatch coordinator whose job was full still counted towards
coordinator demand, so a running coordinator could get no sub-workers.
Only coordinators that could start now count, sub-workers take a slot in
their coordinator's jobs, and a lock guards the per-job counts that
coordinator threads now change. Dispatch takes the head of the ready
queue directly when it has room or the build names no limited job.

The tests now run the real dispatch loop with recording workers through
completions, a failure and the skips it causes.
- An option the command line gives also decides every option it rules
  out, so a job's load_sources = false no longer clashes with --reload,
  and job flags go before -- so selectors after it still name jobs.
- A group compiles only if its jobs agree on every build-wide setting on
  each target that declares them together, the same check, with the same
  message, a build of the group makes.
- A selector that cannot resolve a named job's nodes fails, naming the
  job, instead of silently dropping its settings.
- Plans and the run context name the jobs whose own audit failures are
  ignored and whose models are fully refreshed.
- Document that command-line flags replace job settings and that
  --concurrency turns per-job limits off, and test the concurrency limit,
  connection count, per-node audits, groups, job: patterns and
  plan --no-job-settings end to end.
…s by name (CHI-1533)

When a build names jobs and the command line has no --concurrency, its
limit is the larger of the configured limit and the named jobs'
concurrency added together; each job's own limit still applies inside
it, and an explicit --concurrency still wins and turns the per-job
limits off.

A server job run now builds --select job:<name> over its frozen nodes
and settings instead of turning the job's concurrency into
--concurrency, so the build applies the job's concurrency and node-level
settings as a command-line build does; only build-wide settings still
travel as flags.

A named job's nodes are found the way the build selects them, Python
nodes included, so jobs that select Python paths now take their own
settings instead of failing to resolve.
warehouse is now a node-level job setting. When a build names jobs, each
node runs on the warehouse of the named job that selects it; nodes
outside the named jobs stay on the build's warehouse. --warehouse still
puts the whole build on one warehouse and leaves the jobs' out, and two
named jobs giving one node different warehouses fail with S027.

A connection running a node switches with USE WAREHOUSE only when the
node needs another warehouse than the one the connection is on, and
every switched connection returns to the build's warehouse when the
scheduler finishes. Microbatch sub-workers follow their coordinator,
source loads and Python nodes follow their own node, and code opening
its own connections from connection_config gets the node's warehouse.
A SQL test runs on its models' job warehouse when they agree on one.

A job warehouse is checked like --warehouse: a valid identifier (C261)
on an adapter with a session warehouse (C260), worded for the job. Plan
text, plan JSON and the verbose run context name each job's warehouse.
Server job runs no longer pass a job's warehouse as --warehouse; the
build applies it per node.
…crates (CHI-1533)

The bindings crate was above its code-line cap and its helper layers
imported upward, so the job settings now live with what they build on:

- The job catalog and its entries move from compiled_project into a
  job_settings area of the project crate, with the build-wide settings,
  the conflict and selector errors, the argv a build naming jobs parses
  again, and a frozen run's flags and selectors.
- The per-node settings a plan takes from its named jobs, the build's
  job-raised concurrency and connection count, and the run context's
  job facts move into compile_driver of the compile crate; the
  command-line --warehouse reaches it as a flag.
- The Python node runtime keeps each node's warehouse as a Rust map,
  PythonAwareSelection derives Debug, TokenNodes moves to types.rs, and
  the warehouse tracker, adapter hook and tests follow the keyword-only,
  one-line-docstring and branch-free test rules.

No behaviour changes.
… command line settle conflicts (CHI-1533)

A group's jobs again need to agree only on vars and changed while each
member runs as its own build (D017); the full agreement check returns
with phase 2.

A server job run passes the job's warehouse as --warehouse again, so
setup, planning and bookkeeping run on it. A CLI build whose named jobs
all set the same warehouse treats it as build-wide too.

full_refresh also recreates the functions a job selects, and
ignore_audit_failures also covers singular audits that read the job's
nodes. A setting given on the command line replaces every named job's
before conflicts are checked, and conflict help is worded per scope.

A failed warehouse switch fails the node with its own kind's result:
SQL tests, seeds, sources and functions through the scheduler, and
Python tasks, assets, loaders and checks without aborting their loop;
blocked checks never switch. A SQL test covering nodes with and without
a job warehouse runs on the build warehouse.
…' settings (CHI-1533)

JOB_SETTINGS lists every sqb build option once with its command-line
options, scope (node, build-wide, per-job limit or owned by the run
request), whether Python nodes take it, whether one shared value
becomes the whole run's, and when a group's jobs must agree on it. The
scope decides how named jobs' values merge and which conflict help
applies. The node and Python-node setting lists, the per-job limit,
the group agreement list, the run-request-owned names and the
warehouse exceptions in job run arguments and CLI builds now all read
the table.

resolve_job_settings is the one place named jobs' settings meet the
command line: it removes the settings the command line gives, checks
build-wide agreement, takes whole-run values every named job shares,
and resolves per-node settings when the planner asks. The CLI argv path
and the planner both call it.

Table-driven tests check that the table matches the build parser's
options exactly, that each scope merges, conflicts and yields to the
command line as specified for every setting, that every build-wide
setting renders as its own option, and that every node setting has a
planner consumer.
… help offers only flags that settle it (CHI-1533)

A singular audit reading nodes of an ignoring job and nodes outside the
named jobs now takes the build's ignore_audit_failures for the latter
and is ignored only when every node it reads ignores failures.

Each node setting's row in the job setting table names the plan
consumer that applies it, and the planner reads node settings through
those consumers, so a node row without one never reaches the plan and
fails a test. A test checks that node settings other than the
whole-run warehouse never become job run flags.

Conflict help is generated from the table: it offers a command-line
flag only for the value the flag can give, such as --no-tests to turn
tests off for all, and otherwise points to agreeing jobs or
--no-job-settings.
…I-1533)

--ignore-dynamic-grace, added with dynamic models, is a build-wide job
setting: named jobs must agree on it, the flag can only turn it on, and
it applies to the whole build's dynamic-model holds. The build parser's
namespace defaults and build and plan request fields count both the
dynamic-model options and job_settings.
Job and job group files no longer accept the [on_failure] table or the
retries and retry_delay_seconds keys, and no longer print deprecation
warnings for them. [retry] with attempts and delay_seconds is the only
retry table.

BREAKING CHANGE: a job or job group file that still uses [on_failure],
retries or retry_delay_seconds now fails to compile with an unknown-key
error. Rename [on_failure] to [retry], replace retries = N with
attempts = N + 1, and rename retry_delay_seconds to delay_seconds.
…nore_dynamic_grace at compile, and name [retry] for old [on_failure] tables (CHI-1533)

A dynamic model that sets no refresh_warehouse is now created with the
target's build warehouse, not the warehouse its connection happened to
be on. A job's warehouse, and a --warehouse override, no longer become
a dynamic table's permanent refresh warehouse. When --warehouse
overrides the build, the resolved connection records the target's own
warehouse for this, and Snowflake leaves the key out of the connection.

A job that sets ignore_dynamic_grace = true in a project without
[dynamic] allow_ignore_grace = true now fails to compile at the
setting, instead of failing every run with A719.

An old [on_failure] table in a job or group file now fails with a hint
that names [retry], attempts = retries + 1 and
delay_seconds = retry_delay_seconds.
@kvlonge
kvlonge enabled auto-merge November 19, 2026 12:59
…ENCY (CHI-1533)

SQLBUILD_CONCURRENCY filled the build's concurrency as if --concurrency
were given, which turned off every named job's limit. It now sets the
build's limit as a configured one does, below named jobs, and only a
--concurrency on the command line turns the job limits off.

The offline Snowflake test double records which connection runs each
statement, so a connection's own setup is never counted as a warehouse
switch when connections open in parallel. Tests that assert exact
limits or switch orders clear SQLBUILD_CONCURRENCY, and a new case
builds with it set.
@kvlonge
kvlonge added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 19, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 19, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 559dda5 Nov 19, 2026
35 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/jobs-phase1 branch November 19, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant