Skip to content

feat: create new staged Python tables at their first commit; rename a source's schema to destination_schema - #362

Merged
kvlonge merged 2 commits into
mainfrom
data-planned-create-at-commit
Nov 20, 2026
Merged

kvlonge merged 2 commits into
mainfrom
data-planned-create-at-commit

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Two changes, one commit each.

A new staged Python table is created at its first commit

A Python table planned from its data was created on its first batch. A column whose values first arrived on a later page was then added with ADD COLUMN, after _sqlload_load_id and _sqlload_loaded_at. An API whose early pages leave many fields empty could end up with dozens of data columns behind SQLLoad's own, even when the whole read loaded in one commit.

A table whose rows go through a staging table anyway is now created at its first commit, with every column seen by then and SQLLoad's columns last: merge, delete_insert, and an append with checks, logged state or child tables, plus a merge parent's child tables. The staging table still grows as columns arrive. A first load that fails before its first commit leaves no table behind.

An append written straight into the table, and a replace build table, still create the table at the first batch, so staging their first load costs nothing extra. For those, and for any column first seen after the table exists (a later batch_size window, a commit after about 1 GB, a later run), the column is still added after SQLLoad's columns. "Columns SQLLoad adds" in concepts/projects says so, and says that --reload rebuilds the table with SQLLoad's columns last.

A source's schema key is now destination_schema

On a source, schema set where its tables land, while include patterns and {schema} in table_name name the schema they are read from. The key is now destination_schema in sqlload.toml and in sl.source(...), and sqlload copy --print-toml writes it. There is no alias:

  • schema on a source fails plan with [sources.erp]: schema was renamed destination_schema (the schema tables land in); the source schema goes in include, such as "dbo.*".
  • sl.source(..., schema=...) raises sl.source() got schema=; use destination_schema=.

{schema} in table_name and in files paths, include/exclude, and the target's loader_schema are unchanged.

Tests

  • On main, a fresh merge table whose columns first hold values on pages 3 and 7 ended id, a, _sqlload_load_id, _sqlload_loaded_at, b, c. It now ends with SQLLoad's two columns, for merge, delete_insert and an append with checks. A direct append keeps the documented order, and a windowed first load appends a column first seen in window 2.
  • A merge or range delete_insert parent with child tables, dropped while its child table remains, is created again at its next load's first commit, before its child rows are cleared against it.
  • destination_schema lands a DuckDB source's erp.orders in raw_erp.erp__orders with table_name = "{schema}__{table}". Before the change, both new keys failed as unknown keys. The old key fails with the message above, in TOML and in Python.

make verify (fmt, clippy variants, Rust unit, loader container suites and CLI tests) and make test-python (636 passed, 3 skipped) pass locally on this head.

BEGIN_COMMIT_OVERRIDE
feat(loader): create a new staged Python table at its first commit, with SQLLoad's columns last

feat(project): rename a source's schema key to destination_schema

BREAKING CHANGE: a source's schema key is now destination_schema, in sqlload.toml and in sl.source(...). A source that still sets schema fails plan with a message naming the new key; the schema tables are read from is still chosen by include.
END_COMMIT_OVERRIDE

A Python table planned from its data was created on its first batch, and
columns that first held values on a later page were added after the
_sqlload_* columns. A table whose rows load through a staging table (merge,
delete_insert, and an append with checks, logged state or child tables)
is now created at its first commit, with every column seen by then and
SQLLoad's columns last; its staging table still grows as columns arrive.
A first load that fails before its first commit leaves no table behind.

Appends written straight into the table and replace builds still create
it at the first batch. A column first seen after the table exists is
still appended; the docs say so, and that --reload rebuilds it in order.
A source's schema key set where its tables land, while include patterns
and {schema} in table_name name the schema they are read from. The key is
now destination_schema, on TOML sources and on sl.source(...), and copy
--print-toml writes it. plan refuses schema on a source, naming the new
key and include; sl.source(schema=...) fails the same way.

{schema} in table_name and in files paths, include and exclude, and the
target's loader_schema are unchanged.

BREAKING CHANGE: a source's schema key is now destination_schema, in
sqlload.toml and in sl.source(...). A source that still sets schema fails
plan with a message naming the new key.
@kvlonge
kvlonge merged commit 4ace960 into main Nov 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant