Repository navigation
feat: create new staged Python tables at their first commit; rename a source's schema to destination_schema - #362
Merged
Merged
Conversation
A Python table planned from its data was created on its first batch, and columns that first held values on a later page were added after the _sqlload_* columns. A table whose rows load through a staging table (merge, delete_insert, and an append with checks, logged state or child tables) is now created at its first commit, with every column seen by then and SQLLoad's columns last; its staging table still grows as columns arrive. A first load that fails before its first commit leaves no table behind. Appends written straight into the table and replace builds still create it at the first batch. A column first seen after the table exists is still appended; the docs say so, and that --reload rebuilds it in order.
A source's schema key set where its tables land, while include patterns
and {schema} in table_name name the schema they are read from. The key is
now destination_schema, on TOML sources and on sl.source(...), and copy
--print-toml writes it. plan refuses schema on a source, naming the new
key and include; sl.source(schema=...) fails the same way.
{schema} in table_name and in files paths, include and exclude, and the
target's loader_schema are unchanged.
BREAKING CHANGE: a source's schema key is now destination_schema, in
sqlload.toml and in sl.source(...). A source that still sets schema fails
plan with a message naming the new key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes, one commit each.
A new staged Python table is created at its first commit
A Python table planned from its data was created on its first batch. A column whose values first arrived on a later page was then added with
ADD COLUMN, after_sqlload_load_idand_sqlload_loaded_at. An API whose early pages leave many fields empty could end up with dozens of data columns behind SQLLoad's own, even when the whole read loaded in one commit.A table whose rows go through a staging table anyway is now created at its first commit, with every column seen by then and SQLLoad's columns last:
merge,delete_insert, and anappendwith checks, logged state or child tables, plus a merge parent's child tables. The staging table still grows as columns arrive. A first load that fails before its first commit leaves no table behind.An
appendwritten straight into the table, and areplacebuild table, still create the table at the first batch, so staging their first load costs nothing extra. For those, and for any column first seen after the table exists (a laterbatch_sizewindow, a commit after about 1 GB, a later run), the column is still added after SQLLoad's columns. "Columns SQLLoad adds" in concepts/projects says so, and says that--reloadrebuilds the table with SQLLoad's columns last.A source's
schemakey is nowdestination_schemaOn a source,
schemaset where its tables land, whileincludepatterns and{schema}intable_namename the schema they are read from. The key is nowdestination_schemainsqlload.tomland insl.source(...), andsqlload copy --print-tomlwrites it. There is no alias:schemaon a source failsplanwith[sources.erp]: schema was renamed destination_schema (the schema tables land in); the source schema goes in include, such as "dbo.*".sl.source(..., schema=...)raisessl.source() got schema=; use destination_schema=.{schema}intable_nameand in files paths,include/exclude, and the target'sloader_schemaare unchanged.Tests
mergetable whose columns first hold values on pages 3 and 7 endedid, a, _sqlload_load_id, _sqlload_loaded_at, b, c. It now ends with SQLLoad's two columns, formerge,delete_insertand anappendwith checks. A directappendkeeps the documented order, and a windowed first load appends a column first seen in window 2.mergeor rangedelete_insertparent with child tables, dropped while its child table remains, is created again at its next load's first commit, before its child rows are cleared against it.destination_schemalands a DuckDB source'serp.ordersinraw_erp.erp__orderswithtable_name = "{schema}__{table}". Before the change, both new keys failed as unknown keys. The old key fails with the message above, in TOML and in Python.make verify(fmt, clippy variants, Rust unit, loader container suites and CLI tests) andmake test-python(636 passed, 3 skipped) pass locally on this head.BEGIN_COMMIT_OVERRIDE
feat(loader): create a new staged Python table at its first commit, with SQLLoad's columns last
feat(project): rename a source's schema key to destination_schema
BREAKING CHANGE: a source's
schemakey is nowdestination_schema, in sqlload.toml and insl.source(...). A source that still setsschemafailsplanwith a message naming the new key; the schema tables are read from is still chosen byinclude.END_COMMIT_OVERRIDE