Skip to content

perf(import): send a SQL import's writes to PostgreSQL in pipelined batches - #4356

Merged
datlechin merged 4 commits into
mainfrom
perf/sql-import-batches
Nov 19, 2026
Merged

datlechin merged 4 commits into
mainfrom
perf/sql-import-batches

Conversation

@datlechin

@datlechin datlechin commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

  • P16: PluginKit kit 46 adds PluginDatabaseDriver.executeStatements(_:) (default nil, so a driver keeps sending one statement at a time) and PluginCapabilities.statementBatches. No v* tag ships kit 46 yet (v0.88.0-beta.1 has 45).
  • P16: PostgreSQL runs a batch as one libpq pipeline, only inside a transaction block, so the first failure aborts everything after it. Rows a statement returns are read one at a time and dropped. Transaction control, SET, COPY, CALL, DO and any text the shared SQL scanner reads as two statements under the session's standard_conforming_strings run alone. A pipeline that ends before its Sync drops the connection.
  • P16: PluginDriverAdapter checks each statement as its own call would before any of them is sent.
  • P17: SQL import batches only inside its own transaction (Stop and Rollback, Stop and Commit), up to 200 statements or 4 MiB, and only INSERT, UPDATE, DELETE, CREATE, ALTER, DROP, COMMENT, GRANT, REVOKE and TRUNCATE text holding no backslash and no set_config. Anything else flushes the batch and runs alone.
  • P17: the file is read one statement ahead of a batch. Text without a backslash splits the same under either standard_conforming_strings, and when a batch changed that setting the import stops at the statement it had read past (rolled back under Stop and Rollback, earlier statements committed under Stop and Commit). A read failure after a batch still rolls back in every mode.
  • Data Sync, grid saves, MySQL and MariaDB stay sequential.
  • Docs: docs(features): note batched PostgreSQL SQL imports docs#253, merged after this one.

Measured on PostgreSQL 17.11 through a 200 ms RTT proxy, SQLImportPlugin and the PostgreSQL driver from each tree:

Import main this branch
50 INSERT 10,649 ms 623 ms
pg_dump --inserts of 3 tables, 638 statements 131,161 ms 8,799 ms
50 INSERT, duplicate key at statement 26 5,715 ms, line 26, 0 rows 632 ms, line 26, 0 rows

The dump re-imported byte for byte the same as its source both ways.

Test plan

  • SQLImportBatchTests: batching, every error mode, Stop during the last batch, lone statements, backslash text, changed reading rules, read failures, bounded batches.
  • ImportDataSinkAdapterBatchTests: what may batch, declined batches, changed and restored reading rules.
  • LibPQPipelineTests (scripted server): counts, failure stops the run, row-by-row reads, lost Sync, admission by the shared scanner.
  • PluginDriverAdapterStatementBatchTests: host checks per statement, missing answers.
  • On PostgreSQL 17.11: duplicate key in both stop modes, a file that turns standard_conforming_strings off and on, a function that flips it inside a batch (stops at the next statement, no injected row), flip and flip back (same result as main), INSERT ... RETURNING 5,000 rows.
  • No existing test changed.

@datlechin
datlechin enabled auto-merge (squash) November 19, 2026 18:04
@datlechin
datlechin merged commit 0b0c724 into main Nov 19, 2026
15 checks passed
@datlechin
datlechin deleted the perf/sql-import-batches branch November 19, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant