Repository navigation
perf(import): send a SQL import's writes to PostgreSQL in pipelined batches - #4356
Merged
Merged
Conversation
…atches inside its transaction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PluginDatabaseDriver.executeStatements(_:)(default nil, so a driver keeps sending one statement at a time) andPluginCapabilities.statementBatches. Nov*tag ships kit 46 yet (v0.88.0-beta.1 has 45).SET,COPY,CALL,DOand any text the shared SQL scanner reads as two statements under the session'sstandard_conforming_stringsrun alone. A pipeline that ends before its Sync drops the connection.PluginDriverAdapterchecks each statement as its own call would before any of them is sent.INSERT,UPDATE,DELETE,CREATE,ALTER,DROP,COMMENT,GRANT,REVOKEandTRUNCATEtext holding no backslash and noset_config. Anything else flushes the batch and runs alone.standard_conforming_strings, and when a batch changed that setting the import stops at the statement it had read past (rolled back under Stop and Rollback, earlier statements committed under Stop and Commit). A read failure after a batch still rolls back in every mode.Measured on PostgreSQL 17.11 through a 200 ms RTT proxy,
SQLImportPluginand the PostgreSQL driver from each tree:INSERTpg_dump --insertsof 3 tables, 638 statementsINSERT, duplicate key at statement 26The dump re-imported byte for byte the same as its source both ways.
Test plan
SQLImportBatchTests: batching, every error mode, Stop during the last batch, lone statements, backslash text, changed reading rules, read failures, bounded batches.ImportDataSinkAdapterBatchTests: what may batch, declined batches, changed and restored reading rules.LibPQPipelineTests(scripted server): counts, failure stops the run, row-by-row reads, lost Sync, admission by the shared scanner.PluginDriverAdapterStatementBatchTests: host checks per statement, missing answers.standard_conforming_stringsoff and on, a function that flips it inside a batch (stops at the next statement, no injected row), flip and flip back (same result as main),INSERT ... RETURNING5,000 rows.