Repository navigation
Don't decrypt pages behind the end of the database file - #316
Merged
Merged
Conversation
SQLite reads pages behind the end of the file, for example when it moves pages in auto-vacuum mode, and expects zeros there. Since such a read is passed on to mcReadMainDb, the zero-filled page was decrypted, and ciphers with a MAC reported SQLITE_CORRUPT. Pages that start at or behind the end of the file now stay zero-filled. A page that is only partly in the file is still decrypted, so that a truncated file is reported as corrupt.
A small database with auto_vacuum=INCREMENTAL, then a transaction that adds a large row, creates a table and deletes rows, followed by an incremental vacuum. Creating the table moves a page whose old position lies behind the end of the file.
Owner
|
Thanks for detecting this issue and providing a fix.
To handle this correctly, we'd probably have to figure out how SQLite itself distinguishes between these cases. |
utelle
added a commit
that referenced
this pull request
Nov 21, 2026
Added check for all-zero pages with SQLITE_FCNTL_CHUNK_SIZE Added test case for autovacuum-chunksize to crashtest Related to #316
Owner
|
Commit 076bba7 tries to handle the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In 2.7.0 and 2.7.1, creating a table in an encrypted database with auto_vacuum fails with "database disk image is malformed" when the same transaction has already grown the database:
This happens with sqlcipher, chacha20, ascon128 and aegis, with auto_vacuum FULL and INCREMENTAL, and in every journal mode. Unencrypted databases and the AES ciphers, which have no MAC, are not affected.
To make room for the new root page, SQLite moves the page at that position to the end (relocatePage -> sqlite3PagerMovepage). Afterwards it loads the old position again only to set a flag, and that position lies behind the end of the file. The real VFS returns zeros with SQLITE_IOERR_SHORT_READ, which is what SQLite expects there. mcIoRead passes this read on to mcReadMainDb, which decrypts the zero-filled page, and the MAC check fails.
With this change, mcReadMainDb leaves pages that start at or behind the end of the file zero-filled and does not decrypt them; mcIoRead still returns SQLITE_IOERR_SHORT_READ for them. A page that is only partly in the file is still decrypted, so a truncated file is still reported as corrupt. Returning the short read right away would be simpler, but then a page that is only partly in the file would reach SQLite as raw ciphertext, without the MAC check.
crashtest gets a new operation "autovacuum" (part of the default run): a small database with auto_vacuum=INCREMENTAL, then a transaction that adds a large row, creates a table and deletes rows, followed by PRAGMA incremental_vacuum. Without the fix it fails for every cipher with a MAC; with the fix it passes, including the runs with crashes, power loss and I/O errors.
Not covered here: with SQLITE_FCNTL_CHUNK_SIZE the file is longer than the database, so such a page lies inside the file and arrives as zeros without a short read. That fails the same way, and not only since 2.7.0. Treating every all-zero page as empty would fix it, but then a zeroed last overflow page goes unnoticed: the blob changes and integrity_check still says ok. I left it alone for now.
Testing (macOS arm64): cryptotest, tempfiletest (all four variants), oomtest, rekeytest, threadtest, crashtest (default and
full autovacuum), vfsstacktest. The reproduction above passes for all six ciphers and unencrypted, with auto_vacuum none/full/incremental and journal modes delete/wal/memory, and for chacha20 and sqlcipher also with page sizes from 512 to 65536. A random auto_vacuum stress test (inserts, deletes, CREATE/DROP TABLE, incremental_vacuum, rollbacks, small and large cache) passes as well.