Skip to content

Don't decrypt pages behind the end of the database file - #316

Merged
utelle merged 2 commits into
utelle:mainfrom
SchwarzDigits:fix/shortread-autovacuum
Nov 21, 2026
Merged

utelle merged 2 commits into
utelle:mainfrom
SchwarzDigits:fix/shortread-autovacuum

Conversation

@mtrossbach

Copy link
Copy Markdown
Contributor

In 2.7.0 and 2.7.1, creating a table in an encrypted database with auto_vacuum fails with "database disk image is malformed" when the same transaction has already grown the database:

PRAGMA cipher='chacha20';
PRAGMA key='pass';
PRAGMA auto_vacuum=INCREMENTAL;
CREATE TABLE first (v);
BEGIN;
INSERT INTO first VALUES (randomblob(20000));
CREATE TABLE t0 (a BLOB PRIMARY KEY);
COMMIT;

This happens with sqlcipher, chacha20, ascon128 and aegis, with auto_vacuum FULL and INCREMENTAL, and in every journal mode. Unencrypted databases and the AES ciphers, which have no MAC, are not affected.

To make room for the new root page, SQLite moves the page at that position to the end (relocatePage -> sqlite3PagerMovepage). Afterwards it loads the old position again only to set a flag, and that position lies behind the end of the file. The real VFS returns zeros with SQLITE_IOERR_SHORT_READ, which is what SQLite expects there. mcIoRead passes this read on to mcReadMainDb, which decrypts the zero-filled page, and the MAC check fails.

With this change, mcReadMainDb leaves pages that start at or behind the end of the file zero-filled and does not decrypt them; mcIoRead still returns SQLITE_IOERR_SHORT_READ for them. A page that is only partly in the file is still decrypted, so a truncated file is still reported as corrupt. Returning the short read right away would be simpler, but then a page that is only partly in the file would reach SQLite as raw ciphertext, without the MAC check.

crashtest gets a new operation "autovacuum" (part of the default run): a small database with auto_vacuum=INCREMENTAL, then a transaction that adds a large row, creates a table and deletes rows, followed by PRAGMA incremental_vacuum. Without the fix it fails for every cipher with a MAC; with the fix it passes, including the runs with crashes, power loss and I/O errors.

Not covered here: with SQLITE_FCNTL_CHUNK_SIZE the file is longer than the database, so such a page lies inside the file and arrives as zeros without a short read. That fails the same way, and not only since 2.7.0. Treating every all-zero page as empty would fix it, but then a zeroed last overflow page goes unnoticed: the blob changes and integrity_check still says ok. I left it alone for now.

Testing (macOS arm64): cryptotest, tempfiletest (all four variants), oomtest, rekeytest, threadtest, crashtest (default and full autovacuum), vfsstacktest. The reproduction above passes for all six ciphers and unencrypted, with auto_vacuum none/full/incremental and journal modes delete/wal/memory, and for chacha20 and sqlcipher also with page sizes from 512 to 65536. A random auto_vacuum stress test (inserts, deletes, CREATE/DROP TABLE, incremental_vacuum, rollbacks, small and large cache) passes as well.

SQLite reads pages behind the end of the file, for example when it moves
pages in auto-vacuum mode, and expects zeros there. Since such a read is
passed on to mcReadMainDb, the zero-filled page was decrypted, and ciphers
with a MAC reported SQLITE_CORRUPT. Pages that start at or behind the end
of the file now stay zero-filled. A page that is only partly in the file
is still decrypted, so that a truncated file is reported as corrupt.
A small database with auto_vacuum=INCREMENTAL, then a transaction that adds
a large row, creates a table and deletes rows, followed by an incremental
vacuum. Creating the table moves a page whose old position lies behind the
end of the file.
@utelle
utelle merged commit 4661531 into utelle:main Nov 21, 2026
13 checks passed
@utelle

utelle commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Thanks for detecting this issue and providing a fix.

Not covered here: with SQLITE_FCNTL_CHUNK_SIZE the file is longer than the database, so such a page lies inside the file and arrives as zeros without a short read. That fails the same way, and not only since 2.7.0. Treating every all-zero page as empty would fix it, but then a zeroed last overflow page goes unnoticed: the blob changes and integrity_check still says ok. I left it alone for now.

To handle this correctly, we'd probably have to figure out how SQLite itself distinguishes between these cases.

utelle added a commit that referenced this pull request Nov 21, 2026
Added check for all-zero pages with SQLITE_FCNTL_CHUNK_SIZE
Added test case for autovacuum-chunksize to crashtest

Related to #316
@utelle

utelle commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Commit 076bba7 tries to handle the SQLITE_FCNTL_CHUNK_SIZE case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants