Skip to content

GH-3817: Content defined chunking of data pages - #3818

Open
kszucs wants to merge 12 commits into
apache:masterfrom
kszucs:cdc-pages
Open

kszucs wants to merge 12 commits into
apache:masterfrom
kszucs:cdc-pages

Conversation

@kszucs

@kszucs kszucs commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

Content defined chunking (CDC) ends data pages at boundaries derived from the values, so files that share runs of values share pages, which content-addressable storage (e.g. Hugging Face Xet) can deduplicate. Arrow C++ and arrow-rs already write such files; Iceberg needs it for its write.parquet.content-defined-chunking.* table properties.

What changes are included in this PR?

Opt-in and EXPERIMENTAL. Readers are unchanged, and without CDC the output is byte-identical to master.

  • ContentDefinedChunker ports the Arrow C++ / arrow-rs chunker, so pages end where Arrow's do for the same values.
  • withContentDefinedChunking(boolean) and setters for the minimum and maximum chunk size and the normalization level on the ParquetProperties and ParquetWriter builders, and the matching parquet.content-defined-chunking.* keys in ParquetOutputFormat.
  • ColumnWriterBase asks the chunker before each value whether to end the page; the page limits apply as before.
  • Chunkers live per file and continue across row groups, as in Arrow (GH-51684: [C++][Parquet] Keep the CDC chunking state across row groups arrow#51685), through a new newColumnWriteStore(..., chunkers) overload. Integrations that build column write stores themselves, such as Iceberg, must create the chunkers once per file.
  • CDC adds about 15% write time where hashing dominates (ContentDefinedChunkingWriteBenchmarks).

Pages differ from Arrow's for values Arrow holds in another form (DECIMAL, INT96, dictionary arrays, ...) and for page size cuts inside a chunk. Dictionary fallback is unchanged: parquet-java decides on a column chunk's first page, which chunking can make short, so some column chunks fall back to plain encoding; that is left for a follow-up.

Are these changes tested?

Yes: a port of Arrow C++'s parquet/chunker_internal_test.cc (edit tests over 29 column types, V1 and V2 pages, multiple row groups), two tests comparing page lengths with those an Arrow C++ build writes, and one that the page size checks of ParquetWriter move no chunk boundary.

Are there any user-facing changes?

New EXPERIMENTAL writer options and configuration keys, documented in parquet-hadoop/README.md, and new public API: those builder setters with their ParquetProperties getters and defaults, ContentDefinedChunker.forColumns and calculateMask, and the newColumnWriteStore overload and ColumnWriteStoreV1/V2 constructors that take chunkers.

Changes since the last review

  • The chunker only decides chunk boundaries; the page limits apply as without CDC.
  • CdcOptions is replaced by builder setters, one per configuration key, and build() rejects an invalid size range.
  • INT(8) and INT(16) values are hashed at 1 and 2 bytes, as Arrow holds them.
  • The tests port Arrow C++'s suite.
  • The differences from Arrow are documented.

Closes #3817

Generated with Claude Code

@kszucs

kszucs commented Oct 25, 2026 •

Copy link
Copy Markdown
Member Author

I'm not sure who to ping about reviews, but @sunchao you are probably familiar with the CDC logic we have in the C++/Rust implementations. Could you please take a preliminary look and/or ping other maintainers? I essentially tried to port the libarrow implementation :)

@kszucs

kszucs commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

@wgtmac would appreciate if you could take a look as well

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed e4199d8a48ea19e6f56889e13ce161a46c8c6b94 with five focused agent reviews, followed by local validation. I found no actionable correctness issues in this revision.

This looks useful for publishing successive versions of Parquet datasets that share most of their data: for example, correcting labels, inserting records, or appending batches. Choosing page boundaries from the content helps unchanged runs of values land in the same pages after an edit. Doing this before encoding and compression gives a deduplicating storage layer such as Xet more opportunities to recognize identical byte chunks. The uploader can then send missing chunks and the metadata needed to reconstruct the new file, reusing pieces already stored. The writer still produces a replacement Parquet file; the potential savings are in upload bandwidth and storage across versions.

This brings the Java writer an approach already implemented in Arrow C++/PyArrow and arrow-rs, both merged. Following that design and testing against Arrow-generated examples is sensible. Keeping the feature experimental and disabled by default, while producing ordinary Parquet files that existing readers can read, is appropriate.

The review covered:

  • Algorithm: primitive byte representations, floating-point raw bits, binary slices/direct buffers, definition/repetition levels, minimum-size skipping, pending matches, maximum-size cuts, and mask validation. All 2,048 gear-table entries matched Arrow's constants.
  • Writer integration: row-aligned page boundaries, page size/value/row limits, cached nulls, statistics and counters, V1/V2 page/index handling, dictionaries, failure propagation, and continuation of chunker state across row groups.
  • API and compatibility: immutable options, properties copying, explicit null-disable behavior, Hadoop configuration, existing constructor paths, the disabled path, and the requirement for direct integrations to retain chunkers across row groups for the whole file.
  • Test quality: Arrow golden vectors, edit-locality assertions, actual file round trips, configuration/fail-fast cases, and the distinction between independent expectations and helper tests derived from the implementation.

Local validation on the reviewed head passed with JDK 17:

Test class Tests Failures / errors / skips
TestContentDefinedChunker 29 0 / 0 / 0
TestColumnWriterChunking 24 0 / 0 / 0
TestContentDefinedChunking 12 0 / 0 / 0
TestParquetProperties 16 0 / 0 / 0

That is 81 tests passed, with no skips.

A supplemental probe passed 128 full/filtered readbacks across 16 writer combinations: V1/V2, uncompressed/GZIP, dictionary enabled/disabled, and encrypted/plain files. It exercised nested and sparse data, oversized values/repeated records, and multiple row groups. The probe checked that page indexes actually reduced candidate rows, that encrypted cases encrypted all columns, and that dictionary-enabled cases actually used a dictionary.

A separate standalone Java probe passed 900,000 mask-arithmetic checks against an independent BigInteger calculation, including extreme size envelopes and normalization levels. This was an arithmetic check, separate from the actual writer/readback tests.

The full JDK 17 CI job also executed the four test classes successfully. Its checkout was a synthetic merge containing this head and an older base commit.

Validation limits and practical qualifications:

  • The local build disabled the optional non-aarch64 profile because its Brotli runtime dependency was unavailable from the configured artifact mirror. Local compression coverage here was uncompressed/GZIP.
  • I did not independently reproduce the reported 10–30% write overhead or measure end-to-end storage savings.
  • Reuse depends on the workload, data ordering, writer/encoding/compression settings, and downstream deduplication. Dictionary changes can also affect byte reuse. Cross-language boundary equality requires matching the bytes fed to the chunker; equal logical values alone are insufficient.
  • I did not test a merge with the current target-base commit db9e61050651c2cb808abc05297a0033552850bc.

Within that scope, the design and implementation look sound to me.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Approving reviewed revision e4199d8a48ea19e6f56889e13ce161a46c8c6b94 following the comprehensive review and validation. No actionable correctness issues found. The review documents the use case, existing C++/Rust implementations, and validation limits.

@wgtmac

wgtmac commented Oct 29, 2026

Copy link
Copy Markdown
Member

I haven't had the time to review its detail yet. A full round of agent review reported that this PR is solid. As this is a pretty new feature and default production path is not affected, it generally looks good to me. WDYT? @gszadovszky @Fokko

@gszadovszky gszadovszky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The low-level API implementation looks good. However, the higher-level MessageColumnIO path introduces additional page flushes through ColumnWriteStore.isColumnFlushNeeded(). These flushes are driven by the column-store record-count check schedule, independently of CDC boundaries, and can add fixed page boundaries even when CDC has not requested one.

The PR includes high-level writer tests, but does not explicitly test or document this interaction with the periodic flush schedule. Please add an assertion covering page boundaries through ParquetWriter with the default record-count checks, or clarify that exact CDC boundaries are guaranteed only for the low-level API.

@kszucs

kszucs commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Thanks for the reviews, addressing them!

@kszucs
kszucs force-pushed the cdc-pages branch 2 times, most recently from 177b1f9 to ef414ba Compare November 9, 2026 08:49

@gszadovszky gszadovszky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update @kszucs. It looks good to me.

@kszucs

kszucs commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

@gszadovszky please hold on with merging, polishing a little + doing some parity checks against existing imlementations.

@gszadovszky

Copy link
Copy Markdown
Contributor

I've planned to wait another 24 hours to merge, but I'll wait for your green light.

Add opt-in content defined chunking (CDC) to the writer. Data page
boundaries are chosen by a rolling gear hash over each column's
(definition level, repetition level, value) stream instead of by
position, so an insert or delete only changes the pages around it and
files deduplicate in content addressable storage.

The chunker is a port of Arrow C++ (parquet/chunker_internal) and
arrow-rs (column/chunker/cdc.rs): same gear tables, mask derivation,
defaults and cut rules. Each column keeps one chunker for the whole
file, as arrow-rs does, so chunk boundaries continue across row groups
instead of restarting in each. Arrow C++ builds a new chunker for every
row group, contrary to its own documentation, so its chunks match ours
in the first row group but generally not after it.

- CdcOptions: min/max chunk size and normalization level, defaulting
  to 256 KiB / 1 MiB / 0 as in Arrow.
- ParquetProperties and ParquetWriter.Builder:
  withContentDefinedChunkingEnabled(boolean) and
  withContentDefinedChunking(CdcOptions).
- ParquetOutputFormat: parquet.page.content-defined-chunking.* keys.
- ChunkingColumnWriter wraps the column writer only when chunking is
  enabled. It ends pages at chunk boundaries and, as Arrow C++ does,
  applies the page size and row count limits counted from the page
  start, so they cut a chunk in the same places after an edit.
- The writer keeps each column's chunker for the whole file, in
  CdcChunkers, which makes every row group's store.
- With chunking on, a column falls back from dictionary encoding only
  when its dictionary outgrows its size limit, as in Arrow C++, not on
  its first page, which chunking cuts by content. Pages of nulls alone
  get an empty dictionary page, as Arrow C++ writes one.
- CdcWriteBenchmarks measures the write cost.

Off by default, and with it off the writer behaves as before.
- Chunker state lives per file: InternalParquetRecordWriter creates
  the chunkers of the file up front, one per leaf column, only when
  chunking is enabled, as Arrow C++ and arrow-rs do, and passes them
  to every row group's store through a new
  ParquetProperties.newColumnWriteStore overload. With chunking
  enabled, a store made without them fails fast rather than silently
  restarting the chunking. CdcOptions are validated when built.
- Content defined chunking is configured through one setter,
  withContentDefinedChunking(CdcOptions), where null disables it, and
  named as Arrow names it: CdcOptions, ContentDefinedChunker.
- ChunkingColumnWriter wraps each column writer once, when the store is
  built. It rolls every triplet into the column's chunker and, at record
  starts, ends the page where needNewChunk() says, as Arrow C++ does, or
  where a page limit is reached, counted from the page start; cached
  nulls are flushed on the unchunked schedule's maximum.
- Binary values are hashed through one path.
- The tests fail when chunking is switched off or broken, cover the
  cases the Arrow C++ and arrow-rs suites do (V2 pages, dictionary
  encoding, edits across row groups, a chunker per column, the record
  writer path), and follow the project's test guidelines. The benchmark
  writes one mixed dataset.
ColumnWriterBase carries its column's ContentDefinedChunker, passed in
by the store as the bloom filter writer is, instead of being wrapped by
ChunkingColumnWriter. Without content defined chunking it gets a no-op
chunker, as it gets no-op statistics, which the JIT inlines away, so
the write path has no added branch.

ContentDefinedChunker follows Arrow C++: the column writer rolls each
triplet and asks needNewPage() at record starts, built on Roll-like
primitives and needNewChunk(). ContentDefinedChunker.forColumns takes
the writer properties, for the page limits. Written files are
unchanged.
The column writer asks needNewPage(value, repetitionLevel,
definitionLevel, writer) before writing each triplet and ends the page
when it says so, instead of rolling the value itself and then asking.
Rolling is the content defined chunker's own business, so another
chunker can answer the same question its own way, and the no-op
chunker inlines to a single false.
…oundary

Under content defined chunking each column writer applies the page limits
at every record start, so the column write store's page size check schedule,
which also decides when MessageColumnIO hands its held back nulls to the
column writers, must not move a page boundary. Test it through ParquetWriter
and document it.
…nstructors

The column writers are package private, so their constructors can take the
chunker instead of gaining overloads that leave the old ones unused.
…ilter writer

The column writer resolves it to the no-op chunker once, as the value collector
resolves a missing bloom filter writer to a no-op bloom filter.
- The chunker only decides chunk boundaries, as in Arrow C++; the page
  size and row count limits apply as they do without chunking.
- CdcOptions is renamed to ContentDefinedChunkingOptions. It rejects an
  invalid size range, or one too narrow for the normalization level,
  when built, before a writer opens its file. A writer without chunking
  holds no chunker.
- INT(8) and INT(16) values are hashed at 1 and 2 bytes, the width
  Arrow holds them in.
- ParquetOutputFormat gains setContentDefinedChunking and
  getContentDefinedChunkingOptions, like its other properties.
- The tests port Arrow C++'s parquet/chunker_internal_test.cc: its
  generated tables, its single and multiple row group edit tests, its
  page length differences and chunk size bounds, and its parameter and
  mask tests. Two tests check page lengths against those an Arrow C++
  build writes.
- The differences from Arrow are documented, including where dictionary
  encoding falls back to plain on a short first page.
…chedule

- The docstrings give examples instead of prose, and state the
  differences from Arrow once.
- A test checks that the page size checks of ParquetWriter, which also
  flush held back nulls, move no chunk boundary.
…er properties

ContentDefinedChunkingOptions is gone. ParquetProperties.Builder and
ParquetWriter.Builder take withContentDefinedChunking(boolean) and a
setter for each chunk size setting, one per Hadoop key, each with its
own default, as the bloom filter settings do.

While chunking is enabled, build() rejects an invalid size range through
calculateMask, which now takes Arrow C++'s three arguments and checks the
range as Arrow C++ does. ParquetOutputFormat reads the keys directly.
…unking.*

Content defined chunking is a writer setting rather than a page one,
although it ends pages.
ParquetOutputFormat parsed the chunk size keys even with chunking off, so
a value getLong does not accept, such as "8m", failed every write,
although the README says the keys are read only while chunking is
enabled.

A test also covers a chunk ending before the first value of a page,
which must end no page: without the guard, that write fails with
"writing empty page".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support content-defined chunking (CDC) in the Parquet writer

4 participants