Skip to content

Fix ParquetWriterOptions compression level overrides - #1840

Open
JustinBinber wants to merge 1 commit into
apache:mainfrom
JustinBinber:codex/fix-parquet-compression-level
Open

JustinBinber wants to merge 1 commit into
apache:mainfrom
JustinBinber:codex/fix-parquet-compression-level

Conversation

@JustinBinber

Copy link
Copy Markdown

Which issue does this PR close?

Closes #1839

Rationale for this change

ParquetWriterOptions(compression_level=6) currently constructs zstd(3)(6) from the default compression string, so writing fails. Explicit GZIP/Brotli/ZSTD strings with embedded levels have the same problem.

What changes are included in this PR?

  • Replace the embedded level before appending an explicit compression_level, preserving the existing default and signature.
  • Document override precedence with runnable constructor examples.
  • Exercise both public writing entry points with real Parquet files; check codec metadata and readback. Verify that the new override is range-checked and that None with a level still fails.

Validation:

  • Collected 14 new offline regression cases before running them. Before the fix: 11 failed, 3 passed; after the fix: 14 passed.
  • DataFrame tests, DataFrame-module doctests and docstring checks: 411 passed, 2 skipped (optional Polars), 4 unrelated native interruption stress cases deselected.
  • Configured pre-commit checks pass for both files, including Ruff 0.15.1 lint/format, pinned ty 0.0.84, and codespell. git diff --check passes.
  • Tested current checkout Python sources with the released 55.0.0 native wheel, reporting engine 55.2.0, on Windows/Python 3.11.7/PyArrow 26.0.1. Rust was not rebuilt locally.

Are there any user-facing changes?

The separate compression_level now takes precedence over a level embedded in compression. Existing strings are retained when no override is supplied. Native validation still rejects unsupported codec names and invalid new levels. No API signature, default codec, dependency or Rust implementation changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ParquetWriterOptions compression_level fails with the default codec

1 participant