Skip to content

chore(deps): bump DataFusion to 56.0.0-rc2 - #120

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:df-56-rc2
Draft

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:df-56-rc2

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A — validates the DataFusion 56.0.0 release candidate against this binding.

Rationale for this change

Tests the binding against 56.0.0-rc2 ahead of the release, so any breakage gets reported upstream during the vote and the final 56.0.0 bump is just a version change. Draft: this depends on a git tag and should not merge in this form. Once 56.0.0 is published, the git deps go back to crates.io versions.

What changes are included in this PR?

  • datafusion, datafusion-proto, datafusion-spark and datafusion-substrait now point at git = "https://github.com/apache/datafusion", tag = "56.0.0-rc2". arrow moves from 59 to 60 and object_store from 0.13 to 0.14, matching what DataFusion 56 pulls in.
  • <datafusion.version> is set to 56.0.0-rc2, so the .proto files are downloaded from the same tag. Both <sha512> digests are updated.

Changes needed for the upstream API changes:

  • TableProvider::scan takes Option<&[usize]> for the projection, not Option<&Vec<usize>>. JavaTableProvider is updated to match.
  • ParquetOptions.compression is now Option<DFParquetCompression>, not a String. writeParquet parses the caller's string with DFParquetCompression::from_str. A side effect: an invalid codec or level is now rejected before writing starts, with a Configuration error, instead of failing when the first file is written.
  • ArrowArrayStreamReader was renamed to RecordBatchStreamReader in arrow 60. The old name is a deprecated alias, and it would fail clippy -D warnings.

Are these changes tested?

Covered by the existing suites:

  • ./mvnw test: 349 tests, 0 failures, 13 skipped (TPC-H data-gated tests; no local TPC-H data).
  • cargo test --workspace --features substrait,spark: passing.
  • cargo clippy --all-targets --workspace --features substrait,spark -- -D warnings: clean.
  • cargo fmt --all and ./mvnw spotless:apply: clean.

Are there any user-facing changes?

An invalid Parquet compression string passed to writeParquet now fails earlier, at configuration time. Otherwise there are no API changes.

🤖 Generated with Claude Code

Points the DataFusion workspace dependencies at the 56.0.0-rc2 git tag
to validate the release candidate, bumps arrow to 60 and object_store
to 0.14 to match, and adapts to upstream API changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant