Skip to content

refactor(catalog): return Result from fallible CatalogProvider and SchemaProvider methods - #27644

Open
gratus00 wants to merge 1 commit into
apache:mainfrom
gratus00:catalog-schema-result-v2
Open

gratus00 wants to merge 1 commit into
apache:mainfrom
gratus00:catalog-schema-result-v2

Conversation

@gratus00

Copy link
Copy Markdown

SchemaProvider::table_names and SchemaProvider::table_exist now return Result. Catalogs backed by remote services can report lookup failures (timeouts, auth errors) instead of returning an empty list, None or false, which callers could not tell apart from nothing existing. Closes #21778.

Important

Fork CI setup: For pull requests from a fork to main, enable workflows in your fork's Actions tab before you push.
See the fork CI setup guide.

Which issue does this PR close?

Rationale for this change

Catalogs backed by a remote service (for example paimon-rust or delta-rs Unity Catalog) have to make network calls when DataFusion asks which schemas or tables exist. When such a call fails (403, timeout, connection refused), the current trait signatures leave the implementation no way to report it: it has to return an empty list, None, or false. To the user, a failed request then looks the same as an empty catalog or a missing table, which makes these failures very hard to diagnose.

What changes are included in this PR?

These trait methods now return Result:

Trait Method Before After
CatalogProvider schema_names() Vec<String> Result<Vec<String>>
CatalogProvider schema() Option<Arc<dyn SchemaProvider>> Result<Option<Arc<dyn SchemaProvider>>>
SchemaProvider table_names() Vec<String> Result<Vec<String>>
SchemaProvider table_exist() bool Result<bool>

The change is carried through:

  • the trait definitions in datafusion-session
  • the built-in providers: memory, listing, dynamic file, async/resolved, information schema
  • SessionState / SessionContext catalog resolution, where errors now propagate with ?
  • datafusion-cli, examples, and sqllogictest test providers
  • the FFI wrappers: FFI_CatalogProvider / FFI_SchemaProvider function pointers now return FFI_Result<...>, so errors cross the FFI boundary instead of being dropped. This changes the FFI struct layout.
  • the catalogs.md user guide examples, which are compiled as doctests
  • a new section in upgrading/57.0.0.md describing the migration

Design note: errors propagate, including in information_schema

A missing schema or table is still Ok(None) / Ok(false) and produces the usual "not found" error. Only a catalog that cannot answer returns Err, and that error is propagated with ?.

One trade-off for reviewers: the information_schema tables (and SHOW TABLES) enumerate every registered catalog, so with this PR one unreachable catalog makes the whole listing query fail, where before it would silently appear empty.

The alternative is to skip the failing catalog/schema with a log::warn! and keep listing the others. I chose to propagate because skipping reintroduces the original problem in a narrower form: the user gets an incomplete result with nothing in the result itself saying so. The skip-and-warn variant would only touch the four add_* helpers in information_schema.rs, so it is an easy follow-up if reviewers prefer it.

What is the testing strategy for this PR?

  • Existing catalog, information schema, and create_drop tests are updated for the new signatures.
  • datafusion/ffi/src/catalog_provider.rs: test_ffi_catalog_provider_propagates_failures checks that an error raised by the producer reaches the consumer through the FFI wrapper.
  • datafusion/ffi/integration-tests/tests/cases/catalog.rs: new test_catalog_schema_lookups calls all four methods on a catalog loaded from the separately compiled cdylib, covering both found and not-found cases.
  • catalogs.md doctests pass with the new signatures.

Run locally: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, the FFI unit and integration tests, the catalogs.md doctests, ./ci/scripts/doc_prettier_check.sh, and the extended workspace test suite.

Are there any user-facing changes?

No change to SQL or CLI behavior when catalogs are healthy. When a catalog's lookup fails, the query now returns that error instead of treating the catalog as empty or the table as missing.

This is a breaking public API change for implementers and callers of CatalogProvider / SchemaProvider, and it changes the datafusion-ffi ABI, so it needs the api change label. Migration notes are in docs/source/library-user-guide/upgrading/57.0.0.md.

@github-actions github-actions Bot added documentation Improvements or additions to documentation core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) catalog Related to the catalog crate ffi Changes to the ffi crate labels Nov 21, 2026
SchemaProvider::table_names and SchemaProvider::table_exist now return
Result. Catalogs backed by remote services can report lookup failures
(timeouts, auth errors) instead of returning an empty list, None or
false, which callers could not tell apart from nothing existing. Closes apache#21778.
@gratus00
gratus00 force-pushed the catalog-schema-result-v2 branch from 9d1d2ad to 8fe4c4d Compare November 21, 2026 19:36
@gratus00

Copy link
Copy Markdown
Author

Hi @kosiew could you approve the ci workflows or do I need to do that? Thank you in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate core Core DataFusion crate documentation Improvements or additions to documentation ffi Changes to the ffi crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Result return types to CatalogProvider / SchemaProvider fallible methods

1 participant