Repository navigation
Conversation
SchemaProvider::table_names and SchemaProvider::table_exist now return Result. Catalogs backed by remote services can report lookup failures (timeouts, auth errors) instead of returning an empty list, None or false, which callers could not tell apart from nothing existing. Closes apache#21778.
gratus00
force-pushed
the
catalog-schema-result-v2
branch
from
November 21, 2026 19:36
9d1d2ad to
8fe4c4d
Compare
Author
|
Hi @kosiew could you approve the ci workflows or do I need to do that? Thank you in advance! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SchemaProvider::table_names and SchemaProvider::table_exist now return Result. Catalogs backed by remote services can report lookup failures (timeouts, auth errors) instead of returning an empty list, None or false, which callers could not tell apart from nothing existing. Closes #21778.
Important
Fork CI setup: For pull requests from a fork to
main, enable workflows in your fork's Actions tab before you push.See the fork CI setup guide.
Which issue does this PR close?
This replaces refactor: return Result from fallible catalog and schema methods #21944, which was closed after going stale. It is the same change, rebased onto current
mainand updated for the move ofCatalogProvider/SchemaProviderintodatafusion-session.Rationale for this change
Catalogs backed by a remote service (for example paimon-rust or delta-rs Unity Catalog) have to make network calls when DataFusion asks which schemas or tables exist. When such a call fails (403, timeout, connection refused), the current trait signatures leave the implementation no way to report it: it has to return an empty list,
None, orfalse. To the user, a failed request then looks the same as an empty catalog or a missing table, which makes these failures very hard to diagnose.What changes are included in this PR?
These trait methods now return
Result:CatalogProviderschema_names()Vec<String>Result<Vec<String>>CatalogProviderschema()Option<Arc<dyn SchemaProvider>>Result<Option<Arc<dyn SchemaProvider>>>SchemaProvidertable_names()Vec<String>Result<Vec<String>>SchemaProvidertable_exist()boolResult<bool>The change is carried through:
datafusion-sessionSessionState/SessionContextcatalog resolution, where errors now propagate with?datafusion-cli, examples, and sqllogictest test providersFFI_CatalogProvider/FFI_SchemaProviderfunction pointers now returnFFI_Result<...>, so errors cross the FFI boundary instead of being dropped. This changes the FFI struct layout.catalogs.mduser guide examples, which are compiled as doctestsupgrading/57.0.0.mddescribing the migrationDesign note: errors propagate, including in
information_schemaA missing schema or table is still
Ok(None)/Ok(false)and produces the usual "not found" error. Only a catalog that cannot answer returnsErr, and that error is propagated with?.One trade-off for reviewers: the
information_schematables (andSHOW TABLES) enumerate every registered catalog, so with this PR one unreachable catalog makes the whole listing query fail, where before it would silently appear empty.The alternative is to skip the failing catalog/schema with a
log::warn!and keep listing the others. I chose to propagate because skipping reintroduces the original problem in a narrower form: the user gets an incomplete result with nothing in the result itself saying so. The skip-and-warn variant would only touch the fouradd_*helpers ininformation_schema.rs, so it is an easy follow-up if reviewers prefer it.What is the testing strategy for this PR?
create_droptests are updated for the new signatures.datafusion/ffi/src/catalog_provider.rs:test_ffi_catalog_provider_propagates_failureschecks that an error raised by the producer reaches the consumer through the FFI wrapper.datafusion/ffi/integration-tests/tests/cases/catalog.rs: newtest_catalog_schema_lookupscalls all four methods on a catalog loaded from the separately compiled cdylib, covering both found and not-found cases.catalogs.mddoctests pass with the new signatures.Run locally:
cargo fmt --all,cargo clippy --all-targets --all-features -- -D warnings, the FFI unit and integration tests, thecatalogs.mddoctests,./ci/scripts/doc_prettier_check.sh, and the extended workspace test suite.Are there any user-facing changes?
No change to SQL or CLI behavior when catalogs are healthy. When a catalog's lookup fails, the query now returns that error instead of treating the catalog as empty or the table as missing.
This is a breaking public API change for implementers and callers of
CatalogProvider/SchemaProvider, and it changes thedatafusion-ffiABI, so it needs theapi changelabel. Migration notes are indocs/source/library-user-guide/upgrading/57.0.0.md.