Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions core/src/main/java/org/apache/datafusion/ExecutionPlan.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.datafusion;

import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;

/**
* A bounded Java-backed leaf plan returned by {@link TableProvider#scanWithArgs(ScanArgs)}.
*
* <p>Constructing the scan must not read source rows. Each execution opens a fresh independent
* reader whose schema is either the registered full schema or its requested projection. Reader
* buffers must use the supplied allocator or a child of it. DataFusion owns and closes the reader.
*
* <p>This is a Java-backed scan, not an API for arbitrary child-consuming physical operators.
*/
@FunctionalInterface
public interface ExecutionPlan {
/**
* Open an independent stream for a partition during physical execution.
*
* <p>The current adapter executes only partition {@code 0}; the partition argument is explicit so
* supporting multi-partition plans later does not require changing this method's signature.
*/
ArrowReader execute(int partition, BufferAllocator allocator);
}
48 changes: 48 additions & 0 deletions core/src/main/java/org/apache/datafusion/ScanArgs.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.datafusion;

import java.util.List;
import java.util.Optional;
import java.util.OptionalLong;

import org.apache.datafusion.protobuf.LogicalExprNode;

/**
* A physical scan request using DataFusion's projection, expression and limit semantics.
*
* <p>Projection indices refer to the registered table schema. An absent projection means all
* columns; a present empty list means no columns (for example, {@code COUNT(*)}), not no rows.
* Filters use the bundled DataFusion version's expression protobufs, not SQL strings. Their
* generated types and expression shapes may change when the binding upgrades DataFusion.
*
* <p>The optional limit is the value supplied by DataFusion's planner. Sources may return more
* rows; they must not return fewer than requested when that many matching rows are available.
* DataFusion retains responsibility for deciding whether a limit can safely cross a residual
* filter. Limits outside the non-negative Java {@code long} range are not forwarded; DataFusion
* still applies the query's limit.
*/
public record ScanArgs(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exposes the generated LogicalExprNode as public API. That type tracks DataFusion's datafusion.proto for the bundled version, and its package is excluded from our Javadoc. Pushdown code that matches on BinaryExpr ops and literal variants, like the test's SelectiveSource, could break when we bump DataFusion. I think passing the proto is a reasonable first step, but should we say in table-provider.md that the expression format follows DataFusion's proto and may change between DataFusion versions? It would be good to agree on that compatibility story before this ships in a release.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my own java bindings I'd original made it a goal for the expression APIs (logical and physical) to use their own POJOs of sealed classes and records where it made sense, with the hope of the API feeling friendlier, and hiding the generated classes unless the caller wanted to pass them. I backed off voicing that goal when I saw the protos start to get exposed in this repo and I couldn't really explain why I thought it was important. You do highlight one possible compatibility issue that swings me back in the other direction.

@pgwhalen pgwhalen Nov 21, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, here is my working draft of the "public API" agent instructions that I mentioned wanting to do many months ago. I have larger ambitions for it but tried to slim it down for what is realistic right now.

If that seems directionally correct and only require small tweaks, I'd be happy to open a proper PR. There are some minor divergences from the rules there which I could open up follow-up PRs for.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The guide and ScanArgs Javadoc explain that LogicalExprNode follows the bundled DataFusion version; providers should decline unfamiliar forms with UNSUPPORTED and retest on upgrade. I’d keep a stable Java expression model as a separate API discussion.

Optional<List<Integer>> projection, List<LogicalExprNode> filters, OptionalLong limit) {
public ScanArgs {
projection = projection.map(List::copyOf);
filters = List.copyOf(filters);
}
}
12 changes: 8 additions & 4 deletions core/src/main/java/org/apache/datafusion/SessionContext.java
Original file line number Diff line number Diff line change
Expand Up @@ -559,10 +559,14 @@ public void registerUdf(ScalarUdf udf) {
* reference {@code name} call back into {@code provider} to fetch batches.
*
* <p>{@link TableProvider#schema()} is called once here, on the calling thread, and cached on the
* native side. {@link TableProvider#scan(org.apache.arrow.memory.BufferAllocator)} is called once
* per query that touches the table, on a Tokio worker thread; it must return a fresh, independent
* {@link org.apache.arrow.vector.ipc.ArrowReader} on every call, with its buffers allocated from
* the {@link org.apache.arrow.memory.BufferAllocator} the framework supplies.
* native side. For base providers, {@link
* TableProvider#scan(org.apache.arrow.memory.BufferAllocator)} is called once per physical scan,
* on a Tokio worker thread; it must return a fresh, independent {@link
* org.apache.arrow.vector.ipc.ArrowReader} on every call, with its buffers allocated from the
* {@link org.apache.arrow.memory.BufferAllocator} the framework supplies.
*
* <p>Override {@link TableProvider#scanWithArgs(ScanArgs)} to plan a lazy scan with projection,
* supported filters and a limit supplied by DataFusion. Planning does not open its reader.
*
* <p>This is the Java counterpart to DataFusion's Rust {@code SessionContext::register_table}.
*
Expand Down
55 changes: 46 additions & 9 deletions core/src/main/java/org/apache/datafusion/TableProvider.java
Original file line number Diff line number Diff line change
Expand Up @@ -19,30 +19,43 @@

package org.apache.datafusion;

import java.util.Collections;
import java.util.List;

import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.pojo.Schema;
import org.apache.datafusion.protobuf.LogicalExprNode;

/**
* A Java-implemented table that can be registered with a {@link SessionContext} via {@link
* SessionContext#registerTable(String, TableProvider)}. Mirrors the role of DataFusion's Rust
* {@code TableProvider} trait, but at present only exposes the methods needed for a full table
* scan; future versions may add filter/projection pushdown and multi-partition support as default
* methods so existing implementations keep working.
* {@code TableProvider} trait. Implement {@link #scan(BufferAllocator)} for full-schema reads.
* Optionally override {@link #scanWithArgs(ScanArgs)} to plan a lazy scan with
* projection/filter/limit pushdown.
*
* <p>{@link SimpleTableProvider} is a ready-made implementation for the common case of "I have a
* schema and a function that returns an {@link ArrowReader}".
*
* <p>Each call to {@link #scan(BufferAllocator)} must return a fresh, independent {@link
* ArrowReader} so that queries which touch the table more than once (self-joins, {@code UNION ALL},
* repeated reads) work correctly. The returned reader is closed by the framework when the stream
* ends.
* <p>Each execution must return a fresh, independent {@link ArrowReader} so that queries which
* touch the table more than once (self-joins, {@code UNION ALL}, repeated reads) work correctly.
* The returned reader is closed by the framework when the stream ends.
*
* <p>The schema returned by {@link #schema()} is captured once at registration time. Every batch
* produced by every {@code ArrowReader} returned from {@link #scan(BufferAllocator)} must conform
* to it; a mismatch fails the query.
* produced by {@link #scan(BufferAllocator)} must conform to it. A plan returned by {@link
* #scanWithArgs(ScanArgs)} may instead produce the requested projected schema.
*/
public interface TableProvider {
/** DataFusion's per-filter pushdown contract. */
enum FilterPushdown {
/** The source does not apply this filter; DataFusion keeps it above the scan. */
UNSUPPORTED,
/** The source prunes candidates; DataFusion also applies the filter to the returned rows. */
INEXACT,
/** The source completely applies the filter with DataFusion-compatible semantics. */
EXACT
}

/** The fixed schema of this table. Called once, at registration time. */
Schema schema();

Expand All @@ -56,4 +69,28 @@ public interface TableProvider {
* one it passes here. The allocator contract mirrors the one on {@link ScalarFunction#evaluate}.
*/
ArrowReader scan(BufferAllocator allocator);

/**
* Plan one bounded scan without opening its reader. The default defers to {@link
* #scan(BufferAllocator)} at execution time, preserving existing providers.
*
* <p>Only supported/inexact filters are included in {@code args}. The plan may produce either the
* registered full schema or exactly the requested projection, including zero-column batches that
* preserve row counts. The native adapter projects full-schema batches when needed.
*
* <p>Planning may be repeated and must not read source rows. Callbacks may run on native worker
* threads; shared provider state and reader factories must be thread-safe.
*/
default ExecutionPlan scanWithArgs(ScanArgs args) {
return (partition, allocator) -> scan(allocator);
}

/**
* Return one capability per input filter, in the same order. The default declines every filter.
* An inexact filter must never discard a matching row. Expressions are generated protobuf
* messages; source-specific access-path decisions remain in the provider.
*/
default List<FilterPushdown> supportsFiltersPushdown(List<LogicalExprNode> filters) {
return Collections.nCopies(filters.size(), FilterPushdown.UNSUPPORTED);
}
}
79 changes: 69 additions & 10 deletions core/src/main/java/org/apache/datafusion/internal/JniBridge.java
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.OptionalLong;

import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowArrayStream;
Expand All @@ -31,16 +33,22 @@
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.datafusion.ColumnarValue;
import org.apache.datafusion.ExecutionPlan;
import org.apache.datafusion.ScalarFunction;
import org.apache.datafusion.ScalarFunctionArgs;
import org.apache.datafusion.ScanArgs;
import org.apache.datafusion.TableProvider;
import org.apache.datafusion.protobuf.LogicalExprNode;
import org.apache.datafusion.protobuf.TableScanOptionsProto;

/** Internal trampoline invoked from native code on every UDF call. Not part of the public API. */
import com.google.protobuf.InvalidProtocolBufferException;

/** Internal trampolines for native UDF and table-provider callbacks. Not part of the public API. */
public final class JniBridge {
/**
* Shared allocator for UDF inputs/outputs. Created once at class-load time; never closed.
* Outstanding allocations are released by the FFI structs' release callbacks when the native side
* drops them after {@code from_ffi}.
* Shared allocator for UDF inputs/outputs and table scans. Created once at class-load time; never
* closed. Outstanding allocations are released by the FFI structs' release callbacks when the
* native side drops them after {@code from_ffi}.
*/
private static final RootAllocator ALLOCATOR = new RootAllocator();

Expand Down Expand Up @@ -143,22 +151,73 @@ public static byte invokeScalarUdf(
}
}

/** Plan one physical scan without opening its reader. Called from native code. */
public static ExecutionPlan scanWithArgs(
TableProvider provider, byte[] optionsBytes, byte[][] filterProtos)
throws InvalidProtocolBufferException {
TableScanOptionsProto options = TableScanOptionsProto.parseFrom(optionsBytes);
Optional<List<Integer>> projection =
options.hasProjection()
? Optional.of(options.getProjection().getIndicesList())
: Optional.empty();
ExecutionPlan plan =
provider.scanWithArgs(
new ScanArgs(
projection,
parseFilters(filterProtos),
options.hasLimit() ? OptionalLong.of(options.getLimit()) : OptionalLong.empty()));
if (plan == null) {
throw new IllegalStateException("TableProvider.scanWithArgs returned null");
}
return plan;
}

/** Report per-expression capabilities using the native adapter's closed numeric contract. */
public static int[] supportsTableFilterPushdown(TableProvider provider, byte[][] filterProtos)
throws InvalidProtocolBufferException {
List<TableProvider.FilterPushdown> capabilities =
provider.supportsFiltersPushdown(parseFilters(filterProtos));
if (capabilities == null || capabilities.size() != filterProtos.length) {
throw new IllegalArgumentException(
"supportsFiltersPushdown must return one capability per filter");
}
int[] result = new int[capabilities.size()];
for (int i = 0; i < result.length; i++) {
result[i] =
switch (capabilities.get(i)) {
case UNSUPPORTED -> 1;
case INEXACT -> 2;
case EXACT -> 3;
};
}
return result;
}

private static List<LogicalExprNode> parseFilters(byte[][] filterProtos)
throws InvalidProtocolBufferException {
List<LogicalExprNode> filters = new ArrayList<>(filterProtos.length);
for (byte[] filter : filterProtos) {
filters.add(LogicalExprNode.parseFrom(filter));
}
return filters;
}

/**
* Open a fresh batch stream from a Java {@link TableProvider} and export it through the supplied
* Arrow C Data Interface address. Called from native code; not for application use.
*
* <p>{@link TableProvider#scan(org.apache.arrow.memory.BufferAllocator)} is called with {@link
* #ALLOCATOR} so that the reader's buffers share the same allocator root required by {@link
* Data#exportArrayStream}.
* <p>{@link ExecutionPlan#execute(int, org.apache.arrow.memory.BufferAllocator)} is called with
* {@link #ALLOCATOR} so that the reader's buffers share the same allocator root required by
* {@link Data#exportArrayStream}.
*
* <p>On success, ownership of the returned reader transfers to the FFI stream's release callback,
* so the native side closing the stream also closes the reader. On any failure during export, the
* reader is closed here before the exception propagates.
*/
public static void invokeTableScan(TableProvider provider, long ffiStreamAddr) {
ArrowReader reader = provider.scan(ALLOCATOR);
public static void executePlan(ExecutionPlan plan, int partition, long ffiStreamAddr) {
ArrowReader reader = plan.execute(partition, ALLOCATOR);
if (reader == null) {
throw new IllegalStateException("TableProvider.scan returned null");
throw new IllegalStateException("ExecutionPlan.execute returned null");
}
ArrowArrayStream stream = ArrowArrayStream.wrap(ffiStreamAddr);
try {
Expand Down
Loading