Repository navigation
Conversation
|
|
There was a problem hiding this comment.
🟡 Changes recommended
Required vectors fail to read under null parent structs, while insufficient element-type filtering causes schema-conversion failures, exponential conversion work, and unvalidated NaN/Inf values in VECTOR files.
2 open issues
What changed in this PR
Adds experimental support for the proposed Parquet VECTOR logical type, mapping it to Arrow FixedSizeList in C++ and Python while keeping VECTOR writing disabled by default.
Changes:
- Adds VECTOR serialization, logical-type APIs, and canonical schema validation.
- Implements FixedSizeList mapping and opt-in writing with element validation.
- Exposes Python writer and dataset options, with documentation and round-trip tests.
| File | Description |
|---|---|
python/pyarrow/tests/parquet/test_data_types.py |
Tests VECTOR round-trips, fallbacks, validation, and dataset writing. |
python/pyarrow/parquet/core.py |
Exposes and documents the Python writer option. |
python/pyarrow/includes/libparquet.pxd |
Declares VECTOR and C++ writer-property bindings. |
python/pyarrow/_parquet.pyx |
Connects Python options to C++ and names VECTOR metadata. |
python/pyarrow/_parquet.pxd |
Extends the writer-properties helper declaration. |
python/pyarrow/_dataset_parquet.pyx |
Adds VECTOR to dataset write options. |
docs/source/cpp/parquet.rst |
Documents VECTOR mapping and experimental writing. |
cpp/src/parquet/types.h |
Declares the VECTOR logical-type API. |
cpp/src/parquet/types.cc |
Implements VECTOR properties and serialization. |
cpp/src/parquet/schema.h |
Exposes VECTOR element eligibility validation. |
cpp/src/parquet/schema.cc |
Validates canonical VECTOR group structure. |
cpp/src/parquet/schema_test.cc |
Tests logical-type properties, structure, and serialization. |
cpp/src/parquet/properties.h |
Adds the opt-in C++ writer setting. |
cpp/src/parquet/parquet.thrift |
Defines VECTOR in the format schema. |
cpp/src/parquet/arrow/writer.cc |
Aligns element nullability and validates written values. |
cpp/src/parquet/arrow/schema.cc |
Maps VECTOR and FixedSizeList schemas. |
cpp/src/parquet/arrow/arrow_schema_test.cc |
Tests schema mapping, defaults, and fallbacks. |
cpp/src/parquet/arrow/arrow_reader_writer_test.cc |
Tests nested round-trips and invalid values or lengths. |
cpp/src/generated/parquet_types.tcc |
Adds generated VECTOR protocol serialization. |
cpp/src/generated/parquet_types.h |
Adds generated VECTOR declarations. |
cpp/src/generated/parquet_types.cpp |
Adds generated VECTOR implementations. |
Review effort: Balanced
714d287 to
8acc8ae
Compare
d888630 to
31e7a59
Compare
MashrafeeAryan
left a comment
There was a problem hiding this comment.
I just had few comments on it Rok. Overall, it looks good to me.
| if (root_field != nullptr) { | ||
| ARROW_ASSIGN_OR_RAISE( | ||
| std::shared_ptr<ArrayData> prepared, | ||
| MakeFiniteVectorElementsRequired(array_to_write->data(), *root_field)); |
There was a problem hiding this comment.
If we don’t have any FINITE_VECTOR fields, can we skip this helper? We could check the schema once before the chunk loop. If there are no finite vectors, we can skip the helper instead of having it walk the nested arrays in every chunk.
I am sure how much time this would save. Could we benchmark it?
There was a problem hiding this comment.
Done: each column now checks once, before the chunk loop, whether its schema subtree has a FINITE_VECTOR, and skips the helper otherwise. I didn't benchmark it. The helper only walks the type tree (not the values) and returns the input unchanged when there's nothing to do, so I'd expect the difference to be in the noise.
| field_name_to_idx_.clear(); | ||
| auto field_idx = 0; | ||
| for (NodePtr& field : fields_) { | ||
| field->SetParent(this); |
There was a problem hiding this comment.
We are setting the new group as the child’s parent with field->SetParent(this) before checking if the group is valid. If CheckFiniteVectorStructure throws an error, the parent group goes away. Then the child now points to a parent that no longer exists. I believe if we move the check before setting the parent, it would be great
There was a problem hiding this comment.
Good catch. I moved the check ahead of the SetParent loop, so a rejected group is never left as the parent of its fields. Also added a test that the child's parent stays null after the throw.
…cal type Add the FINITE_VECTOR logical type from parquet-format (apache/parquet-format#624): a canonical 3-level LIST whose non-null values contain exactly num_elements required, finite primitive elements. - Add FiniteVectorType to parquet.thrift and regenerate with thrift 0.23.0. - Add LogicalType::FiniteVector / FiniteVectorLogicalType, mapped to ConvertedType LIST, and validate the canonical structure and allowed element types when a FINITE_VECTOR-annotated GroupNode is created. - Read FINITE_VECTOR groups as fixed_size_list<element: T not null>[n], restoring a nullable element field from the stored Arrow schema. - Fix FixedSizeListReader so that a non-nullable fixed-size list under a null ancestor (e.g. a required FINITE_VECTOR inside a nullable struct) is padded instead of rejected as a wrong-sized list. - Add the experimental ArrowWriterProperties option enable_finite_vector_logical_type() (default off) to write eligible FixedSizeList columns as FINITE_VECTOR. Eligibility is decided from the Arrow value type (extension storage unwrapped, dictionaries excluded); ineligible ones are written as LIST. Writing fails if a non-null vector has a null or non-finite element. - Expose use_finite_vector_logical_type in pyarrow's write_table, ParquetWriter and dataset write options.
- Check the FINITE_VECTOR structure before the fields are given the new group as their parent, so a rejected group is never left as a parent. - Decide once per column whether it contains a FINITE_VECTOR and skip MakeFiniteVectorElementsRequired otherwise. - FINITE_VECTOR is now part of parquet-format (not yet released): update the pyarrow docstring and point the FixedSizeList docs footnote to it.
31e7a59 to
67f89f6
Compare

Rationale for this change
apache/parquet-format#624 added the
FINITE_VECTOR(num_elements)logical type (merged, not yet in a format release): a canonical 3-level LIST whose non-null values each hold exactlynum_elementsrequired, finite primitive elements. Arrow'sFixedSizeListmaps onto it directly. parquet-java counterpart: apache/parquet-java#3876.What changes are included in this PR?
FiniteVectorTypetoparquet.thriftand regenerate with thrift 0.23.0 (same version as before). The generated diff is almost entirely renumbered temporaries and can be skimmed.LogicalType::FiniteVector(n)(n > 0, converted type LIST). Constructing an annotatedGroupNodevalidates the canonical structure and the element type (BOOLEAN, INT32/INT64, FLOAT, DOUBLE, FLOAT16 or DECIMAL), both when building schemas and when reading files.FINITE_VECTOR(n)is read asfixed_size_list<element: T not null>[n]. A nullable element field is restored from the stored Arrow schema.ArrowWriterProperties::Builder::enable_finite_vector_logical_type(), off by default. EligibleFixedSizeListcolumns are written as FINITE_VECTOR, all others as LIST as before. Writing fails withInvalidif a non-null vector contains a null, NaN or ±Inf element.use_finite_vector_logical_typeonwrite_table,ParquetWriterand the dataset write options.Are these changes tested?
Yes, in
schema_test.cc,arrow_schema_test.cc,arrow_reader_writer_test.ccandtest_data_types.py: logical type properties and thrift round-trip, structure validation, schema mapping in both directions, data round-trips (nulls, nesting, extension types), fallback to LIST, write errors, and reading a file with a wrong-sized vector.Are there any user-facing changes?
FixedSizeList.Released Arrow versions cannot open these files ("Logical type Undefined cannot be applied to group node") instead of falling back to
ConvertedType.LIST. That is a separate forward-compatibility problem with unknown group logical types and is not addressed here.Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: