Repository navigation
GH-3877: Add FINITE_VECTOR logical type support - #3876
divjotarora wants to merge 20 commits into
Conversation
Fokko
left a comment
There was a problem hiding this comment.
The biggest issue is around the validation. It feels to me that a lot of visitor that are meant to do other things, are now also validating if it is a valid FINITE_VECTOR. I left some other comments as well. Let me know what you think
There was a problem hiding this comment.
Eventually, these should go in https://github.com/apache/parquet-testing/
There was a problem hiding this comment.
Yes, that's definitely the goal here
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Fokko
left a comment
There was a problem hiding this comment.
Thanks @divjotarora for iterating on this, I think we're getting close 👍 Now, my main concern is the performance penalty that we're having because of the validation.
|
Another thing, I think we don't need to collect that much statistics for the vector:
We can also do this in a separate PR |
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Fokko
left a comment
There was a problem hiding this comment.
Thanks for iterating on this! I ran a benchmark comparing FINITE_VECTOR(n) against a plain LIST holding the same data (4M floats, uncompressed, no dictionary, float[] in/out through a minimal WriteSupport/ReadSupport so the object model doesn't dominate). readPages reads row groups and pages without record assembly, which is where the count checks run:
| ms/op | dim | V1 vector / list | V2 vector / list |
|---|---|---|---|
| write | 8 | 94.9 / 104.0 | 90.5 / 98.6 |
| write | 128 | 75.4 / 76.4 | 78.2 / 82.4 |
| write | 1024 | 70.5 / 71.2 | 71.9 / 82.0 |
| readRecords | 8 | 74.9 / 71.8 | 50.7 / 60.4 |
| readRecords | 128 | 66.3 / 67.7 | 49.3 / 49.3 |
| readRecords | 1024 | 43.6 / 44.5 | 37.8 / 37.1 |
| readPages | 8 | 3.55 / 3.42 | 3.39 / 3.38 |
| readPages | 128 | 3.37 / 3.33 | 3.29 / 3.51 |
| readPages | 1024 | 3.44 / 3.30 | 3.43 / 3.23 |
There's no measurable difference in either direction (single fork, so the swings both ways are run-to-run noise), so removing the per-value validation took care of my performance concern 👍 I can open the benchmark as a separate PR.
My main remaining point is about the reader-side count checks (see inline): now that we delegate validation to the engine, they only catch some invalid files. The rest are small things.
| Type element = list3Levels.getElement(); | ||
| TypeMapping child = fromParquet(element, null, element.getRepetition()); | ||
| ArrowType.FixedSizeList fixedSizeList = | ||
| new ArrowType.FixedSizeList(vectorLogicalType.getNumElements()); |
There was a problem hiding this comment.
Nice, this maps to FixedSizeList now 👍 The other direction still loses the length: fromArrow maps ArrowType.FixedSizeList to a plain LIST (createListTypeMapping()), so FINITE_VECTOR → Arrow → Parquet gives a LIST. Should we map it back to FINITE_VECTOR(listSize) when the child is a non-nullable supported primitive, or do we keep that for a follow-up?
There was a problem hiding this comment.
I think the main issue here is we wouldn't guarantee the values in the Arrow FixedSizeList are all finite, so we could end up with an invalid FINITE_VECTOR on the Parquet side. I chose to go with LIST to stay conservative and provide something that's guaranteed to be correct, but I agree the round-trip inconsistency is confusing. Let me know what you'd prefer here, also happy to take this as a follow-up.
Co-authored-by: Isaac <no-reply@databricks.com>
|
I reran the benchmark from my previous review on 6493ae2, using the same setup: 4M floats, uncompressed, no dictionary, single fork, 3 warmup and 5 measurement iterations (M4 Max, JDK 17).
With this commit the length check runs during record assembly rather than in
That gap was a single-fork outlier. The largest difference left is about 4% at V1 dim=128, which doesn't show up for V2. So validating the lengths in the record reader doesn't add any meaningful overhead 👍 |
Co-authored-by: Isaac <no-reply@databricks.com>
Rationale for this change
The FINITE_VECTOR proposal defines fixed-size vectors of non-null, finite elements on canonical LIST storage, so applications can express vector semantics while retaining existing Parquet storage and interoperability. This change brings parquet-java support in line with the proposal.
What changes are included in this PR?
Adds FINITE_VECTOR schema annotations and mandatory write validation, validates stored levels and elements during reads, and preserves LIST interoperability across existing model readers.
Are these changes tested?
The affected Maven suites for FINITE_VECTOR schemas, writing, reading, adapters, and filtered reads passed, and Spotless passed for all touched modules.
Closes #3877