Repository navigation
Find partition source columns the same way as the writer - #4457
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
_determine_partitions read the partition source from the input by its full name in the table schema, split on dots, while the writer finds each column by following the table's struct path with the input's own names. A dotted struct name raised KeyError, and an input with field IDs from before a rename got its partition values from another column, so filtered scans missed rows. Follow the writer's path and names instead. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Bug:
_determine_partitionsreads the partition source from the input by its full name in the current table schema, split on dots, while the writer follows the table's struct path with the input's own names (task_schema, from the Arrow field IDs or the name mapping). That gives two bugs:a.band an identity partition on its childc,appendraisesKeyError: 'Field "a" does not exist in schema', because the split (from Fix support for writing to nested field partition #2204) looks for a columna.Fix: find the source the same way as the writer, following its struct path in the table schema with each name taken from the input schema, and pass the names to Arrow literally. How the writer matches columns is unchanged. An input without the partition source's field ID now raises a
ValueError, instead of aKeyErroror a wrong partition value.Java:
SparkWriteBuilderaligns the input schema's IDs with the table, andSparkWritereuses aPartitionKeywhoseStructTransformreads each row throughschema.accessorForField(sourceId)of that input schema.Not covered here: dynamic partition overwrite still has #4196 (open fix #4204), and the row filter converter (
_ConvertToArrowExpression._get_field_name) also splits names, so delete rewrites and filtered scans ofadd_filesfiles with dotted struct names can still raiseArrowInvalid.Are these changes tested?
Yes,
test_append_to_identity_partitioned_table_with_dotted_names,test_append_with_field_ids_after_swapping_column_namesandtest_partition_values_follow_writer_struct_pathintests/catalog/test_catalog_behaviors.py, andtest_partition_for_nested_field_with_dotted_namesintests/io/test_pyarrow.py. On main the dotted-name tests fail with theKeyErrorabove and the swap test with partitionyinstead ofx; all pass with this change.Are there any user-facing changes?
Yes, writes to tables partitioned on a nested field with a dot in a name on its path work, and partition values match the written data when the input has field IDs from before a rename.
Please add the
changeloglabel.This pull request and its description were written by Isaac.