Skip to content

Reject columns whose field IDs are not in the table schema on write - #4455

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/append-reject-unknown-field-ids
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/append-reject-unknown-field-ids

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Bug: when every field of an Arrow table carries a field ID, append drops the columns whose field ID is not in the table schema, without an error:

schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "x", LongType()))
tbl = catalog.create_table("db.t", schema=schema)
with_z = Schema(*schema.fields, NestedField(99, "z", LongType()))
tbl.append(pa.Table.from_pylist([{"id": 1, "x": 2, "z": 3}], schema=with_z.as_arrow()))
tbl.scan().to_arrow().to_pylist()
# [{'id': 1, 'x': 2}]

The same happens to a column that was dropped and added back: an Arrow schema taken before delete_column("x") and add_column("x", LongType()) has x with field ID 2, the new x has field ID 3, and the appended values are stored as null. overwrite and dynamic_partition_overwrite lose these columns too, and upsert can lose them on inserts or when updating with a stale ID. Without field IDs, the extra z is already rejected (PyArrow table contains more columns: z). Fokko raised the same concern on #4419 (review): silently dropping a column on append loses data without any signal.

Fix: the four write paths now reject input field IDs that are not in the table schema, before anything is written:

ValueError: PyArrow table contains columns whose field IDs are not in the table schema: z (field ID 99).

add_files is unchanged (#4359). Note that #4444 currently tests that upsert accepts such a stale ID; with this change it raises.

Java: Spark's write builder gives the input the table's field IDs by name (SparkSchemaUtil.convert -> TypeUtil.reassignIds) and fails on unknown columns (Field z not found in source schema) unless merge-schema adds them first. PyIceberg has no merge-schema option on write and keeps the input's own IDs when every field has one, so it rejects the stale x instead of writing it to the new x.

Are these changes tested?

Yes, test_write_rejects_field_ids_not_in_table_schema in tests/catalog/test_catalog_behaviors.py runs both cases through the four write APIs on an evolved table, checks that nothing is committed, and that appends with a renamed column's old name, reordered, promoted, missing optional and ID-less columns still work. It fails on main with DID NOT RAISE ValueError and passes with this change. test_schema_mismatch_unknown_field_ids in tests/io/test_pyarrow.py covers struct fields, list elements and map values.

Are there any user-facing changes?

Yes, writes now reject unknown field IDs, including dropped columns kept in Arrow data read from older Parquet files.

Please add the changelog label.

This pull request and its description were written by Isaac.

Appending, overwriting or upserting an Arrow table whose fields carry field IDs dropped the columns whose ID is not
in the table schema, such as an extra column or one that was dropped and added back, while the same input without
field IDs is rejected. The write paths now pass reject_unknown_field_ids to _check_pyarrow_schema_compatible, which
raises a ValueError that names each such column and its field ID. add_files is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@kevinjqliu, you reviewed #2662 (field IDs in add_files) and @jeroko wrote it. Could you take a look when you have a chance? Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant