fix: schema check of iceberg logical types

[raphaelauv]

close: #855

[raphaelauv]

this is a trivial implementation , probably not enough. thanks all

[raphaelauv]

@kevinjqliu wdyt ?

[kevinjqliu]

I'm +1 on this change in theory. I feel like _check_schema_compatible should be as non-blocking as possible, i.e. if pyarrow can write the dataset, _check_schema_compatible should allow it.

I wonder if there's a more generalized solution for this instead of hardcoding UUID to FixedType(16) conversion.
@syun64 @HonahX @Fokko wdyt

[Fokko]

Thanks @raphaelauv for jumping on this right away.

Could we make this part of the pyarrow_to_schema method? I think we could lookup fields there to double-check if we need to apply any logical types.

This way we can also fix issues like #830.

[raphaelauv]

thanks for the review @Fokko, to make it part of pyarrow_to_schema we must change a lot of things to propagate the table_schema ( that is an iceberg schema ) , that's what I tried first and then I reverted and made a separate function _apply_logical_conversion

[Fokko]

Hey @raphaelauv this looks like a great start! 🙌

Could you add two tests as well? One with a UUID as a top level field, and one where it is nested inside of a struct? Thanks!

[Fokko]

We can defer this to a later PR, but just want to put it out here.

In PyIceberg we have a little bit an obsession with the visitor pattern. Using the SchemaWithPartnerVisitor you can traverse two schema's at once. An example is the ArrowProjectionVisitor and can be found in pyarrow.py. The ArrowAccessor does the lookups by ID, but I think here name makes more sense since we don't have the IDs (yet).

[raphaelauv]

fix by #921