[#2587] feat(spark-connector): Support iceberg metadataColumns #2717
Conversation
…ations to Iceberg catalog
…r spark-connector
…r spark-connector
…r spark-connector
…o seperate-spark-it
…r spark-connector
…r spark-connector
…r spark-connector
…to iceberg-read-write
|
this PR is ready for review after iceberg partition PR is merged. cc @FANNG1 |
...test/java/com/datastrato/gravitino/integration/test/spark/iceberg/SparkIcebergCatalogIT.java
Outdated
Show resolved
Hide resolved
...test/java/com/datastrato/gravitino/integration/test/spark/iceberg/SparkIcebergCatalogIT.java
Outdated
Show resolved
Hide resolved
|
Hi @FANNG1 , may i ask do you have time to help review this PR today? Thanks |
I'm afraid I didn't have enough time to review this PR for preparing 0.5 releases. |
ok. maybe we can review it after 0.5.0 released, Thank you. |
...nector/src/main/java/com/datastrato/gravitino/spark/connector/iceberg/SparkIcebergTable.java
Show resolved
Hide resolved
...rc/test/java/com/datastrato/gravitino/integration/test/util/spark/SparkTableInfoChecker.java
Outdated
Show resolved
Hide resolved
.../src/test/java/com/datastrato/gravitino/integration/test/util/spark/SparkMetadataColumn.java
Outdated
Show resolved
Hide resolved
...test/java/com/datastrato/gravitino/integration/test/spark/iceberg/SparkIcebergCatalogIT.java
Show resolved
Hide resolved
…ino into iceberg-metadata-columns
|
all comments have been addressed. cc @FANNG1 |
...test/java/com/datastrato/gravitino/integration/test/spark/iceberg/SparkIcebergCatalogIT.java
Outdated
Show resolved
Hide resolved
|
LGTM |
|
@caican00 merged to main, thanks for your work |
|
@caican00 would you please cherry-pick this commit to branch-0.5, and create a similar PR against branch-0.5. I'm planning to add a complete iceberg support in 0.5.1. |
…pache#2717) Support retrieve iceberg metadataColumns, such as `_spec_id`, `_partition`, `_file`, `_pos`, `_deleted`. Support retrieve iceberg metadataColumns, row-level operations depend on this. Fix: apache#2587 No. New integration test.
…pache#2717) ### What changes were proposed in this pull request? Support retrieve iceberg metadataColumns, such as `_spec_id`, `_partition`, `_file`, `_pos`, `_deleted`. ### Why are the changes needed? Support retrieve iceberg metadataColumns, row-level operations depend on this. Fix: apache#2587 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New integration test.
got it. done |
…pache#2717) ### What changes were proposed in this pull request? Support retrieve iceberg metadataColumns, such as `_spec_id`, `_partition`, `_file`, `_pos`, `_deleted`. ### Why are the changes needed? Support retrieve iceberg metadataColumns, row-level operations depend on this. Fix: apache#2587 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New integration test.
What changes were proposed in this pull request?
Support retrieve iceberg metadataColumns, such as
_spec_id,_partition,_file,_pos,_deleted.Why are the changes needed?
Support retrieve iceberg metadataColumns, row-level operations depend on this.
Fix: #2587
Does this PR introduce any user-facing change?
No.
How was this patch tested?
New integration test.