Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Infer Iceberg schema from the Parquet file #6997

Merged
merged 47 commits into from
May 2, 2023
Merged

Python: Infer Iceberg schema from the Parquet file #6997

merged 47 commits into from
May 2, 2023

Commits on Mar 23, 2023

  1. parquet schema visitor init draft

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    566ab9b View commit details
    Browse the repository at this point in the history

  2. add unit test for visitor

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    0b4aec0 View commit details
    Browse the repository at this point in the history

  3. use singledispatch to refactor the visitor

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    f542007 View commit details
    Browse the repository at this point in the history

  4. refactor null-check

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    8bf01a8 View commit details
    Browse the repository at this point in the history

  5. add null check for get field id

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    59be7e6 View commit details
    Browse the repository at this point in the history

  6. fix format

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    2505752 View commit details
    Browse the repository at this point in the history

  7. fix schema to pyarrow conversion and add round test

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    7c04fad View commit details
    Browse the repository at this point in the history

  8. fix pyarrow test due to schema to pyarrow change

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    af42d2b View commit details
    Browse the repository at this point in the history

  9. nit fix

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    b5417a8 View commit details
    Browse the repository at this point in the history

  10. nit and typing fix

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    0a355e0 View commit details
    Browse the repository at this point in the history

  11. switch from range to enumerate

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    b742b63 View commit details
    Browse the repository at this point in the history

  12. separate pyarrow test into different files

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    d1c1e37 View commit details
    Browse the repository at this point in the history

  13. add unit tests for new visitor

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    991150a View commit details
    Browse the repository at this point in the history

  14. make conversions consistent and add tests

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    e53f1c9 View commit details
    Browse the repository at this point in the history

  15. refactor the control flow

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    9f1e5ff View commit details
    Browse the repository at this point in the history

  16. add round-trip tests for types

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    d78b1f0 View commit details
    Browse the repository at this point in the history

  17. limit the scope of metadata fetching

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    35e076d View commit details
    Browse the repository at this point in the history

  18. resolve conflict

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    396990e View commit details
    Browse the repository at this point in the history

  19. remove unused fixture

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    fe9e3a7 View commit details
    Browse the repository at this point in the history

  20. extract str to constants

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    a222a57 View commit details
    Browse the repository at this point in the history

  21. substitute string with constants

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    13b1361 View commit details
    Browse the repository at this point in the history

  22. accommodate the fact that pa.struct is iterable

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    b18ddd3 View commit details
    Browse the repository at this point in the history

  23. attach PYTHON before the name

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    93956e9 View commit details
    Browse the repository at this point in the history

  24. fix doc typo

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    f07259c View commit details
    Browse the repository at this point in the history

  25. Revert "attach PYTHON before the name" 

    This reverts commit a72110a.

revert the change made for metadata name
    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    3162919 View commit details
    Browse the repository at this point in the history

  26. rollback to be more flexible about the key

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    40d39a6 View commit details
    Browse the repository at this point in the history

  27. rollback to use id as the end key name

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    736332c View commit details
    Browse the repository at this point in the history

  28. enumerate schema directly

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    8dbb5ad View commit details
    Browse the repository at this point in the history

  29. remove redundant field initialization

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    be83eb0 View commit details
    Browse the repository at this point in the history

  30. fix nit

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    e7871e6 View commit details
    Browse the repository at this point in the history

  31. change to field_id

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    3b51c7f View commit details
    Browse the repository at this point in the history

  32. restrict the flexibility

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    6da6bae View commit details
    Browse the repository at this point in the history

  33. rebase

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    35b04d2 View commit details
    Browse the repository at this point in the history

  34. add integration test for schema conversion from pyarrow to iceberg

    @JonasJ-ap
    JonasJ-ap committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    d3ae307 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. Merge branch 'master' into python_infer_schema_from_parquet

    @JonasJ-ap
    JonasJ-ap committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    1304481 View commit details
    Browse the repository at this point in the history

  2. fix merge issue

    @JonasJ-ap
    JonasJ-ap committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    b561273 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2023

  1. Merge branch 'master' into python_infer_schema_from_parquet

    @JonasJ-ap
    JonasJ-ap committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    52138cc View commit details
    Browse the repository at this point in the history

  2. rollback refactor to pyarrow test

    @JonasJ-ap
    JonasJ-ap committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    9c26acc View commit details
    Browse the repository at this point in the history

  3. clean test

    @JonasJ-ap
    JonasJ-ap committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    6c28d51 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2023

  1. revise nit, fix visitor signature, update visit function

    @JonasJ-ap
    JonasJ-ap committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    5fffa63 View commit details
    Browse the repository at this point in the history

  2. split get_field_id_and_doc to two methods

    @JonasJ-ap
    JonasJ-ap committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    615b700 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. simplify field callbacks

    @JonasJ-ap
    JonasJ-ap committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    ccd4606 View commit details
    Browse the repository at this point in the history

  2. let timezone accept 00:00 offset too

    @JonasJ-ap
    JonasJ-ap committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    4c1c3cd View commit details
    Browse the repository at this point in the history

  3. add test for 0 offset

    @JonasJ-ap
    JonasJ-ap committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    0f17563 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2023

  1. Merge branch 'master' into python_infer_schema_from_parquet

    @JonasJ-ap
    JonasJ-ap committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    878054d View commit details
    Browse the repository at this point in the history

  2. incorporate PR comment

    @JonasJ-ap
    JonasJ-ap committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    ed45a98 View commit details
    Browse the repository at this point in the history

  3. simplify primitive conversion

    @JonasJ-ap
    JonasJ-ap committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    cd6eeb5 View commit details
    Browse the repository at this point in the history