Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix handling of Avro timestamps created in Hive 3.1 #5144

Open
aalbu opened this issue Sep 11, 2020 · 4 comments
Open

Fix handling of Avro timestamps created in Hive 3.1 #5144

aalbu opened this issue Sep 11, 2020 · 4 comments
Labels
bug Something isn't working correctness

Comments

@aalbu
Copy link
Member

aalbu commented Sep 11, 2020

Hive 3.1 has changed the semantics of timestamps in backward incompatible way for a number of file formats (see https://issues.apache.org/jira/browse/HIVE-21002). The impact of this change on Presto was addressed for Parquet and RCBinary, but it still manifests itself for Avro.

0: jdbc:hive2://localhost:10000/default> insert into test_avro values (1, '2020-01-01 12:34:56.123');


presto:default> select * from test_avro;
 id |           ts
----+-------------------------
  1 | 2020-01-01 18:19:56.123
(1 rows)
@findepi findepi mentioned this issue Sep 11, 2020
29 tasks
@findepi findepi added the bug Something isn't working label Sep 11, 2020
@findepi
Copy link
Member

findepi commented Sep 11, 2020

I though this gonna require Hive 3.1.2 (see https://issues.apache.org/jira/browse/HIVE-21002, https://issues.apache.org/jira/browse/HIVE-21291)
But maybe we just do the same as @dain did for Parquet, ORC and RCFile?
OTOH, having 4 toggles (3 existing + 1 new) seems like may be hard for users to set correctly.

@findepi
Copy link
Member

findepi commented Sep 11, 2020

@dain have a look please.

@dain
Copy link
Member

dain commented Oct 13, 2020

@aalbu or @electrum is this still an issue

@aalbu
Copy link
Member Author

aalbu commented Oct 13, 2020

Yes, it can be reproduced in the HDP3 environment.

@jirassimok jirassimok self-assigned this Jul 2, 2021
@homar homar self-assigned this Aug 19, 2021
homar added a commit to homar/trino that referenced this issue Sep 1, 2021
trinodb#5144
Previously timestamps from avro were always read with UTC timezone.
Now this timezone is extracted to configuration and can be set per catalog.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working correctness
Development

No branches or pull requests

5 participants