GH-39672: [Go] Time to Date32/Date64 conversion issues for non-UTC timezones #39674
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
|
@raulcd this should get cherry-picked into a new RC to fix the failing test |
raulcd
approved these changes
Jan 18, 2024
github-actions
bot
added
awaiting merge
lidavidm
approved these changes
Jan 18, 2024
go/arrow/datatype_fixedwidth.go
Outdated
| // the actual value by adjusting the time and converting to UTC | ||
| t = t.Add(time.Duration(offset) * time.Second).UTC() | ||
| } | ||
| t = t.In(time.UTC) |
There was a problem hiding this comment.
Time.Unix() claims it is independent of the location, is this strictly needed?
There was a problem hiding this comment.
Since in theory if this change were needed, the adjustment above and the adjustment here would be the same?
There was a problem hiding this comment.
Good point, you're absolutely right. I'll update and remove those lines
Comment on lines
+132
to
+135
| if _, offset := tm.Zone(); offset != 0 { | ||
| // normalize the tm | ||
| tm = tm.Add(time.Duration(offset) * time.Second).UTC() | ||
| } |
There was a problem hiding this comment.
~this is effectively taking a timestamp with timezone and turning it into a naive timestamp showing the date in the former timezone (but marked as UTC)?
There was a problem hiding this comment.
Yes. Because as you pointed out above, when we call .Unix() it returns the result not according to the location but we want it to represent the date in the timezone's location, without having a timezone mark associated. This matches the C++ implementation of the casting in acero (which is where I got the idea)
There was a problem hiding this comment.
Ok, cool. This just confuses me since I'm used to Java etc. where these are separate types ^_^
github-actions
bot
added
awaiting changes
github-actions
bot
added
awaiting change review
lidavidm
approved these changes
Jan 18, 2024
github-actions
bot
added
awaiting merge
|
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 55afcf0. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them. |
clayburn
pushed a commit
to clayburn/arrow
that referenced
this pull request
Jan 23, 2024
…UTC timezones (apache#39674) A failing unit test in release verification led to discovering an issue with timestamp to date conversions for non-utc timezones. Upon some investigation I was able to determine that it was the conflation of casting conversion behavior (normalize to cast a Timestamp to a Date) vs flat conversion. I've fixed this conflation of concerns and the version of the methods which are exported properly converts non-UTC timezones to dates without affecting Casting behavior. ### Are these changes tested? yes ### Are there any user-facing changes? The methods `Date32FromTime` and `Date64FromTime` will properly handle timezones now. * Closes: apache#39672 Authored-by: Matt Topol <[email protected]> Signed-off-by: Matt Topol <[email protected]>
dgreiss
pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
…UTC timezones (apache#39674) A failing unit test in release verification led to discovering an issue with timestamp to date conversions for non-utc timezones. Upon some investigation I was able to determine that it was the conflation of casting conversion behavior (normalize to cast a Timestamp to a Date) vs flat conversion. I've fixed this conflation of concerns and the version of the methods which are exported properly converts non-UTC timezones to dates without affecting Casting behavior. ### Are these changes tested? yes ### Are there any user-facing changes? The methods `Date32FromTime` and `Date64FromTime` will properly handle timezones now. * Closes: apache#39672 Authored-by: Matt Topol <[email protected]> Signed-off-by: Matt Topol <[email protected]>
raulcd
pushed a commit
that referenced
this pull request
Feb 20, 2024
…mezones (#39674) A failing unit test in release verification led to discovering an issue with timestamp to date conversions for non-utc timezones. Upon some investigation I was able to determine that it was the conflation of casting conversion behavior (normalize to cast a Timestamp to a Date) vs flat conversion. I've fixed this conflation of concerns and the version of the methods which are exported properly converts non-UTC timezones to dates without affecting Casting behavior. ### Are these changes tested? yes ### Are there any user-facing changes? The methods `Date32FromTime` and `Date64FromTime` will properly handle timezones now. * Closes: #39672 Authored-by: Matt Topol <[email protected]> Signed-off-by: Matt Topol <[email protected]>
zanmato1984
pushed a commit
to zanmato1984/arrow
that referenced
this pull request
Feb 28, 2024
…UTC timezones (apache#39674) A failing unit test in release verification led to discovering an issue with timestamp to date conversions for non-utc timezones. Upon some investigation I was able to determine that it was the conflation of casting conversion behavior (normalize to cast a Timestamp to a Date) vs flat conversion. I've fixed this conflation of concerns and the version of the methods which are exported properly converts non-UTC timezones to dates without affecting Casting behavior. ### Are these changes tested? yes ### Are there any user-facing changes? The methods `Date32FromTime` and `Date64FromTime` will properly handle timezones now. * Closes: apache#39672 Authored-by: Matt Topol <[email protected]> Signed-off-by: Matt Topol <[email protected]>
thisisnic
pushed a commit
to thisisnic/arrow
that referenced
this pull request
Mar 8, 2024
…UTC timezones (apache#39674) A failing unit test in release verification led to discovering an issue with timestamp to date conversions for non-utc timezones. Upon some investigation I was able to determine that it was the conflation of casting conversion behavior (normalize to cast a Timestamp to a Date) vs flat conversion. I've fixed this conflation of concerns and the version of the methods which are exported properly converts non-UTC timezones to dates without affecting Casting behavior. ### Are these changes tested? yes ### Are there any user-facing changes? The methods `Date32FromTime` and `Date64FromTime` will properly handle timezones now. * Closes: apache#39672 Authored-by: Matt Topol <[email protected]> Signed-off-by: Matt Topol <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A failing unit test in release verification led to discovering an issue with timestamp to date conversions for non-utc timezones.
Upon some investigation I was able to determine that it was the conflation of casting conversion behavior (normalize to cast a Timestamp to a Date) vs flat conversion. I've fixed this conflation of concerns and the version of the methods which are exported properly converts non-UTC timezones to dates without affecting Casting behavior.
Are these changes tested?
yes
Are there any user-facing changes?
The methods
Date32FromTimeandDate64FromTimewill properly handle timezones now.