datetime: support leap seconds #67762
Comments
|
A leap second will be added in June 2015: The datetime module explicitly doesn't support leap seconds: The following bug in oslo.utils was reported because datetime is indirectly used to unserialize a date, but it fails with ValueError("second must be in 0..59") if the second is 60: Would it be possible to silently drop ignore leap seconds in datetime.datetime constructor, as already done in datetime.datetime.fromtimestamp? Attached patch modified datetime constructor to drop leap seconds: replace second=60 with second=59. I also changed the error message for second (valid range is now 0..60). |
|
Leap seconds are ignored, so a difference of <datetime before the leap second> and <datetime with the leap second> is zero: >>> import datetime
>>> t1=datetime.datetime(2012, 6, 30, 23, 59, 59)
>>> t2=datetime.datetime(2012, 6, 30, 23, 59, 59)
>>> t2-t1
datetime.timedelta(0)Supporting leap seconds might be possible, but it requires much more work. |
|
Ignoring leap seconds introduces unexpected result. datetime.timestamp -> datetime.fromtimestamp drops one second: $ ./python
Python 3.5.0a1+ (default:760f222103c7+, Mar 3 2015, 15:36:36)
>>> t=datetime.datetime(2012, 6, 30, 23, 59, 60).timestamp()
>>> datetime.datetime.fromtimestamp(t)
datetime.datetime(2012, 6, 30, 23, 59, 59)time and datetime modules behave differently: $ ./python
Python 3.5.0a1+ (default:760f222103c7+, Mar 3 2015, 15:36:36)
>>> import datetime, time
>>> t1=datetime.datetime(2012, 6, 30, 23, 59, 59).timestamp()
>>> t2=datetime.datetime(2012, 6, 30, 23, 59, 60).timestamp()
>>> t2-t1
0.0
>>> t3=time.mktime((2012, 6, 30, 23, 59, 59, -1, -1, -1))
>>> t4=time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
>>> t4-t3
1.0
>>> t1 == t2 == t3
True
>>> t3, t4
(1341093599.0, 1341093600.0) |
|
support_leap_seconds.patch: different approach, accept second=60. Problem: fromtimestamp() returns the wrong day. haypo@smithers$ ./python
Python 3.5.0a1+ (default:760f222103c7+, Mar 3 2015, 15:36:36)
>>> import datetime
>>> datetime.datetime(2012, 6, 30, 23, 59, 60)
datetime.datetime(2012, 6, 30, 23, 59, 60)
>>> dt1=datetime.datetime(2012, 6, 30, 23, 59, 60)
>>> t1=datetime.datetime(2012, 6, 30, 23, 59, 60).timestamp()
>>> dt2=datetime.datetime.fromtimestamp(t1)
>>> dt2
datetime.datetime(2012, 7, 1, 0, 0)
>>> dt2 == dt1
False
>>> dt1
datetime.datetime(2012, 6, 30, 23, 59, 60)
>>> print(dt1)
2012-06-30 23:59:60
>>> print(dt2)
2012-07-01 00:00:00
>>> import time
>>> time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
1341093600.0
>>> t1
1341093600.0
>>> t2=time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
>>> t2 == t1
True
>>> time.localtime(time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1)))
time.struct_time(tm_year=2012, tm_mon=7, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=6, tm_yday=183, tm_isdst=1)http://cr.yp.to/proto/utctai.html """ |
|
Oh, mktime() returns the same timestamp with and without the leap second: >>> time.mktime((2012, 6, 30, 23, 59, 59, -1, -1, -1))
1341093599.0
>>> time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
1341093600.0
>>> time.mktime((2012, 7, 1, 0, 0, 0, -1, -1, -1))
1341093600.0 |
No, POSIX is an attempt to bring some sanity to the installed base of human calendars. The established standard tell's us that a year is 365 days. Wait, every 4-th year is 366 days, except some other rule every 400 years. POSIX says: fine as long as we can enumerate all YYYY-MM-DD's, we can live with it. But the line is drawn where each day is divided into 86,400 seconds. The problem is that unlike ancient astronomers who were finding better and better approximations to the ratio of two Earth's rotation periods (around the Sun and around itself) every few hundred years, modern astronomers will tell us how many seconds there will be in any given year with only a six month notice. |
|
POSIX timestamp doesn't count (literally) past/future leap seconds. >>> from datetime import datetime, timedelta
>>> str(datetime(1970,1,1) + timedelta(seconds=2**31-1))
'2038-01-19 03:14:07'If you use "right" timezone then mktime() may count leap seconds: $ TZ=right/UTC ./python
>>> import time
>>> time.mktime((2012, 6, 30, 23, 59, 59, -1, -1, -1))
1341100823.0
>>> time.mktime((2012, 6, 30, 23, 59, 60, -1, -1, -1))
1341100824.0
>>> time.mktime((2012, 7, 1, 0, 0, 0, -1, -1, -1))
1341100825.0It is a different time scale. There are no leap seconds in TAI: >>> str(datetime(1970,1,1, 0,0, 10) + timedelta(seconds=1341100825))
'2012-07-01 00:00:35'i.e., 2012-07-01 00:00:35 TAI that corresponds to 2012-07-01 00:00:00 TAI-UTC in the future (more than 6 months) is unknown but it is less It might be convenient to think about datetime as a broken-down (datetime(2012,6,30,23,59,60) - epoch) == The code [3] that silently truncates 60 to 59 when datetime Use case: parse timestamps that might include a leap second [4] [1] https://hpiers.obspm.fr/iers/bul/bulc/bulletinc.dat |
|
Sorry, I give up on this issue. I don't know how to fix it, nor if it's possible to fix it. |
|
Here's what mxDateTime uses: >>> import mx.DateTime
>>>
>>> t1 = mx.DateTime.DateTime(2012,6,30,23,59,60)
>>> t2 = mx.DateTime.DateTime(2012,7,1,0,0,0)
>>>
>>> t1
<mx.DateTime.DateTime object for '2012-06-30 23:59:60.00' at 7fbb36008d68>
>>> t2
<mx.DateTime.DateTime object for '2012-07-01 00:00:00.00' at 7fbb36008d20>
>>>
>>> t2-t1
<mx.DateTime.DateTimeDelta object for '00:00:00.00' at 7fbb35ff0540>
>>> (t2-t1).seconds
0.0
>>>
>>> t1 + mx.DateTime.oneSecond
<mx.DateTime.DateTime object for '2012-07-01 00:00:01.00' at 7fbb360083d8>It preserves the broken down values, but uses POSIX days of 86400 seconds per day to calculate time deltas. It's a compromise, not a perfect solution, but it prevents applications from failing for that one second every now and then. I don't believe there is a perfect solution, since what your application or users expect may well be different. All I can say is that raising exceptions in these rare cases is not what your users typically want :-) |
|
If you are using mx.DateTime make certain you do not use the .strftime method. If you use .strftime method and have a 60th second in your DateTime object it will crash python with no error message. This occurs because the .strftime method is fully inherited from Python's datetime.datetime. |
|
On 21.07.2015 22:15, dlroo wrote:
Thanks for the report. We will fix this in the next mxDateTime release. |
|
Is it possible to modify datetime so that the check_time_args function in the datetimemodule.c does not error when given a seconds value of greater than 59? I was thinking that if the seconds were greater than 59, the seconds are set to 59 and any extra seconds are kept in a book keeping "attribute" (not a real attribute because its C) that is accessible from the Python side? You would have to make the seconds argue passed by reference (thus returning a modified second). Also would want the book keeping value to be zero in nominal conditions. |
|
Please redirect this discussion to the recently opened datetime-sig mailing list. |
|
Could this be revisited? Especially now that datetime supports E.g. the NTP Leap second file: This get's synced on linux to The datetime also gained a fold argument, which if it is not wanted to support second values of 60 to at least be able to parse those. The 60th second of a minute is a reality with our current civil time keeping, so python should be able to handle it. |
|
One option to explore is to add a "leap seconds" field to datetime.datetime which can be negative (just in case someone decides to add negative leap seconds in the future). It can use in operations which involve time zones, it can be serialized/deserialized, but datetime.datetime.timestamp() would ignore this field ("drop" leap seconds on purpose). |
|
@vstinner No, that is not the correct way to deal with this. In UTC, a positive leap second is really the 60th second. A negative leap second the non-existence of second 59. Leap seconds are introduced either at the end of June or December of each year, so at YYYY-12-31T23:59:60 UTC or YYYY-06-30T23:59:60 UTC. This is first disconnected from the internal representation. The question is then how this can be internally represented and how the absolute date is created from e.g. a time since epoch like the unix time. The problem is here that: a) Unix Time UTC does not count leap seconds But from a user perspective, I think that python's datetime should: a) accept all valid UTC timestamps, including the instance of a leap second E.g. these should all work and e.g. correctly roundtrip via And e.g. this should be a timedelta of two seconds, not one: |
|
I'm inclined to mostly agree with @maxnoe here, though the changes to arithmetic might be a problem from a performance, data and backwards compatibility point of view. It seems to me that the minimum viable change here would be to allow Even though we know the dates of leap seconds, I think trying to restrict If we want something like "actual elapsed time", we need a data source for leap seconds, so maybe that part of it could / should be something that lives in At the end of the day, the big problem I have with the idea of adding support for leap seconds here is that while I frequently get asked about how If we go forward with that, I think the biggest open question is how this affects arithmetic and equality. Presumably |
You need to know them for calculating actual elapsed times between two UTC instances. E.g. cpython currently gets this wrong by 27 seconds: In [1]: from datetime import datetime, timezone
In [2]: t0 = datetime(1972, 1, 1, 0, 0, tzinfo=timezone.utc)
In [3]: t1 = datetime(2020, 1, 1, 0, 0, tzinfo=timezone.utc)
In [4]: (t1 - t0).total_seconds()
Out[4]: 1514764800.0
In [5]: from astropy.time import Time
In [6]: (Time('2020-01-01T00:00:00') - Time('1972-01-01T00:00:00')).to_value('s')
Out[6]: 1514764827.0There are also multiple efforts underway to add support for more timescales to different systems / programming languages (TAI for example to get a mononic timescale) in Linux (https://github.com/torvalds/linux/blob/4a57a8400075bc5287c5c877702c68aeae2a033d/include/uapi/linux/time.h#L64) and C++ (https://en.cppreference.com/w/cpp/chrono/tai_clock). To convert between e.g. between TAI and UTC, you also need a leap second table. Actually, it would make a lot of sense to change the internal representation to be unix time in TAI, not UTC and then convert on input/output in UTC. The C++ 20 proposal for calendars and timezones is here, which also includes dealing with leap seconds: C++ implementation library is here:
At the end of the day, the use case is to give instances in time with a precision of better than a minute and to represent all possible instances of our civil time keeping standard. Not being able to represent actual correct non-naïve datetimes is a bug. The time difference between two instances in time being off by the number of leap seconds introduced between the two timestamps is a bug. Side note: The story is even more complicated for dates before 1972 as there were fractional adjustments between UTC and TAI and the second was slowed down / sped up to keep the relationship between UTC and UT1. E.g. there are 8 seconds and 82 microseconds between |
|
This issue is open for 8 years. Maybe it's time to give up and accept that this issue is not important to motivate someone to through technical challenges? I propose to just close this issue. |
|
I'd propose to keep this issue open to allow for additional meaningful discussion since I agree with @maxnoe remarks. This issue need to be addressed when precision is valued over close enough approximations.
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: