MockClock rate=1.0

[boonhapus]

trio/trio/testing/_mock_clock.py

Line 90 in 1075239

self._rate = 0.0

Hi!

I've just found your project and am loving it so far. I am building an automation platform and had implemented this sort of MockClock synchronously, so I love what you've done here!

class MockClock(Clock):
    ...
    def __init__(self, rate=0.0, autojump_threshold=inf):
        # when the real clock said 'real_base', the virtual time was
        # 'virtual_base', and since then it's advanced at 'rate' virtual
        # seconds per real second.
        self._real_base = 0.0
        self._virtual_base = 0.0
        self._rate = 1.0

Would recommend you use an initial rate of 1.0, as it then implements a sort of "time travel" feature. Unless of course, you're intentionally wanting to keep the virtual time as dissimilar to real-time as possible. This is super cool and I've heavily based my clock off of your MockClock.

This is more of a non-issue as it's about a custom implementation of the clock, but wanted to bring it up and show you another possible use case for Clocks!

Thanks for the cool work, I'm excited to keep exploring.
- SN

[njsmith]

Heh, basically the reason the default is rate=0, autojump_threshold=inf is that my first version didn't have either of those features :-). The motivation was writing tests, where you want to be able to deterministically control the clock, so that you can control exactly what will happen at which point in the test. I actually haven't found any use cases for rate != 0 myself – I pretty much implemented it on a whim, since I realized it was easy to do, and have wondered whether I should keep the feature at all. And in actual practice what I end up using 99% of the time is MockClock(autojump_threshold=0.0).

Can you say more about how and why you want to use a MockClock(rate=1)? I'm afraid I'm not familiar with automation platforms. But I'm always interested in hearing about new use cases :-)

[boonhapus]

Apologies -- maybe I should be a bit more precise and say begin _rate=1.0. My program is essentially an execution environment for smaller programs ("apps"). It houses an internal clock, an event bus of sorts, and a bunch of registered user apps (python classes that do their own thing and utilize the automation api). I expose many methods that allow my users to schedule callbacks (sync or async!) to run under various contexts: at a specific time, after a given duration, on a matching incoming event, etc. Naturally, it would be pretty painful for a user to want to test their little application if the have a callback which only runs "every Saturday", or at a specific time far away in the future.

As-is in your implementation with self._rate=1.0, the virtual base time value will begin at the real base time value and then move forward in time based on the same sentiment you initially started with in MockClock. If you want to move at 2x the speed of time, set MockClock(rate=2.0), or if you want to move at a third the speed of time, set MockClock(rate=0.3) and so on. The key here being that we begin at the current time, instead of Unix epoch.

In my variant, I've also allowed my user to define a "start" to their clock, thus pulling back or pushing forward the virtual base attribute.

In your case, it might not make any sense at all for MockClock to keep the rate operative, however it is definitely a mandatory feature in my implementation of a Clock.

[njsmith]

Ah, I see! You're talking about wall clock / absolute time, like "January 7, 2018, 17:48:12.5839". All of Trio's time-keeping functionality is (currently) about monotonic / relative time. These are very similar and easily confused, but actually pretty different when you look at them carefully.

Wall clock time is matched to the calendar, so it makes sense to say things like "what's today's date", "what's the current day of the week", etc.; you have to think about complications like DST and leap seconds; and it's possible for a computer to be confused about the current wall clock time so you have to be prepared for the time to be set, sometimes to a time in the past. In Python you normally get this from time.time or datetime.now.

Monotonic time is simpler in most ways: it doesn't care about absolute time ("January 7 at 17:39:32"), it only cares about relative time ("this event was 1837.32491 seconds after that one"). So it doesn't try to keep track of its relationship to any absolute epoch – the actual values are arbitrary, the only guarantee is that if you check the time twice and then subtract those values, that difference tells you how many seconds have passed. Since it doesn't have to care about keeping in sync with the earth's rotation, it doesn't have to support stepping backwards, and it can guarantee that it increases by N every N seconds. (POSIX time can't make this guarantee even if we ignore clock setting, because it goes backwards during leap seconds!) Hence the name "monotonic" time. OTOH, in standard operating system implementations of monotonic time, the clock does stop while the computer is suspended, which is totally different from absolute time. In Python you normally get this from time.monotonic.

There are even other more complicated options too – according to man clock_gettime, Linux currently has 6 different kinds of clocks you can query... but plain old monotonic time like I described above is the one that everyone uses for things like I/O timeouts, and in particular is the kind of time that epoll and friends use. So that's what's baked into Trio, and used by trio.sleep and timeouts, and that's what MockClock is mocking.

Since MockClock is intended to be predictable for testing, and since the epoch for monotonic time is totally arbitrary, it always starts at time 0.0. (And Trio's real clock actually uses a randomized epoch, to help catch the common bug where someone uses time.monotonic instead of trio.current_time.)

There is an issue option for possibly adding wall clock / absolute time support to Trio: #173. (It might be better as a library than as part of the Trio core.) I haven't thought too much about the details of how that API might look, but definitely you'd want to have some kind of testing capability, and definitely you'd want to be able to set it to arbitrary times, advance by arbitrary amounts, etc.

Do you think it's really the ability to control the speed of the clock that you want, or mainly just absolute clock support? For example, if I had set up a job to run "every Saturday at midnight", I might write a test like:

1. Start running at virtual "Saturday 6 pm"
2. Move the clock forward to "Saturday 11:59pm"
3. Assert that my job hasn't run yet
4. Move the clock forward to "Sunday 12:01am"
5. Assert that my job just ran

But that doesn't involve ever running the clock at any multiple of real time, just manually stepping it. Do you have an example in mind where running it normally is useful?

[boonhapus]

All of Trio's time-keeping functionality is (currently) about monotonic / relative time.

Yeah, totally! This is how I originally implemented it as well.

Do you think it's really the ability to control the speed of the clock that you want, or mainly just absolute clock support?

If I had to pick, mainly absolute clock support. Controlling the speed of the clock would be a nice side-effect of this, as it's easier, as a human, to think of time passing "in seconds". I'm not sure how you could increase the speed of the clock - but I have not poured over the source code enough yet to determine how the clock advances. I've just been playing around with a custom implementation that would allow me to do what I'm looking for.

Do you have an example in mind where running it normally is useful?

A large part of my platform will involve scheduled callbacks, sometimes days/weeks in the future. As absolute time will be a large component of this, I have already modified your MockClock example and relegated all the annoying timezone, DST, etc interactions to the library Pendulum. The timezone bit is important as I would like to expose a scheduled time of at_sunset and at_sunrise and this naturally will change day-to-day across various timezones (I believe I can do this with the help of astral).

I don't believe I'm asking for anything from the project quite yet as I believe my use case can be handled by a custom implementation of the Clock!

Since MockClock is intended to be predictable for testing, and since the epoch for monotonic time is totally arbitrary, it always starts at time 0.0.

The main benefit of setting it as 1.0 is that the clock would align itself with the start time of the call to run. That's all I was looking for at first. 😄

[njsmith]

Setting rate=1.0 means that time passes at one virtual second per real second. It doesn't change the epoch or anything like that, and we can't provide a way to convert trio.current_time() to a calendar time – they're just on different scales. Which doesn't mean your use case isn't important, it just means we should think about it carefully :-).

My suggestion would be: in normal use, use a wall clock, like time.time or whatever. If you need to sleep until a given wall clock time... well, solving this problem is actually quite tricky because of things like time changes, but there are some hints as to how to solve it properly if you follow the link in #173, or a first-pass hack would be to compute a time delta using POSIX times, and then pretend that that's something you can pass to trio.sleep. (It'll work well enough most of the time.) And make sure this wall clock time stuff is encapsulated in some object, similar to Trio's clock.

Then, for testing, you can swap in a fake version of your clock, and you make the fake clock track an offset between what the Trio monotonic clock says, and whatever fake wall time you want to pretend it is. So if you say "okay fake clock, it's currently 2018-01-07 23:37:10.1349Z", then it converts that to a POSIX time, calls trio.current_time() to get Trio's idea of the time, and then stashes that offset somewhere. Later when you ask your fake clock what time it is, it does trio.current_time() + self._stored_offset. That way, for testing purposes you could combine trio.testing.MockClock with your fake clock, and all the magic -- including autojumping! -- would Just Work.

And you'd probably also want some other tools in your fake clock (or in a second fake clock implementation) for simulating stuff like DST transitions and leap seconds and NTP time updates.

[boonhapus]

Setting rate=1.0 means that time passes at one virtual second per real second. It doesn't change the epoch or anything like that

Either we are disconnected, or I am misunderstanding you. This is where I left off in my code, it's fancy .. because I am terrible with names. If you run...

from Clock import FancyClock
import trio

async def main():
    import time

    r_start = time.monotonic()
    v_start = clock.now

    for _ in range(9):
        print(f'real: {dt.now()} ... virtual: {clock.now}')
        await trio.sleep(1)

    print(f'real elapsed: {time.monotonic() - r_start} seconds')
    print(f'virt elapsed: {clock.now - v_start}')


if (__name__ == '__main__'):
    clock = FancyClock(rate=3)
    trio.run(main, clock=clock)

You'll get that the virtual clock has moved forward 9 seconds, to the 3 seconds of real time that has passed. Quite literally all I've changed so far in MockClock is where the private starting _rate begins (as self._rate=1.0, instead of self._rate=0.0), the private method _real_clock, and then I've added attributes for all the humans out there (self.now, self.timezone). I delegate the "human time" bits (DST transitions, leap seconds, etc) to the Pendulum library so as not to reinvent the wheel, as I'm definitely not as smart as those guys.

I will say that if you chose to support a timezone-based Clock implementation, I don't know how comfortable you are requiring your users to rely on another library. For my purposes, this works very nicely and so far, my dependencies are limited to trio and pendulum, so I'm cool with it. 😄

[njsmith]

I delegate the "human time" bits (DST transitions, leap seconds, etc) to the Pendulum library so as not to reinvent the wheel, as I'm definitely not as smart as those guys.

You're probably just as smart as them, but haven't studied as many bizarre and arcane details about computerized timekeeping. Which is probably a smart move on your part, because computerizing timekeeping has a ton of bizarre and arcane details. But... in that case, please trust me when I say, I do know a lot of bizarre and arcane details about computerized timekeeping, and what I'm trying to tell you is that the code you posted is fundamentally broken and this approach cannot work correctly. Any trio.abc.Clock implementation based on wall-clock time is buggy. Trio's clock does not track wall-clock time, it tracks "monotonic time", which is a related but fundamentally different thing. They aren't interchangeable. If you make a Clock that uses wall-clock time then it will not satisfy Trio's assumptions about how time works, and if/when your program breaks in obscure ways it will be your fault.

That doesn't mean you can't make this work, it just means that implementing a trio.abc.Clock that tracks wall-time isn't the way to do it. We can talk about what strategies would make sense, but first, do you understand what I'm saying about why your current approach doesn't work?

You might also find this article interesting: http://devetc.org/code/2014/01/21/timers-clocks-and-cocoa.html#fnref:unix-realtime

[oremanj]

I don't see an action item on this that's not covered by #173 so I'm closing it - feel free to reopen if I missed something.