setTimeout callback executed before delay elapses #10154

marzelin · 2016-12-06T19:21:02Z

Version: 7.2.0
Platform: Ubuntu 16.10 64-bit

When the delay is set to 1, for about 5% of cases the logged time between executing setTimeout() and the callback execution is 0.

// main.js
const timeJustBeforeCallingSetTimeout = Date.now();
setTimeout(() => {
  const callbackExecutionTime = Date.now();
  console.log(callbackExecutionTime - timeJustBeforeCallingSetTimeout);
}, 1);

# bash script that runs above code 1000 times and counts cases when 0 is logged
rm -f results.txt
for ((n=0; n<1000; n++))
do
  node main.js >> results.txt
done
cat results.txt | grep 0 | wc -l

When the delay value is set to 2, there are no cases where the logged time is less than 2, which is an expected behavior.

The text was updated successfully, but these errors were encountered:

bnoordhuis · 2016-12-06T19:37:50Z

Cannot reproduce locally on a FC25 system.

A possible explanation is that Date.now() is backed by a CLOCK_REALTIME time source (gettimeofday) whereas node/libuv uses a CLOCK_MONOTONIC clock. If ntpdate is adjusting the system clock, you might see <= 0 intervals between two Date.now() calls.

What happens when you use process.hrtime() instead?

marzelin · 2016-12-06T20:13:22Z

For code with process.hrtime() used instead of Date.now(), the percentage of cases with measured delay that is less than 1ms grows to 30%. The lowest time observed is ~930000ns.

// main.js
const timeJustBeforeCallingSetTimeout = process.hrtime();
setTimeout(() => {
  const delayBetweenMainCodeAndCallbackExecutionInNanoSec =
    process.hrtime(timeJustBeforeCallingSetTimeout)[1];
  console.log(delayBetweenMainCodeAndCallbackExecutionInNanoSec);
}, 1);

# bash script that runs js code 1000 times
rm -f results.txt
for ((n=0;n<1000;n++))
do
  node main.js >> results.txt
done
cat results.txt | grep '^.\{6\}$' | wc -l

Fishrock123 · 2016-12-07T17:17:06Z

@marzelin Could you try process.binding('timer_wrap').Timer.now() and see if it makes a difference? That is what timers use internally. (I think process.hrtime also uses that but not 100% sure.)

marzelin · 2016-12-07T18:14:48Z

When time difference is measured with process.binding('timer_wrap').Timer.now() there are no cases where the difference is less than 1, which is the expected behavior.

// main.js
const timeJustBeforeCallingSetTimeout = process.binding('timer_wrap').Timer.now();
setTimeout(() => {
  const callbackExecutionTime = process.binding('timer_wrap').Timer.now();
  console.log(callbackExecutionTime - timeJustBeforeCallingSetTimeout);
}, 1);

# bash script that runs js code 1000 times and counts cases where time difference is 0
rm -f results.txt
for ((n=0;n<1000;n++))
do
  node main.js >> results.txt
done
cat results.txt | grep 0 | wc -l

Fishrock123 · 2016-12-08T15:14:24Z

@bnoordhuis any idea?

bnoordhuis · 2016-12-08T18:16:26Z

@marzelin Are you running Ubuntu inside a VM or a container? I know VirtualBox has (had?) clock skew bugs in the past, perhaps other products do too.

marzelin · 2016-12-08T19:09:55Z

@bnoordhuis Ubuntu is loaded directly by UEFI, no VMs are involved. On Windows 10 (same PC and node version) results are similar.

bnoordhuis · 2016-12-08T22:36:43Z

I have been able to reproduce this a few times now (over 100,000s of runs) but so far I've only been able to establish that the kernel seems to return early from the epoll_wait() system call for no discernible reason whatsoever. This is with a stock 4.8.8-300.fc25.x86_64 kernel.

@marzelin Re: Windows: did you test with Date.now(), Timer.now() or process.hrtime()?

marzelin · 2016-12-08T23:16:21Z

@bnoordhuis All of them, but I did only a few tests there. setTimeout() seems to behave in the same way as on Ubuntu.

bnoordhuis · 2016-12-09T12:24:30Z

I'm coming around to the idea that this might be a hardware issue. I've so far only been able to reproduce on one system and I noticed that this system's hardware clock drifts by one or two seconds every few hours.

@marzelin Can you post the output of:

$ cat /sys/devices/system/clocksource/clocksource*/available_clocksource
# And
$ cat /sys/devices/system/clocksource/clocksource*/current_clocksource

Your system probably only has one hardware clock. Try switching to different sources (e.g. from tsc to hpet) and see if the problem goes away.

EDIT: Oh, and please disable anything that syncs the clock; ntpd, ntpdate cron jobs, etc.

marzelin · 2016-12-09T14:26:47Z

@bnoordhuis

$ cat /sys/devices/system/clocksource/clocksource*/available_clocksource
tsc hpet acpi_pm

$ cat /sys/devices/system/clocksource/clocksource*/current_clocksource
tsc

I've got the same results when clocksource is set to hpet or acpi_pm.

marzelin · 2016-12-09T15:38:56Z

I did some other tests and it seems like process.binding('timer_wrap').Timer.now() takes some arbitrary point of time and looks only at number of milliseconds, omitting the fractional part.

So for example if we have point in time:
123 ms and 990,000 ns
after 10,000 ns Timer.now() will tell 1ms has passed, but in reality only 0,01 ms has passed as process.hrtime will show.

xk · 2016-12-10T10:57:21Z

Node's timers have always been borken, check it out:
https://gist.github.com/xk/eb2e75c11ef0435a43438a373f8c0113

Edit:
Ooops, I was testing this with an old node v0.10.26 which errs all the time but newer nodes do not, at least since v0.12.0, or so it seems... Great! Thanks! :-)

Fishrock123 · 2016-12-16T15:16:21Z

@bnoordhuis any further thoughts?

bnoordhuis · 2016-12-19T12:14:06Z

I'm still investigating this although I probably won't revisit it until January. You can assign it to me if you want.

jBarz · 2017-01-13T04:52:56Z

I suspect that this is because we track loop->time in libuv in milliseconds.
loop->time = uv__hrtime(UV_CLOCK_FAST) / 1000000
So the difference between 2 different loop->time values could potentially show a 1 millisecond difference even when 1 nanosecond has actually elapsed and thus trigger timers too early.
This could be fixed by tracking loop->time in nanoseconds instead of milliseconds.
I tested this change and it seems to fix the issue (no more 0 second delays).
Let me know if this solution makes sense.
jBarz@84fbba5

saghul · 2017-01-13T14:24:53Z

Can you bring this up on the libuv repo? Thanks!

…

On Thu, Jan 12, 2017 at 10:53 PM, John Barboza ***@***.***> wrote: I suspect that this is because we track loop->time in libuv in milliseconds. loop->time = uv__hrtime(UV_CLOCK_FAST) / 1000000 So the difference between 2 different loop->time values could potentially show a 1 millisecond difference even when 1 nanosecond has passed. This could be fixed by tracking loop->time in nanoseconds instead of milliseconds. I tested this change and it seems to fix the issue (no more 0 second delays). Let me know if this solution makes sense. ***@***.*** <jBarz@84fbba5> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10154 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATYGBCyzKHaiXbrsoMEy9vfQ6ne5YjIks5rRwNJgaJpZM4LFyZq> .

-- /Saúl bettercallsaghul.com

jBarz · 2017-01-14T02:45:28Z

I have submitted a PR on the libuv repo
libuv/libuv#1191

Trott · 2017-07-15T17:09:33Z

Should this be closed (or at least labeled blocked) unless and until the libuv issue gets resolved?

Trott · 2017-08-10T04:49:25Z

Labeled blocked as (IIUC) nothing can happen until libuv/libuv#1191 lands first.

ikokostya · 2017-08-11T19:06:20Z

Looks like I have the same issue which stability reproduced on the following test:

'use strict';

const assert = require('assert');

const DELAY = 10;

function call() {
    const start = process.hrtime();
    setTimeout(() => {
        const diff = process.hrtime(start);
        const [seconds, nanoseconds] = diff;
        const ms = seconds * 1e3 + nanoseconds * 1e-6;
        console.log(ms);
        assert(ms >= DELAY);
        call();
    }, DELAY);
}

call();

[kostya@arch ~]$ node -v
v8.1.3
[kostya@arch ~]$ node t
11.467041
9.704634
assert.js:60
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: false == true
    at Timeout.setTimeout [as _onTimeout] (/home/kostya/t.js:14:9)
    at ontimeout (timers.js:488:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:283:5)

Next 3 consecutive runs:

[kostya@arch ~]$ node t
12.213059
9.567727999999999
assert.js:60
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: false == true
    at Timeout.setTimeout [as _onTimeout] (/home/kostya/t.js:14:9)
    at ontimeout (timers.js:488:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:283:5)
[kostya@arch ~]$ node t
11.104375
9.571886
assert.js:60
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: false == true
    at Timeout.setTimeout [as _onTimeout] (/home/kostya/t.js:14:9)
    at ontimeout (timers.js:488:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:283:5)
[kostya@arch ~]$ node t
10.487898999999999
10.68757
10.291964
10.234295999999999
10.246357999999999
10.16991
10.187873
10.203797999999999
9.151824
assert.js:60
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: false == true
    at Timeout.setTimeout [as _onTimeout] (/home/kostya/t.js:14:9)
    at ontimeout (timers.js:488:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:283:5)

ikokostya · 2017-09-04T13:10:23Z

@Trott Could you revisit this issue? Looks like this issue can be solved on nodejs side libuv/libuv#1191 (comment).

Trott · 2017-09-05T03:08:44Z

@ikokostya Your repro triggers the assertion for me on Node.js 8.4.0 but not on 9.0.0-pre (current master branch). Any chance you can compile from the master branch or grab the most recent nightly and see if it's fixed for you there?

ikokostya · 2017-09-05T15:18:33Z

The same error:

[kostya@arch node-v9.0.0-nightly20170904b24e269a48-linux-x86]$ ./bin/node -v
v9.0.0-nightly20170904b24e269a48
[kostya@arch node-v9.0.0-nightly20170904b24e269a48-linux-x86]$ ./bin/node t
10.539683
10.501904
10.166082
10.186332
10.193605
10.178288
10.197446
10.175659
10.184133
10.203318
10.162325
10.159507
10.157172
9.153143
assert.js:41
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: false == true
    at Timeout.setTimeout [as _onTimeout] (/home/kostya/Downloads/node-v9.0.0-nightly20170904b24e269a48-linux-x86/t.js:14:9)
    at ontimeout (timers.js:471:11)
    at tryOnTimeout (timers.js:305:5)
    at Timer.listOnTimeout (timers.js:265:5)

Trott · 2017-09-06T04:01:28Z

Interesting that I can't replicate on macOS.

@Fishrock123 @misterdjules Any thoughts on the fix for #10154 (comment) suggested by @jBarz at libuv/libuv#1191 (comment)?

ikokostya · 2017-11-05T20:20:55Z

Any news? I still get error for nodejs 9.0.0.

apapirovski · 2018-04-11T12:20:26Z

@bnoordhuis Can anything be done here?

bnoordhuis · 2018-04-11T14:06:31Z

@jBarz Per libuv/libuv#1191 (comment), do you want to open a pull request here?

apapirovski · 2018-04-11T14:14:44Z

@bnoordhuis @jBarz I'm not sure the proposed fix there solves the issue. We already have something very similar (setting timeRemaining = 1) which works fine for most cases but it only runs when diff < msecs which in the reproduction above from @ikokostya is not actually the case.

jBarz · 2018-04-11T18:23:42Z

I've tried the fix myself and it seems to resolve the issue.

@apapirovski Are you able to recreate the error using @ikokostya test case with the proposed fix?

You're right, the case is not diff < msecs
However, the case fails because (TimerWrap.now() - timer._idleStart) has at most a 1 millisecond rounding error because they both capture time by dividing the high resolution timer by 1000000.
The fix is simply adding 1ms to timeRemaining for all cases.
I can open a PR if this sounds ok.

apapirovski · 2018-04-12T12:03:26Z

@jBarz yeah, I see what you mean now. Not a huge fan of adding 1ms to all timers (the magic number is a bit of a code smell) that run through that block but I'm not really seeing another solution. I suppose it depends on whether we consider this important to fix or if infrequent 1ms inaccuracy is acceptable. FWIW it should happen a lot less now than it used to when this issue was made.

jBarz · 2018-04-12T14:22:09Z

Yea, I agree that adding 1ms to all timers is not ideal.
I guess we could consider that test case as invalid because it is using the high resolution process.hrtime to measure accuracy of the low res uv_now.
So nothing to fix here.

bnoordhuis · 2018-04-12T21:20:34Z

Okay, I'll close this out then.

apapirovski · 2018-05-12T17:15:01Z

This can be fixed by: #20555

(I've had the test case running for an hour with no failures. Pretty sure it resolves the issue.)

Edit: I've taken out that bit because it made that PR too big but it'll be fixed in a follow-up PR. Either way, I'm keeping this open since it can be solved by Node.js.

Refactor Timers to behave more similarly to Immediates by having a single uv_timer_t handle which is stored on the Environment. No longer expose timers in a public binding and instead make it part of the internalBinding. PR-URL: #20894 Fixes: #10154 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Gus Caplan <[email protected]>

This is no longer necessary in the only place it was used (timers). PR-URL: #20894 Fixes: #10154 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Gus Caplan <[email protected]>

The timers directory test, utilizing FakeTime, has not worked in quite a while and is not truly testing Node.js behaviour. If a similar test is necessary it would be better suited to libuv on which Node.js relies for timers functionality. PR-URL: nodejs#20894 Fixes: nodejs#10154 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Gus Caplan <[email protected]>

The timers directory test, utilizing FakeTime, has not worked in quite a while and is not truly testing Node.js behaviour. If a similar test is necessary it would be better suited to libuv on which Node.js relies for timers functionality. PR-URL: #20894 Fixes: #10154 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Backport-PR-URL: #22039 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Minwoo Jung <[email protected]>

In the modified test file, sleep() is backed by setTimeout(), and some Node runtimes might execute setTimeout callback before the specified delay, meaning that tmi.getValue() will occasionally return `9`. This produces a flaky test. An extra ms leeway in the assertion works round the issue. nodejs/node#10154 Test name updated to reflect that the timer's accuracy does not have single-digit precision

addaleax added libuv Issues and PRs related to the libuv dependency or the uv binding. question Issues that look for answers. timers Issues and PRs related to the timers subsystem / setImmediate, setInterval, setTimeout. labels Dec 6, 2016

Fishrock123 assigned bnoordhuis Dec 19, 2016

gibfahn mentioned this issue Jan 15, 2017

timer: track loop time in nanoseconds libuv/libuv#1191

Closed

Trott added the blocked PRs that are blocked by other issues or PRs. label Aug 10, 2017

bnoordhuis closed this as completed Apr 12, 2018

apapirovski reopened this May 12, 2018

apapirovski assigned apapirovski and unassigned bnoordhuis May 13, 2018

apapirovski mentioned this issue May 22, 2018

src: remove TimerWrap #20894

Closed

3 tasks

apapirovski closed this as completed in 6f63f8d Jun 25, 2018

lpinca mentioned this issue Aug 20, 2018

test: fix flaky test-http-client-timeout-option-with-agent #22403

Closed

2 tasks

misterdjules mentioned this issue Nov 7, 2018

make metrics.test.js less flaky restify/node-restify#1715

Merged

michael-c-snyk mentioned this issue Nov 19, 2020

test: Fix a flaky test snyk/cli#1533

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setTimeout callback executed before delay elapses #10154

setTimeout callback executed before delay elapses #10154

marzelin commented Dec 6, 2016

bnoordhuis commented Dec 6, 2016

marzelin commented Dec 6, 2016 •

edited

Loading

Fishrock123 commented Dec 7, 2016

marzelin commented Dec 7, 2016

Fishrock123 commented Dec 8, 2016

bnoordhuis commented Dec 8, 2016

marzelin commented Dec 8, 2016

bnoordhuis commented Dec 8, 2016

marzelin commented Dec 8, 2016

bnoordhuis commented Dec 9, 2016 •

edited

Loading

marzelin commented Dec 9, 2016

marzelin commented Dec 9, 2016 •

edited

Loading

xk commented Dec 10, 2016 •

edited by Fishrock123

Loading

Fishrock123 commented Dec 16, 2016

bnoordhuis commented Dec 19, 2016

jBarz commented Jan 13, 2017 •

edited

Loading

saghul commented Jan 13, 2017 via email

jBarz commented Jan 14, 2017

Trott commented Jul 15, 2017

Trott commented Aug 10, 2017

ikokostya commented Aug 11, 2017 •

edited

Loading

ikokostya commented Sep 4, 2017 •

edited

Loading

Trott commented Sep 5, 2017

ikokostya commented Sep 5, 2017

Trott commented Sep 6, 2017

ikokostya commented Nov 5, 2017

apapirovski commented Apr 11, 2018

bnoordhuis commented Apr 11, 2018

apapirovski commented Apr 11, 2018

jBarz commented Apr 11, 2018 •

edited

Loading

apapirovski commented Apr 12, 2018

jBarz commented Apr 12, 2018

bnoordhuis commented Apr 12, 2018

apapirovski commented May 12, 2018 •

edited

Loading

setTimeout callback executed before delay elapses #10154

setTimeout callback executed before delay elapses #10154

Comments

marzelin commented Dec 6, 2016

bnoordhuis commented Dec 6, 2016

marzelin commented Dec 6, 2016 • edited Loading

Fishrock123 commented Dec 7, 2016

marzelin commented Dec 7, 2016

Fishrock123 commented Dec 8, 2016

bnoordhuis commented Dec 8, 2016

marzelin commented Dec 8, 2016

bnoordhuis commented Dec 8, 2016

marzelin commented Dec 8, 2016

bnoordhuis commented Dec 9, 2016 • edited Loading

marzelin commented Dec 9, 2016

marzelin commented Dec 9, 2016 • edited Loading

xk commented Dec 10, 2016 • edited by Fishrock123 Loading

Fishrock123 commented Dec 16, 2016

bnoordhuis commented Dec 19, 2016

jBarz commented Jan 13, 2017 • edited Loading

saghul commented Jan 13, 2017 via email

jBarz commented Jan 14, 2017

Trott commented Jul 15, 2017

Trott commented Aug 10, 2017

ikokostya commented Aug 11, 2017 • edited Loading

ikokostya commented Sep 4, 2017 • edited Loading

Trott commented Sep 5, 2017

ikokostya commented Sep 5, 2017

Trott commented Sep 6, 2017

ikokostya commented Nov 5, 2017

apapirovski commented Apr 11, 2018

bnoordhuis commented Apr 11, 2018

apapirovski commented Apr 11, 2018

jBarz commented Apr 11, 2018 • edited Loading

apapirovski commented Apr 12, 2018

jBarz commented Apr 12, 2018

bnoordhuis commented Apr 12, 2018

apapirovski commented May 12, 2018 • edited Loading

marzelin commented Dec 6, 2016 •

edited

Loading

bnoordhuis commented Dec 9, 2016 •

edited

Loading

marzelin commented Dec 9, 2016 •

edited

Loading

xk commented Dec 10, 2016 •

edited by Fishrock123

Loading

jBarz commented Jan 13, 2017 •

edited

Loading

ikokostya commented Aug 11, 2017 •

edited

Loading

ikokostya commented Sep 4, 2017 •

edited

Loading

jBarz commented Apr 11, 2018 •

edited

Loading

apapirovski commented May 12, 2018 •

edited

Loading