Daily grunt work problems again?

[zepumph]

@chrisklus mentioned that something he would have thought would have been updated 2 weeks ago isn't. I'll look into it.

[zepumph]

The log seemed to have complete, but I updated the cron task to better output errors:

# daily run of automated grunt tasks at 3:01am every morning
1 3 * * * cd /data/share/phet/automated-grunt-work/perennial; git pull; ./bin/daily-grunt-work.sh >> ~/daily-grunt-work.log 2>> ~/.daily-grunt-work-err
or.log

I also added a print statement for each section for clarity

I then found multiple repos that seemed to have not pushed. I don't know why. The commit message that was not pushed was:
update copyright dates from daily grunt work. This often works and pushes, so I'm unsure why it didn't here. To be safe I think I will add a push all at the end of the script.

I then saw this on bayes with status.sh:

chipper                                  master
?? js/dual/dual/
circuit-construction-kit-ac              master

So I fixed the cp command from phetsims/chipper#1018. Tagging @samreid so he notes the change.

There were then merge conflicts while cleaning things up in scenery and bamboo, so I manually fixed these.

I also deleted the phet-io-wrapper-sonification because it is old.

I also want to tag @jessegreenberg because of his recent work with this script, I really feel like we are in this together.

I hope this runs successfully tonight!

[zepumph]

I'll check back in tomorrow.

[zepumph]

The error log was helpful! I found a problem in binder, and set up emails so I will get notifications about the scripts problems.

[zepumph]

This is the problem line, likely localTestingURL is not consistent on bayes, let's use createServer instead. I'm thinking like #175.

[zepumph]

I fixed the binder issue I found over in #175. I'll check back to see what the next run of daily grunt work looks like.

[samreid]

So I fixed the cp command from phetsims/chipper#1018. Tagging @samreid so he notes the change.

On my local, cp -r perennial/js/dual/ chipper/js/ is copying the files into chipper/js/SimVersion.js and chipper/js/execute.js. Not into chipper/js/dual. I'm not sure what accounts for this discrepancy, but based on https://unix.stackexchange.com/a/228637 I'll try

cp -R perennial/js/dual/. chipper/js/dual

which works on mine and seems clear.

[zepumph]

That works well on my windows! Thanks.

[zepumph]

I am working on better error handling here for emailed errors.

[zepumph]

I ran it manually, and did successfully get an email of errors this time. One included a puppeteer error, I think about binder:

(node:95915) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:98438) UnhandledPromiseRejectionWarning: Error: Could not find browser revision 818858. Run "PUPPETEER_PRODUCT=firefox npm install" or "PUPPETEER_PRODUCT=firefox yarn install" to download a supported Firefox browser binary.
    at ChromeLauncher.launch (/data/share/phet/automated-grunt-work/perennial/node_modules/puppeteer/lib/cjs/puppeteer/node/Launcher.js:79:23)
    at async /data/share/phet/automated-grunt-work/perennial/js/scripts/hook-pre-commit.js:104:25
(Use `node --trace-warnings ...` to show where the warning was created)
(node:98438) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:98438) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
To https://github.com/phetsims/proportion-playground.git
   b708c22..88dc7ed  master -> master
(node:141941) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
Everything up-to-date
(node:146050) UnhandledPromiseRejectionWarning: Error: Navigation failed because browser has disconnected!
    at /data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:51:147
    at /data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62
    at Array.map (<anonymous>)
    at Object.emit (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:43)
    at CDPSession.emit (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
    at CDPSession._onClosed (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:256:14)
    at Connection._onClose (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:138:21)
    at WebSocket.<anonymous> (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/node/NodeWebSocketTransport.js:17:30)
    at WebSocket.onClose (/data/share/phet/automated-grunt-work/binder/node_modules/ws/lib/event-target.js:136:16)
    at WebSocket.emit (events.js:315:20)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:146050) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:146050) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[zepumph]

The important line in the above is Error: Navigation failed because browser has disconnected!.

See phetsims/chipper#958 and phetsims/chipper#1032 (comment) for previous versions of this problem as it related to using puppeteer for PhET-iO API generation. Poking through those, I see that binder isn't awaiting browser.close(), so I will do that and try again.

[zepumph]

One of things I saw in the errors email was

npm WARN [email protected] No license field.

npm WARN [email protected] No license field.
....

The below commit should help.

[zepumph]

The only error left in the log that matters is this one:

(node:156187) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:159403) UnhandledPromiseRejectionWarning: Error: Could not find browser revision 818858. Run "PUPPETEER_PRODUCT=firefox npm install" or "PUPPETEER_PRODUCT=firefox yarn install" to download a supported Firefox browser binary.
    at ChromeLauncher.launch (/data/share/phet/automated-grunt-work/perennial/node_modules/puppeteer/lib/cjs/puppeteer/node/Launcher.js:79:23)
    at async /data/share/phet/automated-grunt-work/perennial/js/scripts/hook-pre-commit.js:104:25
(Use `node --trace-warnings ...` to show where the warning was created)
(node:159403) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:159403) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:176983) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
Everything up-to-date
(node:180083) UnhandledPromiseRejectionWarning: Error: Protocol error (Runtime.callFunctionOn): Target closed.
    at /data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:217:63
    at new Promise (<anonymous>)
    at CDPSession.send (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:216:16)
    at ExecutionContext._evaluateInternal (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:204:50)
    at ExecutionContext.evaluate (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:110:27)
    at DOMWorld.evaluate (/data/share/phet/automated-grunt-work/binder/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:91:24)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:180083) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:180083) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I tried to figure out where it is coming from, but couldn't find the repo it related to. I think for the next night, I'm going to pipe the entire log into the email to gain the context. That said I'm still not sure I will be able to fix it.

[zepumph]

I can't figure out how to pipe to an email, so I'm going to just log the whole thing out, and then pipe the errors in an email in a separate step. Updated aboe.

[samreid]

Would it help to run ./bin/daily-grunt-work.sh locally, so you can iterate and comment out subtasks? Or does the problem only happen on the remote?

[zepumph]

Well the problem comes from the puppeteer qunit task, and seems to be about the particular version of puppeteer installed/running on bayes, because I don't ever get that error running locally.

[samreid]

Got it, thanks for elaborating. I'll link to phetsims/chipper#114 which may help us get reproducible node_modules in case that is the problem.

[zepumph]

I think it is more about how puppeteer is running cross platform, as opposed to node modules, but I'll keep it in mind.

[zepumph]

phet-io-wrapper-classroom-activity was mispelled in the package.json, so it was failing some grunt tasks.

[zepumph]

I just looked at the log that came in last night, and I realized that I forgot to pipe error output to the same log.

[zepumph]

I saw that the error from #229 (comment) was coming from running grunt update-copyright-dates on bamboo (at least last night). I can consistently get it to reproduce on bayes with:

cd /data/share/phet/automated-grunt-work/bamboo/
node ../perennial/js/scripts/hook-pre-commit.js


[phet-admin@bayes bamboo]$ node ../perennial/js/scripts/hook-pre-commit.js
(node:22875) UnhandledPromiseRejectionWarning: Error: Could not find browser revision 818858. Run "PUPPETEER_PRODUCT=firefox npm install" or "PUPPETEER_PRODUCT=firefox yarn install" to download a supported Firefox browser binary.
    at ChromeLauncher.launch (/data/share/phet/automated-grunt-work/perennial/node_modules/puppeteer/lib/cjs/puppeteer/node/Launcher.js:79:23)
    at async /data/share/phet/automated-grunt-work/perennial/js/scripts/hook-pre-commit.js:104:25
(Use `node --trace-warnings ...` to show where the warning was created)
(node:22875) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:22875) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[zepumph]

I cannot reproduce this on master locally. So this appears to be a bayes-specific (or linux server specific) problem.

[zepumph]

I read through puppeteer/puppeteer#6560 but didn't find any trouble that made sense why it works on other sim repos, but not for bamboo.

I updated to puppeteer version 10.1.0 (latest)
I blew away node modules and reinstalled
I blew away bamboo and re downloaded.

I'm a bit stumped. I don't know why that single repo is having this trouble, and nothing else.

[samreid]

I ran rm -rf node_modules/ and npm install in perennial with our admin account, and now bamboo node ../perennial/js/scripts/hook-pre-commit.js seems to be working OK on bayes. @zepumph can you please verify?

[zepumph]

Wow. That is great news!! Thanks for your help. I confirmed that I didn't see this issue in the log on bayes from last night's run. Things are so much better than before this issue was opened. I am ready to close this issue.