Puppeteer Chrome clients fill up /tmp on bayes

[zepumph]

Looks like #143 has caused /tmp to explode in size, and caused potentially some build failures? I'm not sure, but I looked into things and it is storing user data from the chrome sessions there:

puppeteer/puppeteer#1791

I can set a different user data section with --user-data-dir=BLARG

Here was a script to clean up the user data after use, but I don't think I'll go here first.

let fs = require('fs-extra');
const puppeteer = require('puppeteer');

let browser = await puppeteer.launch();
let chromeTmpDataDir = null;

// find chrome user data dir (puppeteer_dev_profile-XXXXX) to delete it after it had been used
let chromeSpawnArgs = browser.process().spawnargs;
for (let i = 0; i < chromeSpawnArgs.length; i++) {
    if (chromeSpawnArgs[i].indexOf("--user-data-dir=") === 0) {
        chromeTmpDataDir = chromeSpawnArgs[i].replace("--user-data-dir=", "");
    }
}

// ...

browser.close();

if (chromeTmpDataDir !== null) {
    fs.removeSync(chromeTmpDataDir);
}

process.exit(0);

[zepumph]

Ok. I tested this commit locally and saw that this was correctly being set within my git repos. Up next, we will want to delete it every so often. I think a cron job for that could be helpful. I'll wait though to speak with @mattpen.

I restarted the client instances with puppeteer user data being stored in /data/share/phet/continuous-client/tmp/puppeteerUserData

[zepumph]

Oh! And I also cleaned up the puppeteer user data in /tmp too.

[zepumph]

• Has there been a build failure since stopping continuous-client on bayes? No
• Is this problem only in tmp? I don't think so, when I start continuous-client/ and then sudo du -sh /tmp, it is less than a gb. I think that the space is being used somewhere else on /.

[mattpen]

It looks like /tmp is the culprit, but the files have been deleted so they don't show up with du. Running sudo lsof | grep -i deleted shows deleted files, and when the continuous-client is running there are more than 50,000 deleted chromium files in /tmp.

@zepumph mentioned there is a way to use a different temporary directory for chromium (puppeteer? pm2?) which I would hope would alleviate the problem.

The other solution could be to grow the root partition. It is currently 10GB which is not very large. There are still around 3TB unallocated on bayes so we have plenty of room to grow.

However, I think we would want to rule out that this behavior is not buggy somehow before we grow the fs.

[zepumph]

`I got it!!!

Didn't work:
I first attempted to change the temp dir within puppeteer. I found an environment called PUPPETEER_TMP_DIR here: https://github.com/puppeteer/puppeteer/blob/v13.5.2/docs/api.md#environment-variables which seemed like it would do our job, but with this script:

( async () => {
  process.env.PUPPETEER_TMP_DIR = '/c/Users/Michael/PhET/git/sandbox/temp/';
  const puppeteer = require( 'puppeteer' );
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto( 'https://google.com' );
  await browser.close();
} )();

I got an error running locally though which basically was an issue making the puppeteer folders within the new temp dir. Perhaps this was permissions problems, but while trying to research things, I found the actual issue:

What worked:
In puppeteer/puppeteer#1834 (comment) I found that by default, there is a flag that sets the temp directory to root instead of /dev/shm. Ignoring it (see above commit). fixes the issue. There is 125GM available in /dev/shm, and these processes use between 1-15% of it from what I've seen. I have restarted the continuous-client.

Thanks a ton to @mattpen and @jonathanolson for helping me work through this.