Switch from PhantomJS to Chrome's headless mode

[macbre]

PhantomJS 2.5 may never reach the stable state (GitHub milestone shows that it's 27% complete). Chrome has obviously more stable development team 😉 and leads the improve the web initiative - webp and spdy/http2 to name just a few.

Let's consider porting phantomas (and rename it to chromas? 😄 ) to use Chrome (and Firefox) run in a headless mode for performance auditing / monitoring.

Chrome vs PhantomJS vs SlimerJS

Resources

• https://www.chromestatus.com/feature/5678767817097216 (since Chrome 59)
• https://developer.mozilla.org/en-US/Firefox/Headless_mode (since Firefox 55)
• https://chromedevtools.github.io/devtools-protocol/
• https://github.com/ChromeDevTools/debugger-protocol-viewer
• https://github.com/cyrus-and/chrome-remote-interface - sniffing HTTP traffic
• https://medium.com/@marco.luethy/running-headless-chrome-on-aws-lambda-fa82ad33a9eb
• https://www.aymen-loukil.com/en/blog-en/google-puppeteer-tutorial-with-examples/
• Archiving the project: suspending the development ariya/phantomjs#15344 - PhantomJS repository has been archived
• https://docs.travis-ci.com/user/gui-and-headless-browsers/#using-the-chrome-addon-in-the-headless-mode

Libraries

• https://github.com/adieuadieu/node-headless-chrome (work in progress, Windows support is missing)
• https://github.com/GoogleChrome/puppeteer

Puppeteer troubleshooting

• https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md#setting-up-chrome-linux-sandbox

Initial work

Is being done in this gist - https://gist.github.com/macbre/fded35f86893bace033e3790079bee76

[OzzyCzech]

+1

[braincomb]

+1. Chromium is twice as faster than PhantomJS.

[macbre]

v1.20.0 was just tagged and pushed to npm. It's the last version of phantomas that's powered by PhantomJS.

Let's try what headless Chromium has to offer :)

[macbre]

Puppeteer's Chromium binary dependencies:

macbre@bjornoya:/tmp/node_modules/puppeteer/.local-chromium/linux-526987/chrome-linux$ ldd chrome | grep 'not found'
	libX11-xcb.so.1 => not found
	libXtst.so.6 => not found
	libXss.so.1 => not found
	libasound.so.2 => not found
	libatk-bridge-2.0.so.0 => not found
	libgtk-3.so.0 => not found
	libgdk-3.so.0 => not found

$ sudo apt-get install libx11-xcb1 libxtst6 libxss1 libasound2  libatk-bridge2.0-0  libgtk-3-0

[AdriwanKenoby]

I'd like to help you with this project. Could you give me some advice to start with, I was thinking of using yargs to intuitively parse the command. The page object inherits directly from EventEmmiter and a lot of phantomas code is no longer needed. That's the big gap between phantomJS and puppeteer.

#!/usr/bin/env node

"use strict";

const puppeteer = require("puppeteer");

let argv = require("yargs")
	.usage("$0 <url>")
	.command(
		"$0 <url>",
		"collect metric for given url",
		(yargs) => {
			yargs.positional("url", {
				describe: "the url to use to collect metrics",
				type: "string"
			});
		},
		async (argv) => {
			const browser = await puppeteer.launch();
			const page = await browser.newPage();
			await page.goto(argv.url);
			let metrics = await page.metrics();
			console.log(metrics);
			await browser.close();
		}
	)
	.help().argv;

[macbre]

@AdriwanKenoby, thanks for your interest!

I'd definitely like to remove the extra code that was required to make PhantomJS work (and to be able to run SlimerJS as an engine as well) and be able to talk to Node.js layer (ipc.js sending JSON messages via stderr and passing events between both layers). Now we can focus on supporting just a single platform.

[macbre]

Guys,

It's been a while since the last activity in this thread. I want to give phantomas a new kick and refactor it to be more modular and use Puppeteer (headless Chromium) instead of PhantomJS (rest in peace, you've served us well!).

Here's my list of ideas, directions, notes:

• Get rid of two rendering engines (SlimerJS and PhantomJS) and rely solely on headless Chromium (provided by Puppeteer package from Google)
• phantomas v2.0 will be a npm module only - no command line tool, various reporters, etc. Just a basic options handling, core modules and events emitting. Results will be returned as an object via Promise.
• In phantomas-v2 branch I will port modules one by one and make sure tests pass
• Command line tool and various reporters will be moved to a new macbre/phantomas-cli repository
• Provide macbre/phantomas docker container with CLI tool and reporters installed for an easy deployment

Some basic work has started here - https://gist.github.com/macbre/fded35f86893bace033e3790079bee76#file-script-js

@gmetais - your feedback on what will make your life with YellowLabTools easier are more than welcome here :)

[gmetais]

This is great news! (and it could lead to a v2.0 on YLT too!)
I'll help you as much as I can. Probably not on the core development, but by testing, adapting modules and later developing new modules.

One thing that would be great is switching offenders to a JSON format instead of a string. Which means that the CLI reporter will need to do the reverse job to display a string again. Or a string version of the offender could be kept in a JSON key.

I also consider removing from YLT the "Timeline" part, so YLT will stop spying the various jQuery functions. A JavaScript complexity index would still be cool, but a lighter one. Maybe Puppeteer gives access to the Chrome Timeline? Anyway, that's not a priority at all.

How can I help right now?

[macbre]

https://travis-ci.org/macbre/phantomas/jobs/473573512 - phantomas can now talk with Puppeteer's Chrome. And some tests do pass :)

Within a next week or two, all core modules should work as expected. Next step will be testing the npm module against various sites to find incompatibilities between PhantomJS and Chrome. @gmetais, this will be something that you can help with. And thanks!

✗ Errored » 138 honored ∙ 130 broken ∙ 18 errored (11.054s) 

[gmetais]

Surely I will.

[macbre]

Work in progress update:

• phantomas.incrMetric and phantomas.emit called from page scope JS code are now passed to Node.js layer
• test suite for /dom-operations.html now passes

[macbre]

Progress update:

• more modules were ported to run in Chrome: domMutations, localStorage, console, documentHeight
• domComplexity is almost refactored
• - 43abdfa - offenders can now be checked in integration tests

[macbre]

Progress update:

• phantomas now uses its custom user agent, i.e. phantomas/2.0.0-beta (HeadlessChrome/72.0.3617.0)
• dc4231b - jquery module refactored, offenders are now better structured (key / value set instead of plain text)
• a04766f - events module followed

[macbre]

Tests status so far:

✗ Broken » 254 honored ∙ 28 broken (13.168s)

[macbre]

More progress :)

✗ Broken » 272 honored ∙ 19 broken (42.727s)

[macbre]

✗ Broken » 279 honored ∙ 12 broken (12.621s) 

Now it's time to integrate analyze-css into phantomas v2.0.

[macbre]

analyze-css is back in 0712b40

✗ Broken » 315 honored ∙ 6 broken (15.264s)

[macbre]

118ce1b made all remaining tests pass :)

✓ OK » 323 honored (18.380s) 

[macbre]

966ae00 - TravisCI checks are green again 😃

[macbre]

✓ OK » 402 honored (35.484s) 

[macbre]

#726 has been merged into phantomas-v2 branch. Now it's time to start working on macbre/phantomas-cli.

@gmetais, feel free to start testing using examples/index.js Node.js script. Thanks!

[macbre]

Documentation is now automatically generate from metadata JSON file that is built by parsing the code comments / annotations.

• modules and metrics documentation
• events

@gmetais - I hope that these resources will make developing on top of phantomas a bit easier task.

[gmetais]

Hi @macbre,
Sorry for the delay. Yes, thanks a lot for this new doc!

Should I add my comments about v2 to this post or create a new issue each time?

[gmetais]

I see you count brotli compressed files as gzipped (here: https://github.com/macbre/phantomas/blob/phantomas-v2/core/modules/requestsMonitor/requestsMonitor.js#L256)

For clarity, shouldn't Phantomas separate gzipRequests and brotliRequests and maybe add a new compressedRequests?

[gmetais]

Watch up here: https://github.com/macbre/phantomas/blob/phantomas-v2/core/modules/requestsMonitor/requestsMonitor.js#L303

When a request is answered as a redirect, entry.transferedSize is -1. So incrementing the contentLength metric actually decreases it it by 1. Not a big deal, though!

What I find strange is that entry.responseSize is -1 too. The body size is zero, right, but headers are not.

[macbre]

#753 has been merged and the first 2.0.0 npm pre-release package released.