Add a tokenize worker to speed up code highlighting

[kumar303]

Tokenizing a file to highlight the code makes the file render slowly. See #402 for STR -- a minified diff like this is the most dramatic example.

I confirmed in benchmarks that the tokenizer is the main offender so it's worth our time to try switching to a worker model.

Specifically, while profiling a production build from d07fbc3 and loading the minified diff from #402, tokenize() accounts for 95% of the time, i.e. 7.32 seconds.

[kumar303]

We need a Web Worker for this feature, which won't be supported until create-react-app 3.1 (roadmap). PR: facebook/create-react-app#5886 (it's supported now)

Before Web Workers are officially supported, we would need something like react-app-rewired or craco to set output.globalObject = 'this' in the webpack config, at least until webpack/webpack-dev-server#1595 is fixed. I have a basic PoC working but we could run into deployment issues.

@willdurand @bobsilverberg maybe we should try and wait until CRA 3.1 is released? What do you think?

[bobsilverberg]

Sounds reasonable to me.

[willdurand]

I think waiting for CRA 3.1 would be wise, even though the current PR does not support TypeScript. I guess we'll workaround that if needed, though.

I like this suggestion in the meantime: #402 (comment). I would even say that minified code is barely readable anyway, with or without syntax HL (I don't read minified code a lot but usually I find it more confusing when syntax HL is enabled).

[kumar303]

Good call on disabling highlighting on large files. I'll do that in #942

[kumar303]

This is supported in create-react-app out of the box now. This article explains how to do it with a workerize! import: https://kentcdodds.com/blog/speed-up-your-app-with-web-workers

[bobsilverberg]

Removing this from the project as we determined this is not something we want to do at this time. See #1411 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If you think this bug should stay open, please comment on the issue with further details. Thank you for your contributions.