html5lib parses extremely slow certain type of content

[DartBot]

This issue was originally filed by [email protected]

---

The following code demonstrates a very slow parsing speed:

  http.read('https://www.facebook.com/MARCA').then((c) {
    htmlParser.parse(c);
  });

For most websites out there, it's fast, but for a few and this site in particular, it parses in around 20-25 seconds.

[sethladd]

Is this when running on the VM or via dart2js ?

Routing to VM because they might be interested in performance reports like this.

(FWIW I've heard the same report when trying to parse a large SVG document. Fast via dart2js but slower via VM.)

---

cc @jmesserly.
Added Area-VM, Triaged labels.

[DartBot]

This comment was originally written by [email protected]

---

Sorry I wasn't clear. This happens on the VM. Latest stable.

[iposva-google]

Can you please give a bit more context what your example is doing? In particular I am missing how you allocated the htmlParser object and where you got its class from.

Since this is filed against parsing, I assume we can just download the example URL once and feed a test from the file, correct?

---

Added NeedsInfo label.

[DartBot]

This comment was originally written by [email protected]

---

Sure, I don't see why we can't save it on a file. Here's a full code snippet to try against:

  library test;

  import 'package:http/http.dart' as http;
  import 'package:html5lib/parser.dart' as htmlParser;

  void main() {
    http.read('https://www.facebook.com/MARCA').then((c) {
      var w = new Stopwatch()..start();
      htmlParser.parse(c);
      print(w.elapsed);
    });
  }

It prints 11½ seconds for me. This happens consistently.

[sethladd]

Here is a directory with all the pub setup ready to go.

unzip
run pub get
dart bin/xml.dart

---

Attachment:
testxml.zip (2.31 KB)

[iposva-google]

Thanks for the reproduction instructions.

It turns out that for the sample in question we spend a lot of time copying strings due to tokenizer.dart appending characters to a string with interpolation:

currentStringToken.data = "${currentStringToken.data}-${data}";

Using a StringBuffer would be the right way to go in this situation. Also there are many other places like this in this particular source file.

---

Set owner to @jmesserly.
Removed Area-VM label.
Added Area-Pkg, Library-Html5lib, Accepted labels.

[sethladd]

Thanks for the analysis.

[jmesserly]

Nice tracking that down! wow, turns out this is a really old bug :)
dart-archive/html#3

Patches are welcome for this. Probably pretty easy now that we have proper token classes and APIs use types. Originally the Python code used tokens in a very untyped way (they were just maps, "data" had different meaning in different places depending on the tokenizer state) ... now that it's sorted out into "currentStringToken" and APIs are typed, it's probably pretty easy to find the .data concatenations.

---

Removed the owner.
Added PatchesWelcome label.

[kevmoo]

Added Pkg-Html5Lib label.

[kevmoo]

Removed Library-Html5lib label.

[DartBot]

This issue has been moved to dart-lang/tools#1101.