StringInputStream gets stuck at large enough stdin input

[DartBot]

This issue was originally filed by [email protected]

---

What steps will reproduce the problem?

1. Run this program with the txt file attached:

main() {
  var a;
  var ios = new StringInputStream(stdin);
  while ((a = ios.read()) !== null) {
    print(a.length);
  }
  exit(0);
}

1. Run it like this:

dart_release wrangler.dart < mega_sample.txt

1. It should get stuck there.

What is the expected output? What do you see instead?

It should print the length of the input string. But it gets stuck like in an endless loop and doesn't print anything and doesn't exit either.

What version of the product are you using? On what operating system?

Bleeding-edge Dart. Linux manga 3.0.0-13-generic-pae #­22-Ubuntu SMP Wed Nov 2 15:17:35 UTC 2011 i686 i686 i386 GNU/Linux

Please provide any additional information below.

I'm attaching the sample file. Here's the output when the program works for a slightly smaller input file:

$ time dart_release wrangler.dart < mega_sample.txt
837083

real 0m4.274s
user 0m4.236s
sys 0m0.036s

And when it gets stuck and has to be canceled:

$ wc mega_sample.txt
  4960 10416 840472 mega_sample.txt

$ time dart_release wrangler.dart < mega_sample.txt
^C

real 0m52.132s
user 0m52.083s
sys 0m0.032s

---

Attachment:
mega_sample.txt (820.77 KB)

[DartBot]

This comment was originally written by [email protected]

---

Added Area-Library, Triaged labels.

[whesse]

This is due to extremely high new space usage by StringInputStream. Using the
--gc-verbose option (found by using the --print-flags VM option - there is no --help option for the VM), we see that reading this 820 KB file takes more than 20 MB in new space - it performs scavenges in a loop, without making progress, with the default new space allocation of 32 MB (half of that is 16 MB, the default max new space heap size). It completes if we specify --new-gen-heap-size=44. This is a bug - this is requiring more than 25 bytes of live allocation per byte read from the file, and it is not getting promoted to old space.

---

cc @sgjesse.
cc @madsager.

[madsager]

Set owner to @sgjesse.
Added Accepted label.

[whesse]

The problem is the StringBuffer _result field of _StringDecoderBase in string_stream.dart. Characters get added individually to this buffer as long
as the decoder has not exhausted its input. If we limit the size of this StringBuffer, and stop decoding when it reaches a certain size, then the classes
that use this don't retry the read to get more data, and return null before reporting all the data.

Once toString has been called on the buffer, then the buffer can be freed, and we are using much less memory per character. We need some way to do this after about every 10000 - 100000 characters.

I think we need a 2-level scheme of StringBuffers, so that every 100,000 characters get written to a string, and those strings are accumulated in a higher level StringBuffer. Other than that, we would need a way in which all uses of a decoder would keep calling the decoder until they got all the data in large chunks.

[whesse]

A simpler solution would be to use List<int> charcodes to accumulate the string, instead of a StringBuffer, since Lists can be efficiently extended, and create the string from it when it is read from the decoder. There is no good reason to use a StringBuffer, since we are adding a character at a time, by knowing its character code.

Or maybe StringBuffer itself should know to concatenate itself to a single string, and
free all the substrings, when it gets to be thousands of items long. I don't see an issue with that, unless we need to copy the initial big string thousands of times while accumulating a megabyte string in the buffer. That brings us back to the two-level tree solution, but implemented in StringBuffer, not its users.

[whesse]

Added a CL that fixes this, http://codereview.chromium.org/8639004/, by accumulating the character codes in a List<int>, not a StringBuffer. StringBuffers are mainly useful for accumulating many strings, of varying lengths. List<int> is better for single characters.

---

cc @sgjesse.
Set owner to @whesse.

[whesse]

Fixed by commit r1777, but note bug 571 : http://code.google.com/p/dart/issues/detail?id=571
FileInputStream and stdin do not work correctly with StringInputStream at the present time. The read handler and close handler of StringInputStream will not be called.

---

Added Fixed label.