Implement a fast path for character counting in wc. #3735
Conversation
When wc is invoked with only the -m flag, we only need to count the number of Unicode characters in the input. In order to do so, we don't actually need to decode the input bytes into characters. Rather, we can simply count the number of non-continuation bytes in the UTF-8 stream, since every character will contain exactly one non-continuation byte. On my laptop, this speeds up `wc -m odyssey1024.txt` from 745ms to 109ms.
|
Wow that's excellent! Are there any edge cases here we need to think about with invalid utf-8? For example, could there be an off by one error if the last byte is a continuation byte? Looking at the other specialized function, it looks like we don't handle that case at all, but it might still be interesting to document and to check what GNU does. |
The only corner case is when the input stream ends with continuation bytes, which means the stream is invalid UTF-8 to begin with. I tested GNU coreutils wc 8.32. For a file containing three 0x80 bytes (all continuation bytes), |
|
Interesting! Thanks for checking! So the mac |
|
Correct, the Mac |
When wc is invoked with only the -m flag, we only need to count the
number of Unicode characters in the input. In order to do so, we don't
actually need to decode the input bytes into characters. Rather, we can
simply count the number of non-continuation bytes in the UTF-8 stream,
since every character will contain exactly one non-continuation byte.
On my laptop, this speeds up
wc -m odyssey1024.txtfrom 745ms to109ms.