join: add support for multibyte separators

[jtracey]

GNU join used to not support any separators over 1 byte in length, and had tests ensuring such separators would return an error. As such, years ago, we removed our support for multibyte unicode characters, and added support for single non-unicode bytes to match their behavior, and got GNU tests passing. Now, GNU join supports multibyte separators when they're valid characters in the current locale's encoding. We're not in a position to add proper locale support yet, but this adds the next best thing, and should get the new GNU tests to pass in CI by assuming UTF-8 encodings. This is just straight up wrong on Windows, which always uses UTF-16, but that was always weird, and the "right" thing on Windows at this point would be to reject all possible separators other than null, which doesn't sound very useful.

The first commit fixes our tests to better reflect the new GNU behavior. The second commit gets multibyte separators working in a way closely resembling what we were already doing. But this bloats the separator enum we were using, and adds a bunch of clones or (de)refs, which profiling showed added a notable hit to performance (though still significantly faster than GNU), all to support a few extra bytes that will likely be rarely used. The third commit therefore changes the enum into a trait, so we can use generics instead of matching, getting us most of that lost performance back (or possibly all, it's hard to get consistent measurements).

[sylvestre]

clippy is complaining on:

 error: this argument is passed by value, but not consumed in the function body
   --> src/uu/join/src/join.rs:366:56
    |
366 |     fn new<Sep: Separator>(string: Vec<u8>, separator: Sep, len_guess: usize) -> Self {
    |                                                        ^^^ help: consider taking a reference instead: `&Sep`
    |
    = help: for further information visit [https://rust-lang.github.io/rust-clippy/master/](https://rust-lang.github.io/rust-clippy/master/index.html#needless_pass_by_value)

[github-actions]

GNU testsuite comparison:

GNU test failed: tests/join/join. tests/join/join is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/join/join-utf8 is no longer failing!

[sylvestre]

Congrats! The gnu test tests/join/join-utf8 is no longer failing!

well done :)

[github-actions]

GNU testsuite comparison:

GNU test failed: tests/join/join. tests/join/join is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/join/join-utf8 is no longer failing!
Congrats! The gnu test tests/timeout/timeout is no longer failing!

[github-actions]

GNU testsuite comparison:

GNU test failed: tests/join/join. tests/join/join is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/join/join-utf8 is no longer failing!

[github-actions]

GNU testsuite comparison:

Congrats! The gnu test tests/join/join-utf8 is no longer failing!

[jtracey]

Sorry for all the force pushes, made a series of dumb mistakes by trying to fix things too quickly. The new additional commit adds a test that should have been there already and would have saved me some time (join handles whitespace separators differently than every other kind of separator).

[github-actions]

GNU testsuite comparison:

Congrats! The gnu test tests/join/join-utf8 is no longer failing!

[sylvestre]

please document these two, the diff isn't obvious :)

[jtracey]

Sure, done.

[jtracey]

It occurs to me that even though GNU join doesn't allow multibyte separators with invalid or multiple code points, I don't actually know why they have this restriction. When it was only single byte separators allowed, that made perfect sense, as it simplifies the algorithms and makes them faster. But once you allow any code point, I don't know why you would bother restricting it. I suppose there are some additional performance hacks you could do by assuming UTF-8 and knowing the separator is at most 4 bytes, but GNU supports other encodings, and we're not doing any such hacks. Should we just remove those checks, and allow using any arbitrary byte string as the separator?

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/tail/inotify-dir-recreate (fails in this run but passes in the 'main' branch)
GNU test failed: tests/timeout/timeout. tests/timeout/timeout is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/join/join-utf8 is no longer failing!

[sylvestre]

I don't actually know why they have this restriction

maybe ask on their mailing list?

Should we just remove those checks, and allow using any arbitrary byte string as the separator?

Sure but in a different PR
and documented here:
https://github.com/uutils/coreutils/blob/main/docs/src/extensions.md