[8.0.0] Fix most Unicode encoding bugs

[iancha1992]

Bazel aims to support arbitrary file system path encodings (even raw byte sequences) by attempting to force the JVM to use a Latin-1 locale for OS interactions. As a result, Bazel internally encodes Strings as raw byte arrays with a Latin-1 coder and no encoding information. Whenever it interacts with encoding-aware APIs, this may require a reencoding of the String contents, depending on the OS and availability of a Latin-1 locale.

This PR introduces the concepts of internal, Unicode, and platform strings and adds dedicated optimized functions for converting between these three types (see the class comment on the new StringEncoding helper class for details). These functions are then used to standardize and fix conversion throughout the code base. As a result, a number of new end-to-end integration tests for the handling of Unicode in file paths, command-line arguments and environment variables now pass.

Full support for Unicode beyond the current active code page on Windows is left to a follow-up PR as it may require patching the embedded JDK.

• Replace ad-hoc conversion logic with the new consistent set of helper functions.
• Make more parts of the Bazel client's Windows implementation Unicode-aware. This also fixes the behavior of SetEnv on Windows, which previously would remove an environment variable if passed an empty value for it, which doesn't match the Unix behavior.
• Drop the charset parameter from all methods related to parameter files. The ISO-8859-1 vs. UTF-8 choice was flawed since Bazel's internal string representation doesn't maintain any encoding information - ISO-8859-1 just meant "write out raw bytes", which is the only choice that matches what arguments would look like if passed on the command line.
• Convert server args to the internal string representation. The arguments for requests to the server were already converted to Bazel's internal string representation, which resulted in a mismatch between --client_cwd and --workspace_directory if the workspace path contains non-ASCII characters.
• Read the downloader config using Bazel's filesystem implementation.
• Make MacOSXFsEventsDiffAwareness UTF-8 aware. It previously used the GetStringUTF JNI method, which, despite its name, doesn't return the UTF-8 representation of a string, but modified CESU-8 (nobody ever wants this).
• Correctly reencode path strings for LocalDiffAwareness.
• Correctly reencode the value of user.dir.
• Correctly turn ExecRequest fields into strings for ProcessBuilder for bazel --batch run. This makes it possible to reenable the test_consistent_command_line_encoding test, fixing Fix (and reenable) run_test.test_consistent_command_line_encoding #1775.
• Fix encoding issues in TargetCompleteEvents.
• Fix encoding issues in SubprocessFactory implementations.
• Drop obsolete warning if file.encoding doesn't equal ISO-8859-1 as file names are encoded with sun.jnu.encoding now.
• Consistently reencode internal strings passed into and out of FileSystem implementations, e.g. if reading a symlink target. Tests are added that verify the interaction between FileSystem implementations and the Java (N)IO APIs on Unicode file paths.

Fixes #1775.

Fixes #11602.

Fixes #18293.

Work towards #374.

Work towards #23859.

Closes #24010.

PiperOrigin-RevId: 694114597
Change-Id: I5bdcbc14a90dd1f0f34698aebcbd07cd2bde7a23

Commit a58fe3f

[meteorcloudy]

FYI @fmeum please take a look at the test failures

[fmeum]

@iancha1992 This patch should fix the build, could you apply it?

diff --git a/src/main/java/com/google/devtools/build/lib/BUILD b/src/main/java/com/google/devtools/build/lib/BUILD
index 71deae4ac9..898ef5a81d 100644
--- a/src/main/java/com/google/devtools/build/lib/BUILD
+++ b/src/main/java/com/google/devtools/build/lib/BUILD
@@ -325,6 +325,7 @@ java_library(
         "//src/main/java/com/google/devtools/build/lib/actions:file_metadata",
         "//src/main/java/com/google/devtools/build/lib/actions:important_output_handler",
         "//src/main/java/com/google/devtools/build/lib/actions:localhost_capacity",
+        "//src/main/java/com/google/devtools/build/lib/actions:middleman_type",
         "//src/main/java/com/google/devtools/build/lib/actions:package_roots",
         "//src/main/java/com/google/devtools/build/lib/actions:resource_manager",
         "//src/main/java/com/google/devtools/build/lib/actions:runfiles_tree",

[tjgq]

The original (head) PR is getting rolled back - please don't submit this cherry pick until we figure out what to do

[haxorz]

The original (head) PR is getting rolled back - please don't submit this cherry pick until we figure out what to do

fc6b384