Repo rules fail to extract unicode archives due to latin-1 hack

[jvolkman]

Description of the problem / feature request:

rules_go currently fails on some machines due to some unicode characters included in filenames within the Go source archive - specifically the character Ä. For Linux and macOS, Go archives are distributed as tar.gz files with pax headers in the tar files.

Affected systems include:

• ZFS volumes with the utf8only option. This is Ubuntu's default when choosing ZFS at install time.
• macOS (HFS+ and APFS require UTF-8 filenames)

Bazel uses Apache Commons Compress to extract tar archives. For most tar files, Commons Compress defers to the encoding specified by the JVM's -Dfile.encoding param, or the platform default. With ISO-8859-1 - Bazel's preference - UTF-8 encoded filename bytes in tar files basically pass through verbatim when extracted and everything works.

But when the tar entry has a pax path header, the path name is always decoded as UTF-8.

The character Ä in its composed form is unicode character U+00C4. In Java's internal UTF-16, it's simply represented as 0xC4. But encoded as UTF-8 it becomes the multi-byte sequence 0xC3 0x84 since 0xC4 as a single byte is not a valid UTF-8 value.

When Commons Compress parses a pax-formatted tar file with a filename containing Ä as the 0xC3 0x84 UTF-8 string, the resulting Java string contains the value 0xC4 after decoding. But this value is never re-encoded as UTF-8 when creating the file on the filesystem. Instead, Bazel uses the Java char values verbatim as long as they're < 0xff. An attempt to create a filename containing 0xC4 on a filesystem that requires UTF-8 filenames will fail.

But rules_go doesn't currently fail on macOS systems despite them also requiring UTF-8 filenames. This is because the darwin archives use decomposed representations of unicode characters. OS X has a history of preferring the decomposed forms over composed.

So instead of Ä being U+00C4 ("Latin Capital Letter A with Diaeresis"), it's U+0041 (just capital A) followed by U+0308 ("Combining Diaeresis"). Encoded in UTF-8 as seen in the macOS golang tarballs, the byte string is 0x41 0xCC 0x88. Decoded to a Java string (16-bit chars) it's 0x0041 0x0308. Coincidentally I presume, Bazel is able to extract this decomposed form on UTF-8 filesystems because it ignores the diaeresis and replaces it with a literal '?' character. So instead of Äfoo.go as is contained in the Go source archive, Bazel writes A?foo.go on macOS.

Like Linux on ZFS, Bazel fails to extract the linux archive on macOS as reported here.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

test.bzl

def _tar_round_trip_impl(ctx):
    ctx.file("Äfoo.txt", "boo!\n")
    ctx.execute(["tar", "--format=" + ctx.attr.format, "-czvf", "file.tar.gz", "Äfoo.txt"])
    ctx.extract("file.tar.gz", "out")
    ctx.file("BUILD.bazel", 'exports_files(["Äfoo.txt", "out/Äfoo.txt", "file.tar.gz"])', legacy_utf8=False)

tar_round_trip = repository_rule(
    implementation = _tar_round_trip_impl,
    attrs = {
	"format": attr.string(
            mandatory = True,
        ),
    },
)

WORKSPACE

load("//:test.bzl", "tar_round_trip")

tar_round_trip(
    name = "non_pax",
    format = "ustar",  # supported by both macos BSD tar and linux GNU tar
)

tar_round_trip(
    name = "pax",
    format = "pax",
)

BUILD

genrule(
    name = "non_pax_test",
    srcs = ["@non_pax//:out/Äfoo.txt"],
    outs = ["non_pax.txt"],
    cmd = """
        cp $(location @non_pax//:out/Äfoo.txt) "$@"
    """,
)

genrule(
    name = "pax_test",
    srcs = ["@pax//:out/Äfoo.txt"],
    outs = ["pax.txt"],
    cmd = """
	cp $(location @pax//:out/Äfoo.txt) "$@"
    """,
)

# Works
bazel build //:non_pax_test

# Fails, either due to not being able to write the file (utf8 filesystem),
# or because the written filename is mangled.
bazel build //:pax_test

What operating system are you running Bazel on?

Mac OS 10.15.7
Ubuntu 20.04 with a ZFS root partition with utf8only enabled (the default for Ubuntu's ZFS support).

What's the output of bazel info release?

release 4.0.0

Have you found anything relevant by searching the web?

bazel-contrib/rules_go#2771
#374 - pretty generic issue regarding filename characters.
#7055 - an issue with the same problematic file in the Go archive, but targeted at Darwin only.

[illicitonion]

I happened to do some debugging of this yesterday - the probematic call chain is that

bazel/src/main/java/com/google/devtools/build/lib/bazel/repository/CompressedTarFunction.java

Line 104 in 2910d3d

try (OutputStream out = filePath.getOutputStream()) {

ends up calling into

bazel/src/main/native/unix_jni.cc

Lines 1045 to 1058 in 2910d3d

	extern "C" JNIEXPORT jint JNICALL
	Java_com_google_devtools_build_lib_unix_NativePosixFiles_openWrite(
	JNIEnv *env, jclass clazz, jstring path, jboolean append) {
	const char *path_chars = GetStringLatin1Chars(env, path);
	int flags = (O_WRONLY | O_CREAT) | (append ? O_APPEND : O_TRUNC);
	int fd;
	while ((fd = open(path_chars, flags, 0666)) == -1 && errno == EINTR) {
	}
	if (fd == -1) {
	PostException(env, errno, path_chars);
	}
	ReleaseStringLatin1Chars(path_chars);
	return fd;
	}

Ideally we could use the raw buffer of bytes from the tar header to pass to open, rather than round-tripping via a String, but I'm not sure how to holistically go about that...

[aiuto]

cc @tetromino This is a nice summary of the MacOS file handling strangeness I was talking about.

[brandjon]

Addressing the unicode handling situation for strings (replacing latin-1 with utf-8) is on the schedule for this year, perhaps in Q2.

[QIvan]

hi! Here is a victim of this bug (I'm using Ubuntu 20.04 with ZFS and, seems like, it was set up with an utf8only flag, according to the original description).
I tried to investigate the problem yesterday and copy some parts of Bazel's CompressedTarFunction.java to a small snippet. And, probably, I've found a workaround for this issue with just replacing filePath.getOutputStream() to new FileOutputStream(filePath.getPathFile())
So this code works for me https://gist.github.com/QIvan/bf88d152c31a35eccf162845ce05c455

[jvolkman]

@QIvan per my understanding this breaks other cases in which an archive contains a filename as a string of non-UTF-8 compatible bytes and the destination filesystem will accept it. #7757 has more context.

[brandjon]

Update: We haven't had bandwidth to work on eliminating the latin-1 hack, but may reconsider after 2023 Q1.

[aiuto]

I started a playground to observe some actual behaviors. We'll need to know how macos behaves when it sees NFC style names.
https://github.com/aiuto/bazel_samples/tree/main/utf8

[iancha1992]

A fix for this issue has been included in Bazel 6.4.0 RC2. Please test out the release candidate and report any issues as soon as possible. Thanks!