Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux from macOS issues #1962

Closed
henryiii opened this issue Aug 11, 2024 · 4 comments · Fixed by #1961
Closed

Linux from macOS issues #1962

henryiii opened this issue Aug 11, 2024 · 4 comments · Fixed by #1961

Comments

@henryiii
Copy link
Contributor

henryiii commented Aug 11, 2024

Description

Building a linux wheel from macOS is producing warnings and created broken files:

$ pipx run cibuildwheel --only cp313t-manylinux_x86_64
...

Copying project into container...

    + mkdir -p /project
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.FinderInfo'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.xcode.CreatedByBuildSystem'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.xcode.CreatedByBuildSystem'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.FinderInfo'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.FinderInfo'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemTextContentLanguage'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseVersion'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseLabels'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemKeyphraseConfidences'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.lastuseddate#PS'

We can hide that warning, I think, but I'm also getting junk in the files:

/project/src/._register_transforms.cpp:1:1: warning: null character(s) ignored
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      | ^
/project/src/._register_transforms.cpp:1:2: error: stray ‘\5’ in program
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |  ^
/project/src/._register_transforms.cpp:1:3: error: stray ‘\26’ in program
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |   ^
/project/src/._register_transforms.cpp:1:4: error: stray ‘\7’ in program
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |    ^
/project/src/._register_transforms.cpp:1:5: warning: null character(s) ignored
    1 |     Mac OS X                2   ~      �                                      ATTR       �   �                     �     com.apple.lastuseddate#PS    xUF`    +��
      |     ^
...

Using gnu-tar instead works fine. I don't see a --no-xattrs option (https://stackoverflow.com/questions/51655657/tar-ignoring-unknown-extended-header-keyword-libarchive-xattr-security-selinux).

Build log

No response

CI config

No response

@joerick
Copy link
Contributor

joerick commented Aug 11, 2024

Can we go back to using docker cp rather than the tar-pipes now? I recall we had to change due to a bug in docker, but that was a while ago.

(One thing to be careful about would be file ownership/permissions if we go back)

@mayeut
Copy link
Member

mayeut commented Aug 12, 2024

c.f. #1434 for other reference.
Other than gnu-tar, the --format gnutar argument could be used on macOS.

@henryiii
Copy link
Contributor Author

henryiii commented Aug 13, 2024

--format {ustar|pax|cpio|shar}  Select archive format

I assume we could use pax on both ends, though. (Edit: doesn't seem to work)

Do you have a ref to the docker bug?

@henryiii
Copy link
Contributor Author

This works:

diff --git a/cibuildwheel/oci_container.py b/cibuildwheel/oci_container.py
index 0a5fa325..39516f35 100644
--- a/cibuildwheel/oci_container.py
+++ b/cibuildwheel/oci_container.py
@@ -229,7 +229,7 @@ def copy_into(self, from_path: Path, to_path: PurePath) -> None:
         if from_path.is_dir():
             self.call(["mkdir", "-p", to_path])
             subprocess.run(
-                f"tar cf - . | {self.engine.name} exec -i {self.name} tar --no-same-owner -xC {shell_quote(to_path)} -f -",
+                f"tar -c --format ustar -f - . | {self.engine.name} exec -i {self.name} tar --format ustar --no-same-owner -xC {shell_quote(to_path)} -f -",
                 shell=True,
                 check=True,
                 cwd=from_path,

This seems to be a ref for the docker cp bug: #1782 (not the bug, but apparently this workaround kills Windows too, and there's some links there, including a mention that Docker server 24 should be fine)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants