Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tarfile failure with long user ID #1013

Closed
artwyman opened this issue Feb 15, 2017 · 6 comments · Fixed by #2447
Closed

tarfile failure with long user ID #1013

artwyman opened this issue Feb 15, 2017 · 6 comments · Fixed by #2447

Comments

@artwyman
Copy link

Note this is a follow-on from the same effort which produced #1012, but I think the bugs are independent. I switched to Python 3 and the CrystaxNDK in an attempt to avoid NDK build errors, and got far enough to run into this.

While the dist's build.py is building its tar archive, I get ValueError("overflow in number field"):

Traceback (most recent call last):
  File "/usr/local/bin/p4a", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/site-packages/pythonforandroid/toolchain.py", line 754, in main
    ToolchainCL()
  File "/usr/local/lib/python3.5/site-packages/pythonforandroid/toolchain.py", line 343, in __init__
    getattr(self, command_method_name)(unknown)
  File "/usr/local/lib/python3.5/site-packages/pythonforandroid/toolchain.py", line 94, in wrapper_func
    func(self, args)
  File "/usr/local/lib/python3.5/site-packages/pythonforandroid/toolchain.py", line 589, in apk
    build_args = build.parse_args(args)
  File "/Users/testuser/.python-for-android/dists/unnamed_dist_1/build.py", line 488, in parse_args
    make_package(args)
  File "/Users/testuser/.python-for-android/dists/unnamed_dist_1/build.py", line 248, in make_package
    make_tar('assets/private.mp3', tar_dirs, args.ignore_path)
  File "/Users/testuser/.python-for-android/dists/unnamed_dist_1/build.py", line 197, in make_tar
    tf.add(fn, afn)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tarfile.py", line 1938, in add
    self.addfile(tarinfo, f)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tarfile.py", line 1960, in addfile
    buf = tarinfo.tobuf(self.format, self.encoding, self.errors)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tarfile.py", line 802, in tobuf
    return self.create_ustar_header(info, encoding, errors)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tarfile.py", line 821, in create_ustar_header
    return self._create_header(info, USTAR_FORMAT, encoding, errors)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tarfile.py", line 916, in _create_header
    itn(info.get("uid", 0), 8, format),
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tarfile.py", line 211, in itn
    raise ValueError("overflow in number field")
ValueError: overflow in number field

It seems the overflow is in the "uid" field, and I notice that my UID on my Mac (part of a corporate domain) is pretty high at 1730938684. Looking at the package in tarfile.py it seems this is a limitation of the USTAR format which the python-4-android builder is selecting specifically. The default GNU format can support much larger numbers, though I presume the unpacking code used in the Android package might not support it?

I was able to work around this problem by running as a different local user account, so long as that account owns both the .python-for-android directory and the source input directory.

Since UID information isn't relevant when unpacking on Android it would be nice to discard it and make this not matter, if using the GNU tar format isn't an option. It doesn't look like the tarfile package gives the ability to do this directly, but you could chmod all the files to root (in a temporary location) before compressing them.
@pts-dorianpula
Copy link

pts-dorianpula commented Feb 19, 2017 •
edited
Loading

Similar to the OP, I ran into the same issue. Also running in an user account (managed via corporate AD system), so I have a very high uid. I can confirm the same issue occurs under Linux (64-bit Ubuntu 16.04) for both Python 2.7.12 and 3.5.2:

  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build/dists/weather/build.py", line 543, in parse_args
    make_package(args)
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build/dists/weather/build.py", line 255, in make_package
    make_tar('assets/private.mp3', tar_dirs, args.ignore_path)
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build/dists/weather/build.py", line 201, in make_tar
    tf.add(fn, afn)
  File "/usr/lib/python2.7/tarfile.py", line 2023, in add
    self.addfile(tarinfo, f)
  File "/usr/lib/python2.7/tarfile.py", line 2046, in addfile
    buf = tarinfo.tobuf(self.format, self.encoding, self.errors)
  File "/usr/lib/python2.7/tarfile.py", line 1006, in tobuf
    return self.create_ustar_header(info)
  File "/usr/lib/python2.7/tarfile.py", line 1025, in create_ustar_header
    return self._create_header(info, USTAR_FORMAT)
  File "/usr/lib/python2.7/tarfile.py", line 1121, in _create_header
    itn(info.get("uid", 0), 8, format),
  File "/usr/lib/python2.7/tarfile.py", line 212, in itn
    raise ValueError("overflow in number field")
ValueError: overflow in number field
# Command failed: python -m pythonforandroid.toolchain apk --debug --bootstrap=sdl2 --dist_name weather --name Weather --version 0.1 --package org.test.weather --android_api 19 --minsdk 9 --private /home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/app --permission INTERNET --orientation landscape --copy-libs --arch armeabi-v7a --color=always --storage-dir=/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build

And Python 3.5.2:

  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/python-for-android-master/pythonforandroid/toolchain.py", line 903, in <module>
    main()
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/python-for-android-master/pythonforandroid/toolchain.py", line 900, in main
    ToolchainCL()
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/python-for-android-master/pythonforandroid/toolchain.py", line 508, in __init__
    getattr(self, args.subparser_name.replace('-', '_'))(args)
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/python-for-android-master/pythonforandroid/toolchain.py", line 148, in wrapper_func
    func(self, args)
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/python-for-android-master/pythonforandroid/toolchain.py", line 744, in apk
    build_args = build.parse_args(args.unknown_args)
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build/dists/weather/build.py", line 543, in parse_args
    make_package(args)
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build/dists/weather/build.py", line 255, in make_package
    make_tar('assets/private.mp3', tar_dirs, args.ignore_path)
  File "/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build/dists/weather/build.py", line 201, in make_tar
    tf.add(fn, afn)
  File "/home/dorian.pula/.virtualenvs/doric/lib/python3.5/tarfile.py", line 1938, in add
    self.addfile(tarinfo, f)
  File "/home/dorian.pula/.virtualenvs/doric/lib/python3.5/tarfile.py", line 1960, in addfile
    buf = tarinfo.tobuf(self.format, self.encoding, self.errors)
  File "/home/dorian.pula/.virtualenvs/doric/lib/python3.5/tarfile.py", line 802, in tobuf
    return self.create_ustar_header(info, encoding, errors)
  File "/home/dorian.pula/.virtualenvs/doric/lib/python3.5/tarfile.py", line 821, in create_ustar_header
    return self._create_header(info, USTAR_FORMAT, encoding, errors)
  File "/home/dorian.pula/.virtualenvs/doric/lib/python3.5/tarfile.py", line 916, in _create_header
    itn(info.get("uid", 0), 8, format),
  File "/home/dorian.pula/.virtualenvs/doric/lib/python3.5/tarfile.py", line 211, in itn
    raise ValueError("overflow in number field")
ValueError: overflow in number field
# Command failed: python -m pythonforandroid.toolchain apk --bootstrap=sdl2 --dist_name weather --name Weather --version 0.1 --package org.test.weather --android_api 19 --minsdk 9 --private /home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/app --permission INTERNET --orientation landscape --copy-libs --arch armeabi-v7a --color=always --storage-dir=/home/dorian.pula/colonia/hackers-academy/python/kivy/weather_app/.buildozer/android/platform/build

@pts-dorianpula
Copy link

I did a bit of research and Android having busybox should handle GNU style tar files

I manually changed the generated build.py to use the GNU style format on line 181

- tf = tarfile.open(tfn, 'w:gz', format=tarfile.USTAR_FORMAT)
+ tf = tarfile.open(tfn, 'w:gz', format=tarfile.GNU_FORMAT)

and I was able to build and run the package using buildozer android_new debug deploy run.

@pts-dorianpula pts-dorianpula mentioned this issue Feb 24, 2017
Merged
@inclement
Copy link
Member

Should be fixed by #1015

@inclement inclement reopened this Mar 4, 2017
@inclement
Copy link
Member

inclement commented Mar 4, 2017 •
edited
Loading

Reopening as I found another issue with the above PR and had to revert it.

@Bastian82
Copy link

Looking forward solving this issue, as i suffered also becouse of it. Manual modification as suggested helped.

@obfusk obfusk mentioned this issue Apr 8, 2021
Merged
@pushpendre
Copy link

Just wanted to say that this issue still exists. I installed buildozer a few days ago.

The suggestion above
#1013 (comment)

for f in $(fgrep -l USTAR_FORMAT $(find . -name build.py )); do 
  sed -i.bkp 's#USTAR_FORMAT#GNU_FORMAT#g' $f; done

was really useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

build.py: also clean() tarfile directory entries obfusk/python-for-android
5 participants
@inclement @artwyman @Bastian82 @pts-dorianpula @pushpendre