AppImage is missing a library

[probonopd]

On Ubuntu 18.04: python3: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory. It needs to be bundled.

Reference:
AppImage/appimage.github.io#1903

[probonopd]

Maybe this would be a good candidate for https://github.com/niess/python-appimage#for-applications-developers @niess?

[niess]

The user install with a Python AppImage seems to work:

wget https://github.com/niess/python-appimage/releases/download/python3.8/python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage
chmod u+x  python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage
./python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage -m pip install --upgrade guiscrcpy --user
guiscrcpy

It starts the GUI and complains that adb and scrcpy are not found (I didn't install them indeed). Note however that this is not an isolated install. For example, the exe are installed in .local/bin whatever the Python version which can lead to troubles when using several versions of Python.

Installing to the extracted image seems to work at first. E.g as:

./python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage --appimage-extract
mv squashfs-root python3.8
./python3.8/AppRun -m pip install guiscrcpy

However when running the guiscrcpy script it does not start the GUI as previously. Instead I get the following error related to git:

./python3.8/usr/bin/guiscrcpy
ERROR:root:adb could not be found in any of the paths ['bin', '/usr/bin', '~/.local/bin', '~/bin', '/usr/local/bin']
ERROR:root:scrcpy could not be found in any of the paths ['bin', '/usr/bin', '~/.local/bin', '~/bin', '/usr/local/bin']
Traceback (most recent call last):
  File "./python3.8/usr/bin/guiscrcpy", line 6, in <module>
    from guiscrcpy.launcher import bootstrap
  File "guiscrcpy/launcher.py", line 78, in <module>
    v = version()
  File "guiscrcpy/lib/ver.py", line 35, in __init__
    if self.get_repo():
  File "guiscrcpy/lib/ver.py", line 68, in get_repo
    if not repo.git.describe("--tags").startswith('0.'):
  File "git/cmd.py", line 542, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "git/cmd.py", line 1005, in _call_process
    return self.execute(call, **exec_kwargs)
  File "git/cmd.py", line 822, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git describe --tags
  stderr: 'fatal: No names found, cannot describe anything.'

Maybe system install is not supported?

Also, it could make sense to package scrcpy and its deps as well in the AppImage?

[srevinsaju]

Thanks @niess for the quick response.

git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git describe --tags
  stderr: 'fatal: No names found, cannot describe anythin

This is because guiscrcpy gets the latest tags from .git folder, as the folder is absent, it can be just well to remove GitPython package.

---

I am not very sure how AppImage works in reality, for me its just a concept (still learning).
Is appimage isolated, like snaps and flatpak, from the rest of the system. If yes, I could ask users to install scrcpy from their packagers.

guiscrcpy's UI will start, if adb is found. It is not compulsory to have scrcpy to run guiscrcpy.

---

The link to adb by Google, is this, but I am not sure if I am allowed to package this with the AppImage. Is it?

[niess]

@srevinsaju Indeed after uninstalling GitPython as:

./python3.8/AppRun -m pip uninstall GitPython

it works :) At least the GUI starts and prompts for scrcpy location.

Yet, there is this warning message: WARNING:root:GitPython not installed. Fallback to pip3 version reading which might be a bit confusing for end users. In addition, users might already have GitPython installed in there user space (--user) in which case the startup will still fail.

Surprisingly I don't get this issue with the --user install. Maybe you bypass git in this case? Possibly there could be a command line option e.g. --with-git in order to run with GitPython and use the fallback otherwise. Or, if git is only used in order to get the version, then in your setup.py script you could generate a guiscrcpy/version.py file with the git version information when building your package. For example, I am doing this here.

The Python AppImages that I am producing can be considered as relocatable Python installs. There are standard except for this that there can be moved. So there are not strictly isolated. For example, they see the packages in the --user space. Basically, end users could simply get one of those AppImages and then pip install guiscrcpy with it. It should be equivalent to manually installing Python to a custom location and then pip installing your package.

Concerning the legalities related to packaging adb and scrcpy in your AppImage, I have to say that I am not quite sure as well. @probonopd ? Is it enough to add their specific licences in the AppImage?

[srevinsaju]

@niess, Thanks for confirming this.

./python3.8/AppRun -m pip uninstall GitPython

I have pushed a commit to master regarding this on a8da371 .

Surprisingly I don't get this issue with the --user install. Maybe you bypass git in this case? Possibly there could be a command line option e.g. --with-git in order to run with GitPython and use the fallback otherwise.

Interesting. Thanks. I have not yet tried a system-wide installation of guiscrcpy, as all of my mentors warn me not to use pip with sudo. Maybe, it might be fixed by a8da371.

The Python AppImages that I am producing can be considered as relocatable Python installs

I did not understand what relocatable is.
Does that mean, thst I will not be able to access /usr/bin. As scrcpy binaries are not yet released, should we actually build it from source?

[niess]

@srevinsaju @probonopd I put a preliminary version of the guiscrcpy AppImage over there. Is this OK for you?

It was simply generated by extracting an existing Python 3.8 AppImage and pip installing to it. Note that I did not update the meta data for now, e.g. the desktop file etc.

[srevinsaju]

Thanks @niess. I will check after 8hrs; its quite late in my part of the world 😇. I will let you know when I fix the rest of the GitPython bugs, permanently.

[srevinsaju]

@niess I have tested the AppImage. The GUI starts, but I am not able to see my devices connected on adb. And also, adb is found by guiscrcpy. Is it that, the adb AppImage will not be able to access the devices connected to usb ports / LAN.?

[niess]

@srevinsaju I did not understand. Is adb found or not? Note also that adb is not bundled in this AppImage. It only contains Python 3.8 and guiscrcpy with its Python deps, namely PyQt5.

There is no isolation in this AppImage. It mounts to /tmp at startup. So it sees your native system and reciprocally the native system sees it. In a temporary location though.

[srevinsaju]

yea, ADB is found in the AppImage. but my devices cannot be detected by the adb. I am not sure why. If I extract the AppImage, I am able to detect the adb. @niess, are you able to mirror your mobile screen using guiscrcpy?

[srevinsaju]

@niess , I got a new error, I were trying to debug why the AppImage fails to do screen mirroring

opt/python3.8/bin/guiscrcpy: bad interpreter: /home/niess/Soft/grand/python-appimage/python3.8/usr/bin/../../opt/python3.8/bin/python3.8: no such file or directory

[niess]

@srevinsaju There was a bug in python-appimage resulting in the shebang scripts of upgraded packages not being properly pruned. It should be solved now. I updated the guiscrcpy AppImage as well, still over there.

Yet this will not solve your issues related to adb. Does adb work with the extracted image?

[niess]

@srevinsaju I finally managed to install adb a scrcpy. I had to enable testing repositories that for. I am using Debian 9. On my side the updated AppImage of guiscrcpy seems to work fine. I can control my Android tablet from the PC. Its great by the way :)

Does it still fail for you? What Linux do you use?

[srevinsaju]

Thanks @niess. I had tested the first link to the AppImage and not the second link. My linux distribution is Arch Linux. I will check out the second AppImage once again tomorrow, (after 8hrs). I tested the first AppImage twice, and it did not work as expected, meaning, the GUI starts, but not the scrcpy's window.

[niess]

Fine. I didn't change anything except the shebang of guiscrcpy. So you will likely have the same problems :(

Good Night! :)

[srevinsaju]

@niess , I have found the error. The AppImage works under a certain condition. This might be a bug in your AppImage making script, but I am not sure why exactly it happens.

• If the version of guiscrcpy is not installed / is of different version, the AppImage crashes.
  I am working on 3.7 right now. The AppImage you have built is of 3.5. As I test 3.7, I have an installation of 3.7 on by ~/.local/ level folder. In that case, guiscrcpy GUI works, but I cannot work with adb, scrcpy or anything related to hardware.
• If the version of guiscrcpy corresponds to the current guiscrcpy version, the adb connections work, and I am able to do screen mirroring. Perfectly.

I am still debugging on what actually is happening, maybe it is the adb which causes the problem.
guiscrcpy searches the /usr/bin, /usr/local/bin for adb. In the case its not found, it asks for the adb. In this case, in the AppImage, should we add the adb to it, because, that might be the cause of the error.

Thanks for the AppImage and your work for scrcpy. If you are interested, you might like to join https://discord.gg/cc6exT

[niess]

@srevinsaju Python packages installed with --user (i.e. under ~/.local on LInux) have higher priority over system ones. This is the usual Python behaviour. It implies that when your guiscrcpy script imports the guiscrcpy package it loads the user one, not the AppImage one. In the case of a Python based app this migt be problematic indeed. I didn't realize this so far because I have nothing installed under --user for Python 3.8.

According to the CLI help you can prevent this by adding the -s option when calling Python. For example, if you extract the guiscrcpy AppImage you can edit the AppRun file and change:

"${APPDIR}/usr/bin/python3.8" "${APPDIR}/opt/python3.8/bin/guiscrcpy" "$@"

to

"${APPDIR}/usr/bin/python3.8" -s "${APPDIR}/opt/python3.8/bin/guiscrcpy" "$@"

Then you can rebuild the image with appimagetool. I updated the guiscrcpy AppImage with this fix if you'd rather download it.

On my side its working, but I have nothings installed under --user.

P.S: Sorry for the inconvenience and thank your for beta testing :P

[srevinsaju]

Thanks @niess .
Another user came in trying to run guiscrcpy on Linux Mint, with no supported packages for their system, and they were benefited by this AppImage, Another user have tested it on Linux Mint and it works out of the box. Thanks a lot.

What next can I do for this? like testing or something.

P.S: Sorry for the inconvenience and thank your for beta testing :P

Its absolutely fine. It was my error, which caused the problem

[srevinsaju]

@niess Thanks a lot. I have added the -s flag, and now it works perfectly.

@probonopd @niess how to proceed, like updating metadata and related items?
@niess, will you be maintaining guiscrcpy.AppImage?

[niess]

Great :)

I think its better if you do it yourself, if that's OK with you? Concerning the metadata, the logo you'd like to use etc. And so you can publish it directly in the releases of your GitHub repo.

The linux-appimage package allows to automate the generation of the AppImage. You can get it from PyPI for example or from GitHub. You just need to create a directory with a requirements file, an entrypoint script and the app metadata. You can copy one of the examples over there. Maybe the tasmotizer one?

For the requirements.txt you only need to put guiscrcpy. In the entrypoint.sh you need to add the -s option. I will update the examples with this extra flag. The other files are the metadata. When you are done you can run:

python-appimage build app path/to/your/app/meta/folder

You could add those meta to your GitHub repo for example and add a dedicated workflow. You can find example of workflows for building & releasing the AppImages here, e.g. applications.yml and appimage.yml.

[srevinsaju]

Ok, thanks a lot. I appreciate your work on guiscrcpy.AppImage to let me get started on it.
Also glad to know that you liked guiscrcpy.

[srevinsaju]

@niess, thanks a lot for the python-appimage tool.
I have packaged guiscrcpy adding adb and scrcpy pre-packaged.

The AppImage has been created. It works on my system. I will keep testing on other VMs too.

Thanks a lot again.

@probonopd should I update the PR on @appimagehub?

[niess]

@srevinsaju Great that you managed to set this AppImage :)

I just tried the AppImage without adb and scrcpy installed on my system. It doesn't find adb because it expects it under /usr/bin whereas it is located under $APPDIR/usr/bin where $APPDIR is a temporary location set at runtime by the AppImage. Is there a way to tell guiscrcpy where to look for binaries? E.g. an env var? Or CLI option? Then you could adapt the AppRun (entrypoint.sh) exporting this env var before calling Python or adding the CLI option to the call.

Also I had a look at the deps of adb and guiscrcpy. While adb is pretty standalone, scrcpy depends on libSDL2 among others, which is not packaged in the AppImage. Packaging binaries requires a bit more work than Python packages because you need to package most shared libs as well and to modify the binaries & libs $RUNPATH. This can be done with linuxeploy. For example:

./guiscrcpy-x86_64.AppImage --appimage-extract
./linuxdeploy-x86_64.AppImage --appdir=squashfs-root

Note that you need to have all shared libs available on your system when running linuxdeploy. In addition, for best compatibility this should be run on an old enough Linux (e.g. CentOS 6). I am using the manylinux docker images for this. For example as here.

This is a bit technical. You might also not package adb and scrcpy in your AppImage for now? Then the end user would need to install them himself as when doing pip install guiscrcpy.

[srevinsaju]

Sure, I am willing to do the hard work once, before the upcoming release of 3.7 (#76). Thanks for the information on adb and scrcpy. I had not known about that, as I tested it on my own system. My Arch Linux distro have frequent crashes when running python_appimage tool. So I am now depending on ubuntu:18.04 docker images to build AppImages, as I feel those are pretty old and can fix the deps.

Regarding scrcpy, I might try to package scrcpy with its binaries, I will try, If I succeed with packaging scrcpy using linuxdeploy, I might release 3.7 with the perfect appimage. I wanted to package scrcpy too, because installing scrcpy on systems without a prebuilt package is a mess, and people just don't want to waste their time building new stuff. I will try to do this as what you have told.
Thanks a lot

[srevinsaju]

@niess I have added the support for detecting adb as you have mentioned, but the config file problem is still unsolved. Should I write another config file in another folder called ~/.config/guiscrcpy.appimage/* or should I do some other changes? I suppose the ~/.config/guiscrcpy.appimage/* will remain unchanged, and can be manually modified by developers if they would like to use a different version of scrcpy / adb

[srevinsaju]

This is the latest, https://github.com/srevinsaju/guiscrcpy/suites/613234079/artifacts/4717513

[niess]

If you don't expect users to modify this config file then it sounds like an easy fix yes

[srevinsaju]

@niess Alternatively, I can prevent reading the path, to adb and scrcpy if I create another environment variable GUISCRCPY_APPIMAGE = True, then to respect environment vars and not to update the config. That would be a solution too. I am not sure which is the best

[srevinsaju]

@niess, Seems like I had already implemented that too in the previous link. I don't remember what I do 🤦 .

The previous #56 (comment) is the almost final version of 3.7 guiscrcpy

[niess]

I tested your AppImage. It works great for me :) I could run its several times without any issue.
I will use it :P
I am a stargazer now 👍

[srevinsaju]

Thanks for testing 😃

[srevinsaju]

@niess There is a problem now:
The CI on appimage.github.io is failing for some reason

AppImage/appimage.github.io#1967

[niess]

@srevinsaju I don't understand what's going on neither. It seems to be blocked at some stage but I cant see any report? @probonopd ?

[srevinsaju]

@niess, you can see the report on the Travis CI (Checks), it fails at checking some ENV variable $FILES, but your python-3.8.AppImage succeeds the test due to some reason.

[niess]

I see this error:

/tmp/.mount_BeingTbXdxh7/guiscrcpy.desktop: error: value "Python;Screen;Mirroring;Android" for locale string list key "Keywords" in group "Desktop Entry" does not have a semicolon (';') as trailing character

I thought that you already patched it? Because the following tests were successful. If not, apparently you need to add a semicolon in your desktop file after the last item in Keywords, over here, i.e. Android; instead of Android.

[srevinsaju]

Thanks a lot, it was my dumb mistake. I will update the AppImage and let you know if the CI passes then.

[srevinsaju]

@niess, It still fails, I edited the desktop file

https://travis-ci.org/github/AppImage/appimage.github.io/builds/678669680#L612

[niess]

OK, now the reason is that libxkbcommon-x11.so.0 is missing. This library is needed by PyQt5 but not shipped with the PyQt5 wheel. On some Linuses -as the one used for AppImage validations- it seems to be missing. @probonopd it's not clear to me if it should be packaged in PyQt5 AppImages/wheel (as a patch) or in the Docker image that does the validation (I.e. assumed to be available)?

Anyway, @srevinsaju if you manage to grab an old enough version of this library (w.r.t. the AppImage validation Linux) you could simply drop it in your AppImage under site-packages/PyQt5/PyQt5/Qt/lib. The libs over there have RUNPATH=$ORIGIN so they will locate any lib in the same folder.

[probonopd]

@probonopd it's not clear to me if it should be packaged in PyQt5 AppImages/wheel (as a patch) or in the Docker image that does the validation (I.e. assumed to be available)?

Ideally the one from the Docker container would be copied to the AppImage.

[srevinsaju]

@niess @probonopd I have updated the CI to patch PyQt5 with the lib. Thanks @niess for the quick fix.

@probonopd, I guess what @niess meant is that, the travis should probably have that file already on the testing system, without adding it to the AppImage. At most times, it might be present in the users system

[probonopd]

without adding it to the AppImage. At most times, it might be present in the users system

I can say for sure that e.g., Xubuntu does not have libxkbcommon-x11.so.0, so for the AppImage to work "out of the box" the library needs to be bundled privately inside the AppImage. Ideally the one from the Docker container.

[niess]

@probonopd OK, then it would make sense to include it in the PyQt5 wheel directly. It wrote an email to the PyQt5 mailing list about this.

[srevinsaju]

@niess unfortunately, the CI's still failing. I guess I messed up this AppImage thingy. Should I depend on other packing systems like snap or flatpak

[niess]

@srevinsaju I can understand your frustration. Note however that your AppImage works as so on many LInux. People that would pip install the wheel would have the same issues. Even more actually since you also packaged adb and scrcpy in your AppImage.

Do you still have the same error related to libxkbcommon-x11.so.0?

[niess]

@srevinsaju I just found the CI. To me It looks like it doesn't even start because it doesn't even look for your AppImage? I don't know why. Actually, I never did a PR to appimage.io :P

Maybe you could start again from scratch with a new PR?

[srevinsaju]

Ok @niess, Did a re PR, and now it doesn't find another lib (PyQt5).
As, the main issue is fixed, I guess I can close this PR. I guess, I have already put a lot of time into AppImage thingy, but maybe, I should work on it later. I happy to know it works on many systems, as far as the last LTS is cocerned (maybe no xubuntu). I will continue, maybe later.

Thanks again.

[niess]

@srevinsaju I understand. That's quite a work at the end indeed :( Sorry that it didn't go more fluently. I am doing this as a side project. Actually my main motivation was distributing Python to my co-workers for which the python-appimage works great as so.

I think that the PR is still failing because of libxkbcommon-x11.so.0. Probably that the the one that you packaged is not compatible with the Linux used by the AppImage CI. I'll try to add a specific patch to python-appimage for PyQt5. Automatically adding an AppImage compatible libxkbcommon-x11.so.0 if PyQt5 is pip installed. I'll let you know if/when I manage to do so.

[srevinsaju]

Yes @niess, guiscrcpy is also a side project, but I might need to start working on more features for guiscrcpy. I need to start working on reducing the latency of button touch and device actions. Currently it takes 0.8-1.0 seconds approx, and I would like to reduce it to less than 0.1s

How could you understand that the error is still caused by libxbcommon-x11.so.1. that's cool skills to deduce that? Is it there in the Travis logs? I can find some other missing libs as messages other than the above one.

Thanks a lot for your help

[srevinsaju]

@niess Does appimages work on Chrome OS? #103

[niess]

@srevinsaju I understood that this is solved? PyQt5 wheel was not compatible with the user's system / ChromeOS?

Concerning the travis log for guiscrcpy I looked at the latest (closed) PR. There is a link to the Travis log leading to there. If that's indeed the latest trial?

[srevinsaju]

@niess At last fixed the Travis on AppImage/appimage.github.io#1980 ; The libs had to be placed in usr/lib for some reason, and it worked.

[niess]

@srevinsaju Great :) Thank you for the information. Though I don't understand why but nevermind :P