Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update deps #24

Merged
merged 13 commits into from
Oct 4, 2023
Merged

Update deps #24

merged 13 commits into from
Oct 4, 2023

Conversation

auouymous
Copy link
Member

@elelay Do you know how to fix this issue on line 10 in bundle.sh?

Warning, library not available in any prefix: /Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_home/.new_local/share/pyenv/versions/3.10.2/lib/libpython3.10.dylib
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm_compat.4.dylib
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm.6.dylib
Traceback (most recent call last):
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm_compat.4.dylib
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm.6.dylib
Warning, library not available in any prefix: /Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_home/.new_local/share/pyenv/versions/3.10.2/lib/libpython3.10.dylib
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm_compat.4.dylib
  File "/Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_home/.new_local/bin/gtk-mac-bundler", line 6, in <module>
    bundler.main.main(sys.argv[1:])
  File "/Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_bundler/bundler/main.py", line 18, in main
    bundler.run()
  File "/Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_bundler/bundler/bundler.py", line 494, in run
    path.copy_target(self.project)
  File "/Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_bundler/bundler/project.py", line 194, in copy_target
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm.6.dylib
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm_compat.4.dylib
Warning, library not available in any prefix: /usr/local/opt/gdbm/lib/libgdbm.6.dylib
    source = self.compute_source_path(project)
  File "/Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_bundler/bundler/project.py", line 187, in compute_source_path
    raise ValueError("Cannot find source to copy: " + source)
ValueError: Cannot find source to copy: /Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_home/gtk/inst/lib/python3.10/site-packages/pip

@elelay
Copy link
Member

elelay commented Jan 6, 2023

Seems like pip from pyenv gets in the way during the install of python3 in the destination path

2023-01-05T12:06:35.9641110Z Requirement already satisfied: pip in /Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_home/.new_local/lib/python3.10/site-packages (22.3.1)

Then it's not found when packaging.

@elelay
Copy link
Member

elelay commented Jan 6, 2023

Do you know why modulesets/upstream-ref is not updated?

@auouymous
Copy link
Member Author

Do you know why modulesets/upstream-ref is not updated?

Because I manually updated each moduleset file, then I saw upstream-ref and found fetch_modules.sh. It wasn't a waste of my time because gtk-osx.modules has -DCMAKE_DISABLE_FIND_PACKAGE_BrotliDec=TRUE on two packages so they compile on github, and there are a couple stray xml closing tags in that upstream file.

--- a/modulesets-stable/gtk-osx.modules
+++ b/modulesets-stable/gtk-osx.modules
@@ -184,7 +184,7 @@
 
   <meson id="harfbuzz" mesonargs="-Dcoretext=enabled -Dfreetype=enabled -Ddocs=disabled -Dbenchmark=disabled -Dintrospection=enabled">
     <branch repo="github-tarball" version="4.1.0"
-            module="harfbuzz/harfbuzz/releases/download/4.1.0/harfbuzz-4.1.0.tar.xz">
+            module="harfbuzz/harfbuzz/releases/download/4.1.0/harfbuzz-4.1.0.tar.xz"
             hash="sha256:f7984ff4241d4d135f318a93aa902d910a170a8265b7eaf93b5d9a504eed40c8">
     </branch>
     <dependencies>
@@ -285,7 +285,7 @@
 
   <meson id="graphene" mesonargs="-Dtests=false">
     <branch repo="github-tarball" version="1.10.6"
-            module="ebassi/graphene/releases/download/1.10.6/graphene-1.10.6.tar.xz"/>
+            module="ebassi/graphene/releases/download/1.10.6/graphene-1.10.6.tar.xz"
             hash="sha256:80ae57723e4608e6875626a88aaa6f56dd25df75024bd16e9d77e718c3560b25"/>
     <dependencies>
       <dep package="glib"/>
@@ -338,7 +338,7 @@
       <dep package="freetype"/>
     </dependencies>
     <after>
-<!-- Circular dependency if we depend on meta-gtk-osx-gtk3 -->
+          <!-- Circular dependency if we depend on meta-gtk-osx-gtk3 -->
       <dep package="gtk+-3.0"/>
       <dep package="gtk-4"/>
     </after>

@eunikolsky
Copy link
Contributor

@auouymous, re gpodder/gpodder#1348 (comment), I have the pip error when building this branch locally.

@eunikolsky
Copy link
Contributor

I tried ./fetch_modules.sh and building from that. There is a build error:

…
*** Configuring harfbuzz-no-cairo *** [19/39]
meson --prefix /Users/u/proj/py/gpodder-osx-bundle/_home/gtk/inst --libdir lib -Dcoretext=enabled -Dfreetype=enabled -Ddocs=disabled -Dbenchmark=disabled -Dintrospection=disabled --wrap-mode=nofallback /Users/u/proj/py/gpodder-osx-bundle/_home/gtk/source/harfbuzz-4.1.0
The Meson build system
…
Run-time dependency freetype2 found: NO (tried pkgconfig, framework and cmake)
Not looking for a fallback subproject for the dependency freetype2 because:
Use of fallback dependencies is disabled.

../../../../gtk/source/harfbuzz-4.1.0/meson.build:87:0: ERROR: Dependency 'freetype2' is required but not found.
…

Even though freetype-no-harfbuzz was built before that and _home/gtk/inst/lib/libfreetype.dylib exists. There is some weirdness with circular dependencies: https://gitlab.gnome.org/GNOME/gtk-osx/-/blob/master/modulesets-stable/gtk-osx.modules#L124-129, https://github.com/harfbuzz/harfbuzz/blob/main/meson.build#L175-L178.

This is so complicated.

@eunikolsky
Copy link
Contributor

In the meson build log, I found the more specific error message:

Package libbrotlidec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libbrotlidec.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libbrotlidec', required by 'freetype2', not found

Looking back in the logs, I stumbled upon this:

*** Configuring freetype-no-harfbuzz *** [17/39]
…
-- Found BrotliDec: /usr/local/include
…
*** Building freetype-no-harfbuzz *** [17/39]
ninja
[42/43] Linking C shared library libfreetype.6.18.1.dylib
ld: warning: dylib (/usr/local/lib/libbrotlidec.dylib) was built for newer macOS version (12.0) than being linked (10.9)
ld: warning: dylib (/usr/local/lib/libbrotlidec.dylib) was built for newer macOS version (12.0) than being linked (10.9)
[43/43] Creating library symlink libfreetype.6.dylib libfreetype.dylib

Yes, it found libbrotlidec.dylib installed by brew. /facepalm

@auouymous is correct in #24 about disabling the brotli dependency. Related links: https://gitlab.gnome.org/GNOME/gtk-osx/-/issues/62, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964185.

@eunikolsky
Copy link
Contributor

So I ended up with the same pip error as in this PR. If I comment copying that directory and then rsync _build/pythonbase.app/Contents/Resources/{lib,share} into the released gPodder 3.11.1 package (to update the shared libraries), the window has the correct size, but the fonts are blurry. Basically, the same result as in gpodder/gpodder#1348 (comment). I have no idea how to fix that.
Can we release gPodder 3.11.1 using the application package from gPodder 3.10.21? Maybe that would fix the UI bugs?

@auouymous
Copy link
Member Author

Thank you for those links, I will try removing my brotli patch and see if github builds. It probably won't fix the pip issue but will remove the need for the patch.

As for rolling back to the 3.10.21 bundle, changes were made for yt-dlp to work, and there is a change to fix a syncing issue. I don't know if the 3.10.21 bundle will build or not, but the last working commit used for 3.11.0 no longer builds. Even if the 3.10.21 bundle builds or can easily be made to build, it would require a gtk-osx update to make yt-dlp work.

@auouymous
Copy link
Member Author

Seems that disabling brotli is still required. And latest gtk-osx fails to build libxml2, so no magic fix for the pip issue yet.

@eunikolsky
Copy link
Contributor

As for rolling back to the 3.10.21 bundle, changes were made for yt-dlp to work, and there is a change to fix a syncing issue. I don't know if the 3.10.21 bundle will build or not, but the last working commit used for 3.11.0 no longer builds. Even if the 3.10.21 bundle builds or can easily be made to build, it would require a gtk-osx update to make yt-dlp work.

@auouymous, this makes sense in general, some updated packages may be required. However why would yt-dlp require gtk-osx?! In any case, in my opinion, the first priority is to make sure the basic functionality of gPodder works out of the box (correct window size and non-blurry text), and the second priority is to make sure the bundled extensions work (not everyone needs yt-dlp).

For reference, my update branch: https://github.com/eunikolsky/gpodder-osx-bundle/tree/latest_moduleset.

@auouymous
Copy link
Member Author

I didn't add support for yt-dlp in the bundle, I just remember the bundle being updated to make yt-dlp work on Mac. Also keep in mind that not all Mac users have the hidpi issue, and they might not agree with removing the yt-dlp support they depend on.

As for https://github.com/eunikolsky/gpodder-osx-bundle/tree/latest_moduleset, I don't know why it would fail to download python with curl, but maybe it is because you used fetch_modules.py to download 2023-02-14 (https://github.com/eunikolsky/gpodder-osx-bundle/blob/latest_moduleset/modulesets/upstream-ref) but https://github.com/eunikolsky/gpodder-osx-bundle/blob/latest_moduleset/bootstrap.sh#L18 is still pinned to 2023-01-03. I'm rebuilding this PR with your patch to ignore pip...

One of my recent commits in this PR pinned the repo in fetch_modules.py and added a comment to sync with the pin in bootstrap.py. Once this is working it might be a good idea to add a file sourced by both that holds the hash for gtk-osx and bundler, and mention that file in the readme.

@auouymous
Copy link
Member Author

It finally built!

@elelay Does this work? I had to remove the Tango theme, is that an issue?

If this works, I will try putting pip back in, clean up the patches and get a gpodder build for testing.

@elelay
Copy link
Member

elelay commented Jul 9, 2023

Hi,
sorry, I took too long to get the artifacts and they seem to be gone. I've triggered a re-run and will try to test it.

@elelay
Copy link
Member

elelay commented Jul 12, 2023

@auouymous the re module is not available!
In launcher.py import platform fails because of that
Even after working around this in launcher.py, still re is not available when running run-python

@elelay
Copy link
Member

elelay commented Jul 12, 2023

This is because re is now implemented as a folder with init.py, etc. vs re.py in previous version.
It should be added in https://github.com/auouymous/gpodder-osx-bundle/blob/update-deps/misc/bundle/pythonbase.bundle#L121

Here are build logs grepped using grep -P 'Creating .+gtk/inst/lib/python3.11/[^/]+$' for lib/python3.11 subdirectories:

  • - for directories explicitely removed in pythonbase.bundle
  • MISSING for directories not listed in pythonbase.bundle, re, tomllib, zoneinfo should be added, maybe also xmlrpc (412kB)
  • distutils, ensurepip, venv I'm not sure. They should certainly be explicitly marked ignored
  • __phello__ I don't know. It's to check that frozen modules work if I understand correctly
lib/python3.11/asyncio
lib/python3.11/collections
lib/python3.11/concurrent
lib/python3.11/config-3.11-darwin
lib/python3.11/ctypes
lib/python3.11/curses
- lib/python3.11/dbm
MISSING lib/python3.11/distutils
lib/python3.11/email
lib/python3.11/encodings
MISSING lib/python3.11/ensurepip
lib/python3.11/html
lib/python3.11/http
- lib/python3.11/idlelib
lib/python3.11/importlib
lib/python3.11/json
- lib/python3.11/lib2to3
lib/python3.11/logging
lib/python3.11/multiprocessing
- lib/python3.11/pydoc_data
MISSING lib/python3.11/re
lib/python3.11/sqlite3
lib/python3.11/test
-lib/python3.11/tkinter
MISSING lib/python3.11/tomllib
-lib/python3.11/turtledemo
lib/python3.11/unittest
lib/python3.11/urllib
- lib/python3.11/venv
lib/python3.11/wsgiref
lib/python3.11/xml
MISSING lib/python3.11/xmlrpc
MISSING lib/python3.11/zoneinfo
MISSING lib/python3.11/__phello__

@auouymous
Copy link
Member Author

I added the missing modules, can you test again when it finishes building?

@elelay
Copy link
Member

elelay commented Jul 23, 2023

It works but pip3 is missing

@auouymous
Copy link
Member Author

@elelay Do you know how to make it install pipenv 2022.11.4 or newer. I don't know if it would fix the failure, but the runtime instrospection warnings are caused by an older version that doesn't fully support python 3.11.

@elelay
Copy link
Member

elelay commented Jul 26, 2023

@elelay
Copy link
Member

elelay commented Jul 26, 2023

nice try :-)
I'm rebuilding locally on Catalina to see if I can spot the issue.
Don't expect results until Tomorrow night

@elelay elelay closed this Jul 26, 2023
@elelay elelay reopened this Jul 26, 2023
@elelay
Copy link
Member

elelay commented Jul 27, 2023

Yeah, so when python3 install bootstraps pip by calling python3 -E -m ensurepip --root xxxx it finds the one in _home/.new_local and says nothing is to be done.
the environment variable PYTHONUSERBASE defined by jhbuild is at fault.

@elelay
Copy link
Member

elelay commented Jul 27, 2023

Can you create runit.sh with following content and executable?

#!/bin/sh
unset PYTHONUSERBASE
python3 -E -m ensurepip --root /Users/jojolapin/gpodder-osx-bundle/_home/gtk/inst --default-pip

then call jhbuild run ./runit.sh at the end of build.sh? In my install it successfully installed pip in gtk/inst/bin

@auouymous
Copy link
Member Author

build.sh is failing, so runit.sh is not called.

@elelay
Copy link
Member

elelay commented Jul 27, 2023

maybe rollback 59ebd8d?
It caused the failed build.sh because meson is not installed/found, right?

@elelay
Copy link
Member

elelay commented Jul 27, 2023

My mistake: the --root .... argument should not be there. See

WARNING: The scripts pip, pip3 and pip3.11 are installed in '/Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_home/gtk/inst/Users/runner/work/gpodder-osx-bundle/gpodder-osx-bundle/_home/gtk/inst/bin' which is not on PATH.

in the logs.

Please replace in runit.sh with:

python3 -E -m ensurepip --default-pip

@auouymous
Copy link
Member Author

That worked!

@auouymous
Copy link
Member Author

@elelay Does this build work any better?

@elelay
Copy link
Member

elelay commented Oct 2, 2023

I need to rebuild to access artifacts. If this build doesn't work, do you mind pushing to a branch in gpodder-osx-bundle ? Maybe then I access your artifacts?

@auouymous
Copy link
Member Author

I added you to my fork, hopefully that gives you access to future artifacts there.

@elelay
Copy link
Member

elelay commented Oct 3, 2023

YES!
Finally it works 🥳

@elelay
Copy link
Member

elelay commented Oct 3, 2023

I was able to create a package from gpodder, I just had to s/3.9/3.11/ in release_on_macosx.sh .
So I'm ready to merge and tag a new version of gpodder-osx-bundle shortly thereafter.

@auouymous
Copy link
Member Author

I'll clean up the commits and try to get a gpodder build out today for others to test! 🎆

@auouymous auouymous marked this pull request as ready for review October 4, 2023 00:53
@auouymous auouymous merged commit 0aedaf1 into gpodder:master Oct 4, 2023
1 check failed
@auouymous auouymous deleted the update-deps branch October 4, 2023 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants