Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gnuradio: rewrite #99685

Merged
merged 7 commits into from
Dec 5, 2020
Merged

gnuradio: rewrite #99685

merged 7 commits into from
Dec 5, 2020

Conversation

doronbehar
Copy link
Contributor

@doronbehar doronbehar commented Oct 6, 2020

Motivation for this change

Address some issues in #82263 .

Close #102116 .

Things done

It's in the commit messages and the comments. All affected packages should build.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@wirew0rm
Copy link
Contributor

wirew0rm commented Oct 6, 2020

Hey, first of all, thanks for all your work on this, gnuradio is a real beast with all its reverse dependencies and wrappers.

I tried to test your branch, checking out your branch and doing nix-build -A gnuradio and the build runs through. However I cannot get gnuradio companion to work:

$ git log --oneline -n 1
8b984f5c78d (HEAD -> pkg/gnuradio, doronbehar/pkg/gnuradio) treewide/(gnuradio plugins): Add log4cpp as it's needed now. 

$ nix-build -A gnuradio
[...]

$ ls -l result
lrwxrwxrwx 1 akrimm users 68  6. Okt 10:13 result -> /nix/store/pysmpxcsbrb49shlqdm0qlhgknwzyh4c-gnuradio-wrapped-3.8.2.0

$ result/bin/gnuradio-companion 
Traceback (most recent call last):
  File "/nix/store/pysmpxcsbrb49shlqdm0qlhgknwzyh4c-gnuradio-wrapped-3.8.2.0/bin/.gnuradio-companion-wrapped", line 59, in check_gtk
    gi.require_version('Gtk', '3.0')
  File "/nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gi/__init__.py", line 129, in require_version
    raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gtk not available

During handling of the above exception, another exception occurred:
``` bash
Traceback (most recent call last):
  File "/nix/store/pysmpxcsbrb49shlqdm0qlhgknwzyh4c-gnuradio-wrapped-3.8.2.0/bin/.gnuradio-companion-wrapped", line 100, in <module>
    check_gtk()
  File "/nix/store/pysmpxcsbrb49shlqdm0qlhgknwzyh4c-gnuradio-wrapped-3.8.2.0/bin/.gnuradio-companion-wrapped", line 67, in check_gtk
    die(err, "Failed to initialize GTK. If you are running over ssh, "
  File "/nix/store/pysmpxcsbrb49shlqdm0qlhgknwzyh4c-gnuradio-wrapped-3.8.2.0/bin/.gnuradio-companion-wrapped", line 41, in die
    gi.require_version('Gtk', '3.0')
  File "/nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gi/__init__.py", line 129, in require_version
    raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gtk not available

From your comment in the meta issue i gathered that the wrapper is working for you. Any idea what I'm doing different?

@doronbehar
Copy link
Contributor Author

doronbehar commented Oct 6, 2020

Hey and thanks for testing. It's all in the environment.

Any idea what I'm doing different?

It's not your fault, it's due to a known impurity of Nixpkgs' & the OS' design. I use Gnome and by default my environment is more polluted. What I pushed now should (I hope) fix the issue for you - I tested gnuradio-companion to start with env -u GI_TYPELIB_PATH.

@doronbehar
Copy link
Contributor Author

Oh and don't worry these changes won't trigger a rebuild of it all :) - just the wrapper :). [That's the beauty of an external wrapper].

@wirew0rm
Copy link
Contributor

wirew0rm commented Oct 6, 2020

Ok, thanks for the fast reply and the fix.

There was one more 'impurity' preventing grc from launching, since 3.8.2 gnuradio expects gsettings to be on the path to detect the dark theme, had to add nixpkgs.glib to my installed packages. This post on the debian mailing list suggests to add the patch from gnuradio master to make this functionality optional (it only detects if it should use dark theme).

I'm also missing all the icons, but I have that with other programs, too, I think thats a problem of my session configuration.

But I have a running gnuradio-companion.
Next i tried to run the preconfigured sample, but this only works if I change the generate option from QT GUI to No GUI.

With with qtgui enabled, I get the following stacktrace

Executing: /nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/bin/python3.8 -u /home/wirew0rm/gnuradiotests/top_block.py

Warning: failed to XInitThreads()
Traceback (most recent call last):
  File "/nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gnuradio/qtgui/qtgui_swig.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1101, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /nix/store/0n3h96is5klddhnin5kxf7bfxqx6ard3-qtbase-5.14.2/lib/libQt5Core.so.5: version `Qt_5.15' not found (required by /nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gnuradio/qtgui/_qtgui_swig.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gnuradio/qtgui/__init__.py", line 32, in <module>
    from .qtgui_swig import *
  File "/nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gnuradio/qtgui/qtgui_swig.py", line 17, in <module>
    _qtgui_swig = swig_import_helper()
  File "/nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gnuradio/qtgui/qtgui_swig.py", line 16, in swig_import_helper
    return importlib.import_module('_qtgui_swig')
  File "/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_qtgui_swig'

This is probably a similar problem as the first problem but this time for QT instead of gtk, right?
I'll try to look into it some more later, but I wanted to report what worked for me and what not.

@doronbehar
Copy link
Contributor Author

3.8.2 gnuradio expects gsettings to be on the path to detect the dark theme, had to add nixpkgs.glib to my installed packages

Once again thanks for the tests. Your help is valuable. We could add glib to the PATH of the wrapper.

I'm also missing all the icons, but I have that with other programs, too, I think thats a problem of my session configuration.

What Window manager are you using?

With with qtgui enabled, I get the following stacktrace

Interesting stacktrace. It seems as if Gnuradio's qt-gui feature is not compatible with qt5.14, but expects 5.15. The build of this feature is used with qt5 which points as of today to qt515 according to this:

https://github.com/doronbehar/nixpkgs/blob/7678dd675453a19674d2f20f1c748acd7c035fdd/pkgs/top-level/all-packages.nix#L15003-L15004

But, the final derivation doesn't even retain references to qtbase for instance. Hence I conclude that it's an environment issue.

  1. Could you tell how do you attempt at running that preconfigured sample please?
  2. What's the output of env | grep QT_PLUGIN_PATH on your machine when you run the companion?

I've updated once more the wrapper to set QT_PLUGIN_PATH and QML2_IMPORT_PATH as well. Could you please recheck now @wirew0rm ?

@wirew0rm
Copy link
Contributor

wirew0rm commented Oct 7, 2020

I'm also missing all the icons, but I have that with other programs, too, I think thats a problem of my session configuration.

What Window manager are you using?

i3wm, but I've just confirmed that I have Icons on a different machine, so I guess it's related to my configuration.

But, the final derivation doesn't even retain references to qtbase for instance. Hence I conclude that it's an environment issue.

1. Could you tell how do you attempt at running that preconfigured sample please?

I launch ./result/bin/gnuradio-companion from the command line, then there was a flowchart with a cosine signal source connected to an audio sink (not sure if that was from previous experimentation or if it is the default demo flowgraph) and then I press F6 to generate and launch the flowgraph.
It also prints the command it is using to the terminal (just above the stacktrace), if I run that manually I get the same result.

2. What's the output of `env | grep QT_PLUGIN_PATH` on your machine when you run the companion?
$ env | grep QT_PLUGIN_PATH
QT_PLUGIN_PATH=/home/akrimm/.nix-profile/lib/qt4/plugins:/home/akrimm/.nix-profile/lib/kde4/plugins:/etc/profiles/per-user/akrimm/lib/qt4/plugins:/etc/profiles/per-user/akrimm/lib/kde4/plugins:/nix/var/nix/profiles/default/lib/qt4/plugins:/nix/var/nix/profiles/default/lib/kde4/plugins:/run/current-system/sw/lib/qt4/plugins:/run/current-system/sw/lib/kde4/plugins

your wrapper then adds: /nix/store/44c67yi9hwrl6if4szkgd37s2wi0n0vy-qtbase-5.15.0-bin/lib/qt-5.15.0/plugin

I've updated once more the wrapper to set QT_PLUGIN_PATH and QML2_IMPORT_PATH as well. Could you please recheck now @wirew0rm ?

Still the same stacktrace unfortunately :/

Through the use of LD_DEBUG and ldd I found this:

$ ldd /nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/gnuradio/qtgui/_qtgui_swig.so | grep Qt5Core
	libQt5Core.so.5 => /nix/store/3hgcmz192jqdkc98r6pixamv3v09ibfn-qtbase-5.15.0/lib/libQt5Core.so.5 (0x00007f4921478000)

$ ldd /nix/store/4c1scdiqkyjlplqmd0f2w62yxm5l0fp5-python3-3.8.5-env/lib/python3.8/site-packages/PyQt5/QtCore.so | grep Qt5Core
	libQt5Core.so.5 => /nix/store/0n3h96is5klddhnin5kxf7bfxqx6ard3-qtbase-5.14.2/lib/libQt5Core.so.5 (0x00007fdfd6c59000)

I also tried generating C++ instead of python, as the dependency problems seem to come through the python modules, but it does not compile (didn't really expect it to, as the feature is pretty new and I'm not sure where it should get its compile environment from in nix, I guess first priority is getting existing functionality to work and then the new features).

@doronbehar
Copy link
Contributor Author

Ok thanks again for the investigation! The issue is: #99937 also experienced here: #99465 (comment) .

@doronbehar
Copy link
Contributor Author

This PR should wait for #99937 .

@markuskowa
Copy link
Member

@doronbehar thanks for tackling this mega task. I will do some testing on weekend.

@doronbehar
Copy link
Contributor Author

I've updated once more the wrapper to set QT_PLUGIN_PATH and QML2_IMPORT_PATH as well. Could you please recheck now @wirew0rm ?

Still the same stacktrace unfortunately :/

I think I'll leave these env vars never the less, just in case. In the meantime, I think the mismatched qt versions issue should be fixed, but it will trigger another rebuild of the gnuradio-unwrapped, so spare your CPU :).

What Window manager are you using?

i3wm, but I've just confirmed that I have Icons on a different machine, so I guess it's related to my configuration.

Likely. I think you should install an icon pack, such as breeze-icons.

I also tried generating C++ instead of python, as the dependency problems seem to come through the python modules, but it does not compile (didn't really expect it to, as the feature is pretty new and I'm not sure where it should get its compile environment from in nix, I guess first priority is getting existing functionality to work and then the new features).

If the qt gui test will pass now, this is worth checking once more, in your own time @wirew0rm . Thanks again for your help.

@wirew0rm
Copy link
Contributor

wirew0rm commented Oct 7, 2020

Yep, I just made the same fix myself based on the information from the other referenced issues. So far everything seems to work, which is really great! Thanks for all the work you put into this.

image

@doronbehar
Copy link
Contributor Author

Rebased to master without the qt override, thanks to #99956 now merged.

@doronbehar
Copy link
Contributor Author

I've had a slight misunderstanding regarding swig python deps which upstream explained to me in gnuradio/gnuradio#3874 . Now all patches are fetched with fetchpatch as they were accepted upstream:

A unnecessary patch was removed and gr-osmosdr and gqrx still build.

@evrim
Copy link
Contributor

evrim commented Nov 6, 2020

  • The top-level attribute: gnuradio-with-packages is gone now. I think it should be an alias pointing to gnuradio3_7.
  • I tried gnuradio3_7: the source plugins rtlsdr-source, osmosdr-source, limesdr-sink are gone now. Shouldn't they be in there?

I've tried to use gnuradio-with-packages but the modules are failing to import.

<<< Welcome to GNU Radio Companion 3.7.14.0 >>>

Block paths:
	/nix/store/dkyp8x0pqn540zd1x9smlw5xc2dg525n-gr-nacl-2017-04-10/share/gnuradio/grc/blocks
	/nix/store/bsbknbwv1sj9gikkas10la70jx4b98zy-gr-gsm-2016-08-25/share/gnuradio/grc/blocks
	/nix/store/kq2irc7f03cynzw2qfmhpcd0b4h25vwy-gr-ais-2015-12-20/share/gnuradio/grc/blocks
	/nix/store/f4zyfl769w97gzm0hm60q65d58vz97y5-gr-limesdr-2.0.0/share/gnuradio/grc/blocks
	/nix/store/9rcxfqln5gjb1xgh66xs168s9hljkfac-gr-rds-1.1.0/share/gnuradio/grc/blocks
	/nix/store/gxx5wda58b614yhg0jawzlh3vh75pxy6-gr-osmosdr-0.1.5/share/gnuradio/grc/blocks
	/nix/store/2awmkcws3jhdn28kwva0hmffrgg2928x-gnuradio-3.7.14.0/share/gnuradio/grc/blocks

Generating: '/home/void/gnuradio/top_block.py'

Executing: /nix/store/lhz9zglnxm8s7087clgprf4q3r3k4p9n-python-2.7.18-env/bin/python2.7 -u /home/void/gnuradio/top_block.py

Warning: failed to XInitThreads()
Traceback (most recent call last):
  File "/home/void/gnuradio/top_block.py", line 26, in <module>
    import osmosdr
ImportError: No module named osmosdr

Done

@doronbehar
Copy link
Contributor Author

I've tried to use gnuradio-with-packages but the modules are failing to import.

Thanks for the report @evrim. Could you please recheck now? Indeed I had a python path missing for the wrapper. it should not trigger for you a rebuild of the unwrapped derivation.

@markuskowa
Copy link
Member

I checked again. The source plugins are back now but the wx GUI widgets are gone now. It would be good to have them in the standard build since a lot of flow graphs are still make heavy use of them.

@doronbehar
Copy link
Contributor Author

The source plugins are back now but the wx GUI widgets are gone now. It would be good to have them in the standard build since a lot of flow graphs are still make heavy use of them.

OK. I disabled them by default initially because even versions pre 3.7.14 considered them deprecated, but I guess there's no harm in enabling them.

I do get some warnings though:

/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libc.so.6: version `GLIBC_2.32' not found (required by /nix/store/4d35k0mp5rc8lz4gzckiml3dg2y9s53x-spidermonkey-60.9.0/lib/libmozjs-60.so)
Failed to load module: /nix/store/a67a2md0lx0krb95zqn4vnnhvn1cfpnc-glib-networking-2.64.3/lib/gio/modules/libgiolibproxy.so
<<< Welcome to GNU Radio Companion 3.7.14.0 >>>

Block paths:
        /nix/store/lr3ar1w1i8jfcpxcak076cjiz8r12ykb-gr-nacl-2017-04-10/share/gnuradio/grc/blocks
        /nix/store/61a4838fczqxprvff6z4fip5vsc0dpdf-gr-gsm-2016-08-25/share/gnuradio/grc/blocks
        /nix/store/x66qx131fbz1jbf17kfxsyk106q0xdsb-gr-ais-2015-12-20/share/gnuradio/grc/blocks
        /nix/store/1zkqsgqziina5b6l8jcdh1cm8rxwd0rq-gr-limesdr-2.0.0/share/gnuradio/grc/blocks
        /nix/store/svarah9hmmkxwpcniz76sbfy6hrndhaq-gr-rds-1.1.0/share/gnuradio/grc/blocks
        /nix/store/q79x702wi2pfwbi8zc4vwdlk8lhfsk5z-gr-osmosdr-0.1.5/share/gnuradio/grc/blocks
        /nix/store/bsbjx6spjz2m2r5hi493iz4af53zz5az-gnuradio-3.7.14.0/share/gnuradio/grc/blocks
/nix/store/hzqxmzgxg03mcm0zl6vps9amcjrh0yaq-python-2.7.18-env/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py:16629: UserWarning: wxPython/wxWidgets release number mismatch
  warnings.warn("wxPython/wxWidgets release number mismatch")

And I suspect this is due to the fact that the wx packages are a bit of a mess:

wxPython30 = callPackage ../development/python-modules/wxPython/3.0.nix {
wxGTK = pkgs.wxGTK30;
inherit (pkgs) pkgconfig;
};
wxPython_4_0 = callPackage ../development/python-modules/wxPython/4.0.nix {
inherit (pkgs) pkgconfig;
wxGTK = pkgs.wxGTK30.override {
withGtk2 = false;
withWebKit = true;
};
};
wxPython = self.wxPython30;

wxGTK = wxGTK28;
wxGTK30 = wxGTK30-gtk2;
wxGTK31 = wxGTK31-gtk2;
wxGTK28 = callPackage ../development/libraries/wxwidgets/2.8 { };
wxGTK29 = callPackage ../development/libraries/wxwidgets/2.9 {
inherit (darwin.stubs) setfile;
inherit (darwin.apple_sdk.frameworks) AGL Carbon Cocoa Kernel QuickTime;
};
wxGTK30-gtk2 = callPackage ../development/libraries/wxwidgets/3.0 {
withGtk2 = true;
inherit (darwin.stubs) setfile;
inherit (darwin.apple_sdk.frameworks) AGL Carbon Cocoa Kernel QTKit;
};
wxGTK30-gtk3 = callPackage ../development/libraries/wxwidgets/3.0 {
withGtk2 = false;
inherit (darwin.stubs) setfile;
inherit (darwin.apple_sdk.frameworks) AGL Carbon Cocoa Kernel QTKit;
};
wxGTK31-gtk2 = callPackage ../development/libraries/wxwidgets/3.1 {
withGtk2 = true;
inherit (darwin.stubs) setfile;
inherit (darwin.apple_sdk.frameworks) AGL Carbon Cocoa Kernel QTKit;
};
wxGTK31-gtk3 = callPackage ../development/libraries/wxwidgets/3.1 {
withGtk2 = false;
inherit (darwin.stubs) setfile;
inherit (darwin.apple_sdk.frameworks) AGL Carbon Cocoa Kernel QTKit;
};

But this is out of scope for this PR.

@markuskowa
Copy link
Member

Let's give it a try if it works. Unfortunately a lot of legacy flow graphs and flow graph examples still make use of the wx widgets.To make things worse: the QT GUI blocks are not 100% identical replacements of the wx GUI blocks.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python-2-7-and-3-9-and-hydra-builds/10130/3

doronbehar and others added 7 commits December 5, 2020 13:23
Write (similar) expressions for GNURadio 3.7 and 3.8 and make 3.8
available as `gnuradio`, and `gnuradio3_7` point to the 3.7 build.

Teach both 3.7 & 3.8 expressions accept a `features` attribute set, that
tells them what features to compile. There are dependencies within the
different features, and we rely on upstream's cmake scripts to make sure
the `configurePhase` will fail if a feature is not enabled and needed by
another feature.  All features are enabled by default.

Put shared Nix functions and attributes for both 3.7 and 3.8 in:
pkgs/applications/radio/gnuradio/shared.nix

Add 2 patches accepted upstream, that don't install some python related
examples if python-support is not enabled.

Remove cmake python reference in 3.8 with removeReferencesTo, if
python-support is turned off.

Update gqrx (reverse dependency) to use a build of gnuradio3_7 without
gui components and for it's gr-osmosdr as well.

Write an external, `wrapper.nix` (shared for both 3.7 and 3.8). Teach it
to handle extra `gr-` packages via `GRC_BLOCKS_PATH`. Likely enable it
to accept extra python packages. Wrap the executables with env vars
wrapGAppsHook and wrapQtAppsHook would have likely given them (hence,
fix NixOS#87510). Point `gnuradio` to the wrapped 3.8 derivation.

Add @doronbehar to maintainers of both 3.8 and 3.7.

dirty: use upstreamed patches
Since gnuradio-runtime is compiled with log4cpp, all plugins that need
gnuradio-runtime (which is all of them probably) need log4cpp as well.

Add python.pkgs.cheetah to inputs of gr-osmosdr as it somehow started to
be needed.
Mention 3.8 and that it's now possible to override it and 3.7 to compile
only certain features.
@doronbehar
Copy link
Contributor Author

Fixed some merge conflicts along with a rebase. Sill waiting for a final approval @markuskowa :).

Copy link
Member

@markuskowa markuskowa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! It looks pretty good now.

@doronbehar
Copy link
Contributor Author

Thanks! It looks pretty good now.

Hallelujah 🍻 .

@doronbehar doronbehar merged commit a5e4403 into NixOS:master Dec 5, 2020
@doronbehar
Copy link
Contributor Author

I have plans for a gnuradio packages set and more improvements, but I don't have time yet to submit them. They are available if anyone is interested in some branches of mine:

https://github.com/doronbehar/nixpkgs/compare/pkg/gnuradio.dirty/packages

The only commit there is:

doronbehar@b3cedee

And it needs to be split and perhaps more comments are needed in the files.

@doronbehar doronbehar deleted the pkg/gnuradio branch December 10, 2020 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: clean-up 8.has: documentation This PR adds or changes documentation 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants