Package request: lem

[Ashvith10]

Project description

Lem is an editor/IDE well-tuned for Common Lisp development. It is designed to be lightweight and fast, while still providing a rich set of features for Common Lisp development. Lem supports other programming languages thanks to its built-in LSP client. You can choose between an Emacs and a Vim mode.

Other Lem features include:

• A project explorer tree view
• Project-aware commands
• Interactive grep and git interfaces
• Multiple cursors
• Tetris
• and more...

Metadata

• homepage URL: https://lem-project.github.io/lem-page/
• source URL: https://github.com/lem-project/lem
• license: mit
• platforms: unix

You can refer the code snippet was working on before:

https://github.com/Ashvith10/nixpkgs/blob/3580e652c4f5f5ef9d0dd55bffdc9a6cb77438b5/pkgs/applications/editors/lem/default.nix#L41-L61

Pinging @Uthar so that we can continue our conversation here...

[dariof4]

Just seeing this now, but I had a flake for lem that seems to work fine if it may help (loading contrib systems doesn't work, they need to be (asdf:load-system) in the derivation (which unfortunately causes a rebuild), in which case some work, but some other don't since they use (ql:quickload) within themselves (such as calc-mode), they'll need to be patched to use (asdf:load-system)).

https://github.com/dariof4/lem-flake/

I added a lem.withPackages to my flake so you can have lem with some extra lisp packages able to be (asdf:load-system)'ed, this would probably be nicer if they were automatically loaded, but I think that would require making the executable a lisp script that loads the specified systems and then runs (lem:main) rather than turning lem into an executable sbcl image like I did.

There's a bit of a problem packaging lem into an executable binary though imo, and that is the fact that it uses it's own sbcl image for lisp-mode and the lisp repl, so if you have a flake that defines a sbcl.withPackages lem won't be aware of those packages, and due to not using quicklisp on nixos it can't quickload them from whatever, this means that you'd need add lem (along with any lem extension) in sbcl.withPackages in your lisp development shell, but that's annoying since I think most people would like to define their editor (and extension packages) not in their dev shells, but in their user/system configuration.

I think there are ways to go about it though, setting CL_SOURCE_REGISTRY in your lisp dev shell to point to lisp packages makes them able to be loaded with (asdf:load-system) in lem even if it's using it's own sbcl image.

I'd also like to mention that the roswell installation method works fine afaik, although it's not really a nix way to go about it :P.

[Uthar]

Also see this for some progress (without extensions yet)

Uthar@7c51d8e

[AndersonTorres]

Should we package the precompiled binary first, just to have something to offer and compare?

[dariof4]

I think the best way to go about it is what @Uthar has done by packaging lem as a standalone lisp package, then packaging lem-ncurses and lem-sdl2 as separate lisp binaries using (sb-ext:save-lisp-and-die) (see: https://github.com/lem-project/lem/blob/main/scripts/build.lisp) like stumpwm does it, packaging them as binaries seems to make startup faster compared to a script with starts sbcl and (asdf:load-system)s lem.
Packaging them as lisp executables is also nice so you can install it globally and don't have to add lem to sbcl.withPackages in lisp dev shells.

The only questions I have with this approach are three

1. Should we separately package all the lem extensions one by one and add them as dependecies for lem or just directly bundle all their dependecies to lem itself?
2. It would be nice to have a lem.withPackages or something similar to bundle extra lisp packages with lem, something similar to emacs.withPackages
3. What would be the best way to add lisp packages to the lem executable's sbcl from dev shells? This is important since if lem is built as an executable it uses its image's sbcl when using lem's lisp-mode and lisp-repl rather than the sbcl.withPackages you would define in your shell.nix, I'm not sure if there's a way to use another sbcl instance though.

[AndersonTorres]

Well, in the case of Emacs we leverage the Emacs' own package system, package.el, with some post, mid and preprocessing.

It allows a huge deal of automation, to the point basically all ELPA and MELPA packages are built by our automation scripts.

[Ashvith10]

@Uthar could you update your branch with the current changes in upstream? Your main branch is dated by almost six months, and it looks like changes in ql.nix has already been resolved through the PR that solved the issue created by me about a few weeks ago.

About our previous conversation, I would like to think that lem is a SBCL-only package - but I could be wrong. I am not sure if the actual maintainers for the project want interoperability between the different compilers. @Sasanidas , what would you like to say over this?

[dariof4]

There was some work to make lem work on different CL implementations (https://lem-project.github.io/lem-page/development/implementations-details/, lem-project/lem#686), although SBCL is the main supported one.
Having lem be buildable on different implementations in nixos is nice, although not too big of a deal since I assume most people would run it with SBCL anyway.

[Uthar]

could you update your branch with the current changes in upstream?

Good idea - merged master just now

Should we separately package all the lem extensions one by one and add them as dependecies for lem or just directly bundle all their dependecies to lem itself?

We could try either way - bundling them to lem would be a test of how multi-system pacakaging works, I don't remember if it's done like this for anything so far.

What would be the best way to add lisp packages to the lem executable's sbcl from dev shells?

Maybe appending to CL_SOURCE_REGISTRY would work? Or maybe we would have to start writing FASLs to $out/share/common-lisp/ or something instead of next to the source files (or maybe I'm thinking about using packages via nix profile install and this is unrelated)

[Ashvith10]

I was going to start packaging kandria, but CommonLisp reminded me of this pending issue. @Uthar what is the state of that branch? I tried installing lem-ncurses, but I could not figure out a way to run the executable.

[Sasanidas]

As a reminder, we changed the way that Lem is installed, now we use https://github.com/fukamachi/qlot , so we change slightly the way that Lem is installed https://lem-project.github.io/installation/ncurses/linux/

[Ashvith10]

@Sasanidas thankfully, qlot is already available on Nixpkgs, but it might have to be updated. But I don't really know much about this.

[Sasanidas]

Then I think that can solves quite a few issues regarding the installation of Lem (as now it uses qlot for dependencies)

[Ashvith10]

@Sasanidas unfortunately, not for NixOS. Since packages have to be reproducible, we cannot use external sources. But I'm assuming that we can use pre-packaged SBCL packages, which should be accessible to sbclPackages.qlot and make it skip the download part or something? Again, I have no hardly any exposure to CL, and just came across Guile, so I can't say much. Most of my packaging effort is just checking out the other examples, followed by trial and error.

[Uthar]

I'm trying to package the SDL2 variant. Editing works, but I can't connect to LSP and there is an issue with "SDL-SURFACE null pointer" sometimes (when calling M-x documentation-describe-bindings).

Maybe someone else wants to debug: nix run github:uthar/nixpkgs?rev=27906f65445ea72f33f75cc91a4f5ad717cdd123#lem

Patches: https://github.com/NixOS/nixpkgs/compare/nixpkgs-unstable...Uthar:nixpkgs:lem-sdl2-scratch.patch

[Ashvith10]

@Uthar I have moved over to Guix for almost half a year. Unfortunately, I won't be able to test it out properly.

[eriedaberrie]

there is an issue with "SDL-SURFACE null pointer" sometimes (when calling M-x documentation-describe-bindings)

@Uthar IIRC that's a problem with SDL2_ttf packages that were built using the builtin freetype, nixpkgs no longer does as of #308379. Maybe try with the newer SDL2_ttf package and see if the problem is still there?

WRT to dependencies, IMO lem packaging is a lot simpler and more maintainable in the long run if we just use qlot in a FOD instead of trying to package every single lem dependency in nix, since qlot comes with a lockfile and is in theory reproducible. I'd been working on my own lem package (both sdl2 and ncurses) at https://github.com/eriedaberrie/my-nix-packages/blob/main/pkgs/lem/default.nix that does just that. (The package itself is very hacky and WIP, the executable launches perfectly fine and can edit basic code, although there's some problems when loading systems during runtime and trying to write to /nix/store or reading from /build that will need to get fixed.)

[Uthar]

thanks @eriedaberrie - it works with that flag.

yeah, it sounds better to use the qlfile.lock. I assumed that it doesn't pin quicklisp, but it does.