Make escapeSystemdPath implement the correct systemd escaping algorithm

[dali99]

Description of changes

There's a regression in regards to systemd mount units on 22.05:
Since #152372 we validate unit names but that validation isn't the same as what escapeSystemdPath does, notably it doesn't replace ( ) in my case
filesSystems uses escapeSystemdPath to convert the mountpoint to the unit name.

I've added a few functions to help convert the paths to the proper C escapes implementing the algorithm described in man systemd.unit But this is my first time working or changing anything significant in lib, so I'd be really happy for pointers on how to do this better utilizing functions we maybe already have?

I tested this on my configuration (cherry-picked to 22.05) and it seems fine... but this is a basically-8 year old function I'm changing which is pretty scary...

I also only did ASCII special chars, but without something like a builtins function for the UTF-8 conversion stuff I don't know if it's possible to generalize

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[ajs124]

Just commenting so this isn't only documented on matrix. I also tried to fix this recently, in #176115.

So far, we've figured out that this PR here properly translates characters, while mine handles the / special case and deduplicates consecutive /s.

I don't really have any preference over which PR should be merged into which one or how else to proceed, as long as we have an implementation that's as complete as possible, in the end.

[Mindavi]

Can we add tests to make the behavior more clear and to document use cases? Also can help fixing any issues we find after getting this in, by improving the tests then.

[dali99]

@Mindavi We don't seem to have any testing infrastructure for nixos/lib So I've only added tests for the new string functions (and only positive ones at that)

I feel like the escapeSystemdPath is complicated enough to warrant some testing but I'm not sure if this PR is the right place to add tests to nixos/lib

[tadfisher]

An issue should be opened in nix to provide a built-in for string formatting. So much of this would be a simple printf in C, even codepoint conversion.

[symphorien]

mention that it only works with printable ascii characters. For non printable ones, I wonder if it is correct to map the caracter 7 to \x7 instead of \x07.

[dali99]

I've now documented that it only works for printable characters, You're quite right about the non-printable ones as well, but I think we would need a nix builtin to deal with those anyways

[symphorien]

@GrahamcOfBorg test systemd

[symphorien]

(note that github does not always send notifications on force-push, so when you address a review, please post a full-fledged comment)

[symphorien]

this breaks nixosTest.systemd (it looks "currently building" but the test vm is just stuck failing to boot)
problem is: the vm uses the autoformat feature, but the unit running mkfs does not run, I don't know why.

[dali99]

@GrahamcOfBorg test systemd

[symphorien]

thank you!

[zowoq]

nix-build -A nixosTests.simple --max-jobs 0 --eval-store auto --store ssh-ng://**** --system x86_64-linux
error: invalid regular expression ''

       at /Users/zowoq/src/nixpkgs/lib/strings.nix:540:51:

          539|       s = builtins.unsafeDiscardStringContext _s;
          540|       splits = builtins.filter builtins.isString (builtins.split (escapeRegex sep) s);
             |                                                   ^
          541|     in
(use '--show-trace' to show detailed location information)

Only seems to be a problem when running nixosTests remotely? Works again after reverting 3251123.

[symphorien]

If this happens only when evaluating on Darwin but not on linux it's probably a discrepancy between the glicxx and libcxx implem of std::regex in nix. I don't have a darwin system to investigate further, could you do it as you seem to have both kinds of systems, @zowoq ? 22 oct. 2022 02:49:17 zowoq ***@***.***>:

…

*nix-build -A nixosTests.simple --max-jobs 0 --eval-store auto --store ssh-ng://**** --system x86_64-linux error: invalid regular expression '' at /Users/zowoq/src/nixpkgs/lib/strings.nix:540:51: 539| s = builtins.unsafeDiscardStringContext _s; 540| splits = builtins.filter builtins.isString (builtins.split (escapeRegex sep) s); | ^ 541| in (use '--show-trace' to show detailed location information) * Only seems to be a problem when running nixosTests remotely? Works again after reverting 3251123[3251123]. — Reply to this email directly, view it on GitHub[#177273 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ADADGA5TE4MX7KLM2FEUGJDWEM2YJANCNFSM5YP4G32Q]. You are receiving this because you modified the open/close state.[Image de pistage][https://github.com/notifications/beacon/ADADGA3IR2CHNXZM7D3FT43WEM2YJA5CNFSM5YP4G322YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOJS7KMMA.gif]

[zowoq]

Yeah, only seems to be darwin->linux. If it is a nix bug I doubt it will be quick or simple to fix. I'd like to revert this PR until it is resolved.

[dali99]

I'd be okay with that if we also reverted #152372 I don't want to maintain a nixpkgs channel with this patch for this coming release as well

[symphorien]

So the code in this PR does not involve regex at all except with the hard coded regex "/" and "" which are valid, could you report the error with a stacktrace and possibly with --debugger so that we know the value of the faulty regex?

[symphorien]

(besides this is a bugfix in nixos code that is only faulty when evaluating code on mac, which is more niche than the use cases it fixes imo, so I'd rather fix this than revert it).

[zowoq]

Can you suggest another workaround apart from reverting this locally on every branch that I'm working on?

I don't see that it matters if we revert it for a bit so we aren't introducing new breakage.

[dali99]

This fixes a (at the time new) breakage on actual NixOS systems which IMO probably takes priority over evaluating nixos tests from darwin. I believe one can just ssh to a Linux box and run tests there in the mean time.

This exposing something broken in nix on darwin hopefully incentives someone who has access and cares about darwin to investigate and follow up the bug.

[zowoq]

This fixes a (at the time new) breakage on actual NixOS systems which IMO probably takes priority over evaluating nixos tests from darwin.

It is old code that has been broken for months, there certainly didn't seem to be any urgency in merging this PR. I don't see the problem with reverting it for a bit.

I believe one can just ssh to a Linux box and run tests there in the mean time.

Doesn't seem to be that simple as would need to sync code as well?

This exposing something broken in nix on darwin hopefully incentives someone who has access and cares about darwin to investigate and follow up the bug.

Yes, I'll try to do that. Not having my current workflows broken would mean I have more time to try to fix it?

[symphorien]

@zowoq: can you try #197374 ?

[symphorien]

The discrepancy is problematic so I opened NixOS/nix#7208

[zowoq]

Seems this is a recurring problem.

NixOS/nix#1537 (comment)
Empty regular expressions are not allowed according to the POSIX extended regex grammar