Regex behavior differs across platforms

[copumpkin]

I seem to remember seeing some regex-related changes in 1.12 so this might no longer be an issue, but I want to capture it in case it is. If I fire up nix-repl on 1.11.x on Linux and Darwin and run

builtins.match "^((|\..*)\.sw[a-z]|.*~)$" "foo"

Darwin will complain that error: compiling pattern ‘^((|..*).sw[a-z]|.*~)$’: empty (sub)expression, whereas Linux will return null.

[copumpkin]

This is slightly painful because that happens to be the regex we use to filter junk from nixpkgs in the standard NixOS image builder, and I need to be able to evaluate the image build process on Darwin 😦

cc @edolstra

[copumpkin]

Yeah, so I guess we moved from 1.11's calls to regcomp which I guess varies from platform to platform, to std::regex in 1.12, which I assume is consistent across platforms.

[copumpkin]

@edolstra do you have some sort of plan for the cutover to 1.12 with differing behavior in builtins.match? Do we need to bump the minimum Nix version on nixpkgs to make it work? Otherwise, silently changing behavior could cause a lot of woe.

[edolstra]

I don't have a plan. I would suggest rewriting that regexp into something that doesn't rely on Linux-specific behavior.

Also, builtins.match was undocumented in 1.11 so using it was at your own risk anyway ;-)

[edolstra]

Anyway, the behavior no longer differs across platforms, so I'll close this.

[clacke]

Still an issue on 2.0pre:

Darwin:

$ nix-instantiate --version
nix-instantiate (Nix) 2.0pre5968_a6c0b773
$ nix-instantiate --eval -E 'builtins.match "(|.*/)([^/]*)" "path/to/blah"'
error: invalid regular expression '(|.*/)([^/]*)', at (string):1:1

Linux:

$ nix-instantiate --version
nix-instantiate (Nix) 2.0pre5914_48c192ca
$ nix-instantiate --eval -E 'builtins.match "(|.*/)([^/]*)" "path/to/blah"'
[ "path/to/" "blah" ]

[clacke]

Verified that Darwin nix 2.0 also has this issue.

[copumpkin]

That's surprising! The full source to match is this:

nix/src/libexpr/primops.cc

Lines 1771 to 1806 in cfdbfa6

	static void prim_match(EvalState & state, const Pos & pos, Value * * args, Value & v)
	{
	auto re = state.forceStringNoCtx(*args[0], pos);
	try {
	std::regex regex(re, std::regex::extended);
	PathSet context;
	const std::string str = state.forceString(*args[1], context, pos);
	std::smatch match;
	if (!std::regex_match(str, match, regex)) {
	mkNull(v);
	return;
	}
	// the first match is the whole string
	const size_t len = match.size() - 1;
	state.mkList(v, len);
	for (size_t i = 0; i < len; ++i) {
	if (!match[i+1].matched)
	mkNull(*(v.listElems()[i] = state.allocValue()));
	else
	mkString(*(v.listElems()[i] = state.allocValue()), match[i + 1].str().c_str());
	}
	} catch (std::regex_error &e) {
	if (e.code() == std::regex_constants::error_space) {
	// limit is _GLIBCXX_REGEX_STATE_LIMIT for libstdc++
	throw EvalError("memory limit exceeded by regular expression '%s', at %s", re, pos);
	} else {
	throw EvalError("invalid regular expression '%s', at %s", re, pos);
	}
	}
	}

which suggests that std::regex doesn't behave consistently across platforms? Seems odd!

[copumpkin]

Probably worth including the regex_error in the "invalid regular expression" error message to help track this down further?

[shlevy]

Probably the example case here is enough to reproduce independent of Nix?

[clacke]

I'm assuming std::regex on Linux comes from libstdc++ and the one on Darwin comes from LLVM?

[edolstra]

This appears to be a bug in libstdc++. Empty regular expressions are not allowed according to the POSIX extended regex grammar (http://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap09.html#tag_09_05_03).

This also affects grep:

linux$ echo foo | grep -E "(|x)" 
foo

macos$ echo foo | grep -E "(|x)"
grep: empty (sub)expression

[LnL7]

We could use libcxxStdenv for all platforms 😄

[tomberek]

Another variation of the same problem:
builtins.split "" "abcd"

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-regex-match/7946/9

[stale]

I marked this as stale due to inactivity. → More info