custom_target replaces back-slashes with slashes in the command arguments #1564

keszybz · 2017-04-05T05:46:27Z

Not sure about the details, but at least \t becomes /t and \\ becomes \:

project('meson3', 'c')

printf = find_program('printf')
custom_target('foo',
	      output : 'foo',
	      command : [printf, 'a\n\tb\nccc\n'],
	      capture : true)

custom_target('foo2',
	      output : 'foo2',
	      command : [printf, 'a\\n\\tb\\nccc\n'],
	      capture : true)

$ ninja-build && tail foo foo2
==> foo <==
a
/tb
ccc

==> foo2 <==
a
/tb
ccc

This makes it quite hard to write regexps and printf lines and such.

#1512 is probably related.

The text was updated successfully, but these errors were encountered:

keszybz · 2017-04-05T06:14:34Z

... or maybe meson should do a single layer of unescaping by itself, like python, so that it's possible to embed newlines and tabs in strings. Meaning that if I want to pass "\t\n\" to the program, I should write "\\t\\n\\".

QuLogic · 2017-04-05T06:48:04Z

See #737.

jpakkane · 2017-04-05T07:26:23Z

As the link indicates this is a thorny problem because we need to support Windows natively. Sadly there does not seem to be a way to pass all possible characters through both shell and cmd.exe (and other backends) that would work reliably.

The recommended way is to instead write your pipelines in standalone scripts and running those from Meson. Granted this is some amount of work beforehand but it is beneficial in several ways:

different programming languages (and thus paradigms) are cleanly separated in separate files
standalone scripts are a lot easier to edit and debug
no need to think about quoting quotes and all that terror when you have characters that have special meaning in Meson, shell and awk (for example)
portability becomes easier (this is not such a big issue in Linux only projects, but most are cross platform)
standalone scripts can be converted to Python one by one, and once that is done you no longer have a build dependency on Unix userland and can compile your project with nothing but Python 3 + Ninja

nirbheek · 2017-04-05T12:58:48Z

Honestly the piece of code that does that is wrong and we should remove it, but we just need to figure out why GStreamer needs it on Windows and fix whatever is broken in that regard.

Some of that will be on the GStreamer end, but I suspect some of that is bugs in Meson too.

jpakkane · 2017-04-26T11:26:52Z

To clarify: yes, we should pass all special characters through all shells and systems properly. Not doing that is a bug. However in practice this is is a really thorny problem and it is unlikely to ever work 100% reliably. Standalone scripts that are just invoked are always reliable, can be tested in isolation and are easy to convert to a different language (e.g. from Awk to Python or what have you) should the need arise.

johandc · 2017-08-21T10:03:08Z

This is also a problem when using the redirect symbol on Unix >.

Here is an example:
custom_target('lss', output: lss_name, command: [find_program('arm-none-eabi-objdump'), '-h', '-S', exec.full_path(), '>', lss_name], depends: exec, build_by_default: true)

This will call arm-none-eabi-gcc with quotes around the redirect symbol '>'.

/usr/bin/arm-none-eabi-objdump -h -S /xx/yy/zz '>' zz.lss

jpakkane · 2017-08-21T10:26:48Z

You should use the capture keyword instead.

jpakkane · 2017-08-21T10:44:34Z

Also, don't do exec.full_path() and depends : exec. Just put the exec object in the command list and Meson will take care of everything for you.

nirbheek · 2017-11-29T11:55:59Z

I think we should try removing that snippet and see what breaks. It will only get harder to remove it as time goes on.

nirbheek · 2018-03-25T13:52:37Z

I'll get to the bottom of why this was needed in gstreamer this week. FTR, the first step is setting up cerbero which is a massive yak-shave unless you've done it before.

This is no longer required since we always use `/` as the path separator for file arguments on Windows now. The only effect this now has is to mangle the string args people pass to custom targets. Closes #1564

Re-apply mesonbuild#737; revert mesonbuild#780. Fixes mesonbuild#1564.

rulatir · 2019-12-07T21:05:48Z

The recommended way is to instead write your pipelines in standalone scripts and running those from Meson.

So, I already knew Meson required wrappers to be written:

whenever you need to execute a pipeline of several binaries
whenever the build tool must be run from a specific directory

Now this list is extended with:

whenever the command contains backslashes

This is outright ridiculous. I think Meson should come with a BIG BOLD CAPITALIZED FLASHING SIDE SCROLLING WARNING that using it requires writing TONS of code, and it can only possibly make sense for your project if the amount of code you WILL have to write (and debug, and test) in order to accommodate all the quirks and deficiencies and opinionatednesses of Meson is less than say 15% of the actual codebase you want to build with it.

Re-apply mesonbuild#737; revert mesonbuild#780. Fixes mesonbuild#1564.

I want to avoid having to build the .o file before extracting malloc.o from libc.a. To parse the source file, we need a clear marker, which I also used to avoid having to have duplicate declarations. Because of Meson is incapable of using backslashes in custom_target() commands (see [1]), I had to write a shell script to execute sed, so I decided to move the ar x command there too. [1] mesonbuild/meson#1564 Signed-off-by: Thiago Macieira <[email protected]>

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved.

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b)

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b) (cherry picked from commit ded73e6)

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b) (cherry picked from commit ded73e6) (cherry picked from commit d31ac84)

I added the filtering in 752fedbea7c02c82287c7ff2a4139f528b3f7ba8 as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b013bbc8c6290348f701ddb862928cc1a0)

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b) (cherry picked from commit ded73e6) (cherry picked from commit d31ac84)

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b) (cherry picked from commit ded73e6) (cherry picked from commit d31ac84) (cherry picked from commit f924288)

I added the filtering in 752fedbea7c02c82287c7ff2a4139f528b3f7ba8 as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b013bbc8c6290348f701ddb862928cc1a0) (cherry picked from commit ded73e68fcbfbf82beba15a8f0280e5c010c8bde) (cherry picked from commit d31ac846b22dc7d1245b4df01a1868866b529447) (cherry picked from commit f9242887943fa2e7345f915fbfacd235d67dbed7)

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved.

I added the filtering in 752fedb as a way to reduce the number of items in the tables. I thought it's "obvious", but it might not be so. One immediate problem is that the filter is broken, because on arm64, os.uname().machine returns "aarch64", so we incorrectly filter out the arm syscalls (there is just one: arm_fadvise64_64). Of course we could fix the filter, but I think it's better to nuke it altogether. The filter on applies to 1 arm syscall and 5 s390 syscalls, and we have 500+ other syscalls, so this "optimization" doesn't really matter. OTOH, if we get the filter wrong, the result is bad. And also, the existence of the filter at all creates problems for cross-builds. I wanted to get rid of 'generate-syscall-list.py', but we need to generate a backslash in the output. mesonbuild/meson#1564 makes this very very hard, since any attempt to put a backslash an inline argument results in the backslash being replaces by a forward slash, which doesn't quite have the same meaning. So let's use a standalone script until mesonbuild/meson#1564 is resolved. (cherry picked from commit 58fcc6b) (cherry picked from commit ded73e6) (cherry picked from commit d31ac84) (cherry picked from commit f924288) (cherry picked from commit b23b268)

tmccombs · 2024-06-16T05:30:20Z

Could there at least be an option to custom_target to tell it whether it should replace backslashes with forward slashes or not?

I have a meson configuration where I use a foreach to generate several files from a template using m4, with variables set to different values (and the values are pretty small). In one case I need a backslash in one of the variables.

Using a wrapper script doesn't really solve the problem, because the backslash is part of the input. I suppose I could make a script the handles the full loop, but then the targets are less granular.

declantsien · 2024-06-26T13:45:59Z

https://sr.ht/~lattis/muon/ doesn't have this issue. Which is a
meson implementation in C.

theoparis · 2024-07-08T02:25:47Z

https://sr.ht/~lattis/muon/ doesn't have this issue. Which is a meson implementation in C.

Muon also doesn't support features needed to compile major projects such as GTK.

lf- · 2024-08-31T04:42:06Z

If meson is going to corrupt the input to custom_target, can this gotcha at least be documented in the manual?

tristan957 · 2024-11-11T16:36:49Z

We just ran into this in Postgres. Really unfortunate for a build tool not to do what the developer asks of it.

meson makes the backslashes in text2macro.pl's --strip argument into forward slashes, effectively disabling comment stripping. That hasn't caused us issues before, but it breaks the test case for b7e3a52. We don't really need the pattern to be adjustable, so just hard-wire it into the script instead. Context: mesonbuild/meson#1564 Security: CVE-2024-10979

meson makes the backslashes in text2macro.pl's --strip argument into forward slashes, effectively disabling comment stripping. That hasn't caused us issues before, but it breaks the test case for b7e3a52a8. We don't really need the pattern to be adjustable, so just hard-wire it into the script instead. Context: mesonbuild/meson#1564 Security: CVE-2024-10979 (cherry picked from commit 4cd4f3b97492c1b38115d0563a2e55b136eb542a)

keszybz mentioned this issue Apr 5, 2017

meson: build systemd using meson systemd/systemd#5704

Merged

nirbheek added bug generators labels Apr 5, 2017

nirbheek mentioned this issue Nov 29, 2017

Backslashes are converted to slashes in custom_target ? #2705

Closed

QuLogic mentioned this issue Mar 24, 2018

Double backslash is replaced by single forwardslash in custom_target command #3313

Closed

sarum9in self-assigned this Mar 25, 2018

nirbheek mentioned this issue Apr 13, 2018

custom targets: Don't replace \\ with / #3400

Closed

sarum9in assigned nirbheek and unassigned sarum9in Apr 21, 2018

QuLogic added a commit to QuLogic/meson that referenced this issue Oct 19, 2018

Stop "fixing" command line in CustomTargets again.

ebaf487

Re-apply mesonbuild#737; revert mesonbuild#780. Fixes mesonbuild#1564.

QuLogic linked a pull request Oct 19, 2018 that will close this issue

Stop "fixing" command line in CustomTargets again. #4393

Open

QuLogic mentioned this issue May 16, 2019

run_command seems to fail when there is a backslash in the command name #5400

Open

evverx mentioned this issue May 17, 2019

tests: make docker-run and helper.py happy by not using colons in fil… systemd/systemd#12589

Merged

QuLogic added a commit to QuLogic/meson that referenced this issue Oct 23, 2020

Stop "fixing" command line in CustomTargets again.

b1aca30

Re-apply mesonbuild#737; revert mesonbuild#780. Fixes mesonbuild#1564.

QuLogic added a commit to QuLogic/meson that referenced this issue Oct 27, 2020

Stop "fixing" command line in CustomTargets again.

1feb2e7

Re-apply mesonbuild#737; revert mesonbuild#780. Fixes mesonbuild#1564.

QuLogic added a commit to QuLogic/meson that referenced this issue Nov 30, 2020

Stop "fixing" command line in CustomTargets again.

242cfad

Re-apply mesonbuild#737; revert mesonbuild#780. Fixes mesonbuild#1564.

QuLogic mentioned this issue Apr 19, 2021

'\\' is getting escaped as '/' rather than '\' #8677

Closed

thiagomacieira mentioned this issue Jun 29, 2023

Meson: Fix sysdeps build flags by reorganising the flags opendcdiag/opendcdiag#283

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

custom_target replaces back-slashes with slashes in the command arguments #1564

custom_target replaces back-slashes with slashes in the command arguments #1564

keszybz commented Apr 5, 2017 •

edited by TingPing

Loading

keszybz commented Apr 5, 2017

QuLogic commented Apr 5, 2017

jpakkane commented Apr 5, 2017

nirbheek commented Apr 5, 2017 •

edited

Loading

jpakkane commented Apr 26, 2017

johandc commented Aug 21, 2017

jpakkane commented Aug 21, 2017

jpakkane commented Aug 21, 2017

nirbheek commented Nov 29, 2017

nirbheek commented Mar 25, 2018

rulatir commented Dec 7, 2019 •

edited

Loading

tmccombs commented Jun 16, 2024

declantsien commented Jun 26, 2024

theoparis commented Jul 8, 2024

lf- commented Aug 31, 2024

tristan957 commented Nov 11, 2024

custom_target replaces back-slashes with slashes in the command arguments #1564

custom_target replaces back-slashes with slashes in the command arguments #1564

Comments

keszybz commented Apr 5, 2017 • edited by TingPing Loading

keszybz commented Apr 5, 2017

QuLogic commented Apr 5, 2017

jpakkane commented Apr 5, 2017

nirbheek commented Apr 5, 2017 • edited Loading

jpakkane commented Apr 26, 2017

johandc commented Aug 21, 2017

jpakkane commented Aug 21, 2017

jpakkane commented Aug 21, 2017

nirbheek commented Nov 29, 2017

nirbheek commented Mar 25, 2018

rulatir commented Dec 7, 2019 • edited Loading

tmccombs commented Jun 16, 2024

declantsien commented Jun 26, 2024

theoparis commented Jul 8, 2024

lf- commented Aug 31, 2024

tristan957 commented Nov 11, 2024

keszybz commented Apr 5, 2017 •

edited by TingPing

Loading

nirbheek commented Apr 5, 2017 •

edited

Loading

rulatir commented Dec 7, 2019 •

edited

Loading