segmentation fault

[eworm-de]

Version 0.9.2 crashes with a segmentation fault:

$ multipath -t 2
612.793257 | couldn't open configuration dir '/etc/multipath/conf.d': Permission denied
[1]    13266 segmentation fault (core dumped)  multipath -t 2

It crashes with snprint_keyword() inside snprint_defaults()...

https://github.com/opensvc/multipath-tools/blob/master/libmultipath/print.c#L1635

The final crash happens inside snprint_keyword()...

https://github.com/opensvc/multipath-tools/blob/master/libmpathutil/parser.c#L181

[mwilck]

Thanks for the report! Adding @bmarzins, for information.

This appears to be a subtle interaction between the separation of libmultipath and libmpathutil we did in 0.9.1 and the libmultipath ABI version bump in 0.9.2.

Some symbols ({get,put}_multipath_config, libmp_verbosity, and udev) are exported by both libmpathutil and
libmultipath. The symbols exported by libmpathutil are dummies, intended to be overridden by libmultipath using the "breadth-first" symbol lookup specified by ELF ¹. The dummy symbols are only used by programs not linking to libmultipath, i.e. only multipathc.

Because we're using ABI versioning, code in libmpathutil actually exports and references get_multipath_config@LIBMULTIPATH_16.0.0. That worked until 0.9.1, because both libraries were using the same ABI version, and get_multipath_config@LIBMULTIPATH_16.0.0 from libmultipath overrides the symbol from libmpathutil. But in 0.9.2, we bumped the ABI version of libmultipath to 17.0.0. libmultipath now exports (only) get_multipath_config@LIBMULTIPATH_17.0.0, which doesn't override get_multipath_config@LIBMULTIPATH_16.0.0 from libmpathutil.

The effect is that references to these duplicate symbols, when done from libmpathutil, are resolved to the libmpathutil dummy symbols, which is wrong and causes the crash.

This affects only multipath, not multipathd, because multipathd overrides get_multipath_config with yet another symbol. Same for put_multipath_config and libmp_verbosity, but not for udev.

Footnotes

1. Executable and Linkable Format (ELF), p. 2-15 ("Shared Object Dependencies") ↩

[mwilck]

Error case (0.9.2):

# LD_DEBUG=symbols,reloc,bindings ./sbin/multipath -t
     28311:     binding file /tmp/mp/lib64/libmpathutil.so.0 [0] to /tmp/mp/lib64/libmpathutil.so.0 [0]: normal symbol `get_multipath_config' [LIBMULTIPATH_16.0.0]
     28311:     binding file /tmp/mp/lib64/libmultipath.so.0 [0] to /tmp/mp/lib64/libmultipath.so.0 [0]: normal symbol `get_multipath_config' [LIBMULTIPATH_17.0.0]
     28311:     binding file ./sbin/multipath [0] to /tmp/mp/lib64/libmultipath.so.0 [0]: normal symbol `get_multipath_config' [LIBMULTIPATH_17.0.0]

We can see that the symbol is resolved differently (first line resolves to libmpathutil, the others to libmultipath). For multipathd, we get:

# LD_DEBUG=bindings ./sbin/multipathd -d 2>&1 | egrep '`get_multipath_config'
     31936:     binding file /tmp/mp/lib64/libmpathutil.so.0 [0] to ./sbin/multipathd [0]: normal symbol `get_multipath_config' [LIBMULTIPATH_16.0.0]
     31936:     binding file /tmp/mp/lib64/libmultipath.so.0 [0] to ./sbin/multipathd [0]: normal symbol `get_multipath_config' [LIBMULTIPATH_17.0.0]
     31936:     binding file /tmp/mp/lib64/libmpathpersist.so.0 [0] to ./sbin/multipathd [0]: normal symbol `get_multipath_config' [LIBMULTIPATH_17.0.0]

All locations are resolved to multipathd's symbol.

[mwilck]

The easiest workaround for this is to revert e5eb226. But that's not a final solution, it would hide our ABI changes. It should work as temporary workaround.