LIBFFI_TMPDIR should point somewhere that permits executables #77014

DaveCTurner · 2021-08-30T10:10:52Z

JNA requires a temporary directory which is not mounted noexec, but JNA also uses libffi which itself requires a temporary directory that isn't mounted noexec. If none of the usual suspects work, libffi will fall back to $HOME which will often work, but in general might also not permit executables (or even exist).

We override the temporary directory that JNA uses by setting the java.io.tmpdir system property to ES_TMPDIR. I believe we should do the same thing for libffi, by setting the LIBFFI_TMPDIR environment variable. We should also document that this can be overridden much as we do for the jna.tmpdir system property (which trumps java.io.tmpdir if set).

Fixing this would, I believe, also fix #73309.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-08-30T10:10:55Z

Pinging @elastic/es-delivery (Team:Delivery)

pugnascotia · 2021-08-30T13:50:48Z

This sounds like a reasonable change. Do we have any ideas around testing this?

DaveCTurner · 2021-08-30T16:18:22Z

I see a couple of options:

We could have packaging tests that run on a SELinux installation that's sufficiently well-locked-down to reproduce the crash, and then confirm that setting LIBFFI_TMPDIR does resolve the problem. I now believe I have a decent handle on why the crash happens and how to reproduce it, so it's just™ a matter of setting up SELinux packaging tests 😉
We could just verify that Elasticsearch sees the LIBFFI_TMPDIR variable we expect it to see. Maybe we could even fail the node if running on Linux and that variable is not set, given that the startup scripts should be setting it?

DaveCTurner · 2021-08-31T16:30:06Z

After a bit of digging, it turns out that support for $LIBFFI_TMPDIR was added in libffi-3.4.2 but the latest release of JNA is using libffi-3.3.0, so we'll need to wait for JNA to pick up the newer version, then pick up the new JNA, and only then can we proceed with this.

mark-vieira · 2021-08-31T22:28:00Z

We could have packaging tests that run on a SELinux installation that's sufficiently well-locked-down to reproduce the crash

Yeah, seems we don't have such a testing environment right now but probably should. Do we have an idea of the specific controls that need to be enabled to reproduce?

DaveCTurner · 2021-09-01T06:43:43Z

Do we have an idea of the specific controls that need to be enabled to reproduce?

We know at least some of them I think:

SELinux enabled & configured to prevent mmap() from creating anonymous executable & writeable mappings
/tmp, /var/tmp, /dev/shm all nonexistent or mounted noexec
$HOME nonexistent for the elasticsearch user
All mount points in /etc/mtab not writable by elasticsearch, or mounted noexec
$ES_TMPDIR (and hence java.io.tmpdir and/or jna.tmpdir) pointing somewhere writeable and not mounted noexec.
$TMPDIR unset

That might not be everything, but this problem comes up frequently enough that I don't think it's going to need anything especially weird to reproduce. If the list above isn't enough then we can use strace to see what we're still missing I think (by comparing to the log shared in #77053 (comment)).

Upgrade to 5.10.0 The latest JNA release introduces the features needed to resolve elastic/elasticsearch#77014. This commit updates the version we repackage to pick these features up.

DaveCTurner · 2021-11-10T15:54:32Z

Didn't intend to close this, reopening.

This commit upgrades JNA to 5.10.0. The primary reason for this upgrade is to adopt a newer version of `libffi` which supports the `LIBFFI_TMPDIR` environment variable so that we can change the location of temporary executables and resolve elastic#77014. This commit also switches to the upstream JNA releases rather than using the specially-repackaged ones we've used in the past.

spricer · 2021-11-10T17:48:32Z

It seems that -Djava.io.tmpdir alone is not enough to allow elasticsearch startup.
I analyzed running tasks of elasticsearch when /tmp is mounted with noexec removed and there is additional path required for elasticsearch to work.
This variable is called TMPDIR.
So in order to make it work the best way is to append next in /etc/sysconfig/elasticsearch file:

ES_TMPDIR=/var/lib/elasticsearch/tmp
TMPDIR=$ES_TMPDIR

If you have issue with above path (e.g. also mounted with noexec flag) you can introduce new path that is not under
noexec flag:

mkdir /opt/tmp
chown elasticsearch:elasticsearch /opt/tmp
chmod 755 /opt/tmp

Then update /etc/sysconfig/elasticsearch file accordingly:

ES_TMPDIR=/opt/tmp
TMPDIR=$ES_TMPDIR

I think developers need to hardcode somewhere TMPDIR=$ES_TMPDIR

DaveCTurner · 2021-11-10T17:53:16Z

That's one possible workaround @spricer, but TMPDIR has a wide impact and it's not always acceptable to redirect all temporary files to a location that permits executables. Instead we plan to set LIBFFI_TMPDIR instead as described above.

This commit upgrades JNA to 5.10.0. The primary reason for this upgrade is to adopt a newer version of `libffi` which supports the `LIBFFI_TMPDIR` environment variable so that we can change the location of temporary executables and resolve #77014. This commit also switches to the upstream JNA releases rather than using the specially-repackaged ones we've used in the past.

This commit upgrades JNA to 5.10.0. The primary reason for this upgrade is to adopt a newer version of `libffi` which supports the `LIBFFI_TMPDIR` environment variable so that we can change the location of temporary executables and resolve elastic#77014. This commit also switches to the upstream JNA releases rather than using the specially-repackaged ones we've used in the past.

This commit upgrades JNA to 5.10.0. The primary reason for this upgrade is to adopt a newer version of `libffi` which supports the `LIBFFI_TMPDIR` environment variable so that we can change the location of temporary executables and resolve #77014. This commit also switches to the upstream JNA releases rather than using the specially-repackaged ones we've used in the past.

DaveCTurner · 2021-11-11T14:32:02Z

Ugh Github keeps misinterpreting commit messages and closing this. Reopening again...

Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285 Co-authored-by: Rory Hunter <[email protected]>

Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <[email protected]>

Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285 Co-authored-by: Rory Hunter <[email protected]>

Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <[email protected]> Co-authored-by: Rory Hunter <[email protected]>

spricer · 2021-11-15T14:37:46Z

That's one possible workaround @spricer, but TMPDIR has a wide impact and it's not always acceptable to redirect all temporary files to a location that permits executables. Instead we plan to set LIBFFI_TMPDIR instead as described above.

Hi Dave I understand your concern but please also consider that elasticsearch is executing under elasticsearch user session (at least Linux side) with its own env vars. That means that TMPDIR would be affected for that session/user only and if no other JAVA app is running noting else would be affected.

* Set LIBFFI_TMPDIR at startup (#80651) Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <[email protected]> * Fix incorrect SSL usage Co-authored-by: Rory Hunter <[email protected]>

DaveCTurner · 2021-12-09T09:02:46Z

Sorry for the delay in getting back to you on this @spricer. I'm not sure I understand your last message. Are you saying that the fix in 7.16 (now released) doesn't work for you? If so, can you share more details of how it fails?

DaveCTurner added >enhancement :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts labels Aug 30, 2021

elasticmachine added the Team:Delivery Meta label for Delivery team label Aug 30, 2021

DaveCTurner mentioned this issue Aug 30, 2021

[DOCS] Troubleshooting SIGSEGV on SELinux when elasticsearch home directory is missing #74545

Closed

pugnascotia self-assigned this Aug 31, 2021

This was referenced Aug 31, 2021

Use ES_TMPDIR instead of java.io.tmpdir #74163

Closed

Service failing to start RHEL 7 #77053

Closed

DaveCTurner added the stalled label Aug 31, 2021

DaveCTurner mentioned this issue Sep 21, 2021

Please upgrade libffi to ≥3.4.2 java-native-access/jna#1381

Closed

DaveCTurner mentioned this issue Nov 10, 2021

Upgrade to 5.10.0 elastic/jna-build#9

Merged

DaveCTurner closed this as completed in elastic/jna-build@e7c1c44 Nov 10, 2021

DaveCTurner reopened this Nov 10, 2021

DaveCTurner mentioned this issue Nov 10, 2021

Upgrade JNA to 5.10.0 #80617

Merged

droberts195 mentioned this issue Nov 10, 2021

elasticsearch fails to start when tmp directory defined #18406

Closed

DaveCTurner closed this as completed in #80617 Nov 11, 2021

DaveCTurner reopened this Nov 11, 2021

DaveCTurner mentioned this issue Nov 11, 2021

Set LIBFFI_TMPDIR at startup #80651

Merged

DaveCTurner closed this as completed in #80651 Nov 15, 2021

DaveCTurner mentioned this issue Nov 15, 2021

Set LIBFFI_TMPDIR at startup (#80651) #80698

Merged

DaveCTurner mentioned this issue Nov 15, 2021

Set LIBFFI_TMPDIR at startup (#80651) #80699

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LIBFFI_TMPDIR should point somewhere that permits executables #77014

LIBFFI_TMPDIR should point somewhere that permits executables #77014

DaveCTurner commented Aug 30, 2021 •

edited

Loading

elasticmachine commented Aug 30, 2021

pugnascotia commented Aug 30, 2021

DaveCTurner commented Aug 30, 2021

DaveCTurner commented Aug 31, 2021

mark-vieira commented Aug 31, 2021

DaveCTurner commented Sep 1, 2021 •

edited

Loading

DaveCTurner commented Nov 10, 2021

spricer commented Nov 10, 2021

DaveCTurner commented Nov 10, 2021

DaveCTurner commented Nov 11, 2021

spricer commented Nov 15, 2021

DaveCTurner commented Dec 9, 2021

LIBFFI_TMPDIR should point somewhere that permits executables #77014

LIBFFI_TMPDIR should point somewhere that permits executables #77014

Comments

DaveCTurner commented Aug 30, 2021 • edited Loading

elasticmachine commented Aug 30, 2021

pugnascotia commented Aug 30, 2021

DaveCTurner commented Aug 30, 2021

DaveCTurner commented Aug 31, 2021

mark-vieira commented Aug 31, 2021

DaveCTurner commented Sep 1, 2021 • edited Loading

DaveCTurner commented Nov 10, 2021

spricer commented Nov 10, 2021

DaveCTurner commented Nov 10, 2021

DaveCTurner commented Nov 11, 2021

spricer commented Nov 15, 2021

DaveCTurner commented Dec 9, 2021

DaveCTurner commented Aug 30, 2021 •

edited

Loading

DaveCTurner commented Sep 1, 2021 •

edited

Loading