[RFC] Add static analysis to CI to determine redundant seccomp rules #4924
+657
−204
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roypat
added
the
Status: Awaiting review
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4924 +/- ##
=======================================
Coverage 84.07% 84.07%
=======================================
Files 251 251
Lines 28067 28067
=======================================
Hits 23597 23597
Misses 4470 4470
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
These have been determined by static analysis of a Firecracker binary
(see also follow up commits): The removed seccomp rules here trigger
syscalls that are either not present at all in the entire Firecracker
binary, or are not reachable from the entry point of a specific
Firecracker thread (this analysis has only been done for the vcpu thread
for now, due to being fairly tricky).
Some explanations for why some of these entries are no longer needed can
be found below
- The `uname` syscall was used back when we supported 4.14, to query the
host kernel version and disable specific Firecracker features that
were not supported pre-5.10 (io_uring and hugepages for memfd). With
4.14 support dropped, there are no such checks anymore.
- Various KVM_SET_* ioctls do not need to be allowed on the vcpu thread,
because they are all called _before_ the vcpu seccomp filters are
installed (as they are only used during snapshot restore / when
preparing KVM state for boot).
- on aarch64, we can additionally remove KVM_{SET,GET}VCPU_EVENTS, as we
never call this ioctl on arm (only on x86)
Signed-off-by: Patrick Roy <[email protected]>
roypat
force-pushed
the
seccomp-cleanup
branch
2 times, most recently
from
9dd9379
to
25ba8a0
Compare
Since Firecracker uses the musl target, it is statically compiled against all of its dependencies. This means that all syscalls that Firecracker can possibly do, are represented by `syscall`/`svc #0x0` instructions inside its binary. By doing some primitive static analysis, it is possible to determine, for each of these instructions, what the actual syscall number is (the idea being: it will probably be a constant loaded into a specific register, determined by Linux's syscall convention, very near the instruction itself). On a best-effort basis, we can also try to determine the values of all registers that might holds syscall arguments (for things like ioctls, this works very well, because the ioctl number is almost always a constant loaded very closely to the syscall instruction). The static_analysis.py file implements the logic for this analysis. We essentially start at each syscall instruction, and then work through the preceding instructions in reverse order, trying to symbolically execute the program in reverse until we figure out the value of the register that holds the syscall number. Thankfully, because of the above mentioned locally, this means we only need to really understand a handful of instructions (based on the assumption that we can ignore instructions that do not explicitly refer to registers that we care about). For example, on x86_64, the syscall number is stored in the eax register, so if we encounter the following instruction sequence xor %ebx,%ebx mov %ebx,%eax syscall we first symbolically execute the "mov", and determine that to figure out the syscall number, we actually need to determine the value of the ebx register. So we go back one more instruction, and find out that %ebx is explicitly set to 0. Thus, forward propagating this 0 to the syscall instruction again, we find that we're doing a `read` syscall. Things get a bit more complicated on ARM, where we don't have to only care about register-to-register transfers, but also things like `add` instructions. For these, we must also keep a "log" of manipulations to apply during forward propagation. For example, on ARM the syscall number is passed in the w8 register, so in the following sequence mov w19, #0x6 add w8, w19, #0x1 svc #0x0 we determine that to compute the syscall number, we must first determine the value of the w19 register, but also we must remember that when we forward propagate the value 0x6 from the w19 register, we must add 1 to it when propagating into the w8 register. Sometimes, we additionally have to deal with wrapper functions around syscalls (e.g. `libc::syscall`). For these, we need to translate the list of syscall number and syscall arguments that the wrapper received according to to the architectures C calling convention (cdecl on x86_64, w0-w7 for arguments on ARM) into what the wrapper will eventually pass to the syscall instruction. On x86_64 this comes with the complication that cdecl only allows passing 6 arguments in registers, but syscall nr + all arguments are up to 7 values. Thus, when a syscall is invoked via libc::syscall, we can only determine up to 5 syscall arguments (the final one would be on the stack, and stack analysis would significantly complicate this script). Lastly, we sometimes also have wrappers for specific syscalls (e.g. ioctl). Here, we do something very similar to the above, however since now the syscall number is "implicit" (e.g. not an argument to the wrapper function, but rather hard-coded into the wrapper), we no not actually run into problems with argument counts on x86_64. We implement this logic because it allows us to determine more arguments (for example, if we did not explicitly look at callsites of the ioctl wrapper, we would not be able to determine which ioctl numbers are actually invoked, greatly reducing the usefulness, as our seccomp filters are large ioctls). After determining all the syscalls, we then compare it to our seccomp filters. We look at all our allowed rules, and compute the set of rules that does not match _any_ of the actual syscalls invoked by the binary. If any such are found, we fail inside a new integration test. There are fairly big limitations to this approach. Because we're doing a global analysis of the firecracker binary, we cannot differentiate between syscalls on different threads, and also not between syscalls done before and after actual application of seccomp filters. For this, we would need some sort of control flow graph analysis, which is made significantly harder due to the presence of dynamic dispatch. Nevertheless, this simple approach already showed 4 redundant syscall rules (and playing around with applying the script to reduced compilation units showed a lot more unneeded rules for the vcpu thread). For all of this to work, we need to compile Firecracker as a non-position independent executable (to get the compiler to general direct calls with absolute addresses, instead of instruction pointer relative ones). Since all dependencies also need to be compiled this way, we have to use cargo's unstable `-Zbuild-std` feature, which thus means we need a nightly toolchain. Signed-off-by: Patrick Roy <[email protected]>
Add the rust-src component and the $(uname -m)-unknown-linux-musl targets for the nightly toolchain, and install python3-seccomp and rustfilt. Since the python bindings for libseccomp are not published to pip, we have to install it into the global python installation via apt-get, and then copy into our venv. Signed-off-by: Patrick Roy <[email protected]>
this commit should be dropped once we have rebuilt the devctr. Signed-off-by: Patrick Roy <[email protected]>
roypat
force-pushed
the
seccomp-cleanup
branch
from
25ba8a0
to
13f4cf4
Compare
|
Mhh, it works on ARM with the newest nightly, maybe related to rust-lang issue 125619? maybe need to part this one for a while then |
roypat
added
Status: Parked
ShadowCurse
reviewed
Nov 25, 2024
Comment on lines
+448
to
+450
| disassembly = subprocess.check_output(["rustfilt"], stdin=p_objdump.stdout).decode( | ||
| "utf-8" | ||
| ) |
There was a problem hiding this comment.
objdump -C rust should do the trick as well.
ShadowCurse
reviewed
Nov 25, 2024
Comment on lines
+1
to
+2
| # Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 |
There was a problem hiding this comment.
Can this be a rust binary instead of python script? Smth that takes binary and a seccomp json as inputs and tells if rules are correct?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use static analysis to determine the set of syscalls (+ arguments) that are actually present in the Firecracker binary, and cross-reference this with the seccomp filters, to determine redundant seccomp rules. Additionally, remove redundant seccomp rules identified by this process. Please see the individual commit messages for details.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.