-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add blogpost for etcd fuzzing audit #566
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
a262b3e
Add blogpost for etcd fuzzing audit
AdamKorcz eaffba2
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz 5102e75
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz ed4b240
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz 9384f32
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz 88b4516
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz 2bcce28
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz 73acb0e
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz 446a071
Update content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
AdamKorcz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
31 changes: 31 additions & 0 deletions
31
content/en/blog/2022/etcd-integrates-continuous-fuzzing.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
--- | ||
title: etcd Integrates Continuous Fuzzing | ||
spelling: cSpell:ignore Gyuho | ||
author: "[Adam Korczynski](https://twitter.com/AdamKorcz4), [David Korczynski](https://twitter.com/Davkorcz), [Sahdev Zala](https://twitter.com/sp_zala)" | ||
date: 2022-03-11 | ||
draft: false | ||
--- | ||
|
||
In the last few months, the team at [Ada Logics](https://adalogics.com) has worked on integrating continuous fuzzing into the etcd project. This was an effort focused on improving the security posture of etcd and ensuring a continued good experience for etcds users. The fuzzing integration involved enrolling etcd in the OSS-Fuzz project and writing a set of fuzzers that would bring the test coverage of etcd up to a mature level. In total, 18 fuzzers were written, and eight bugs were found, demonstrating the work’s value for etcd both short term and long term. All fuzzers were implemented by way of go-fuzz and when running in OSS-Fuzz instrumented by way of libFuzzer, and as such, etcd uses state-of-the-art open source fuzzing capabilities. | ||
The full report of the engagement can be found [here](https://github.com/etcd-io/etcd/blob/main/security/FUZZING_AUDIT_2022.PDF). | ||
|
||
The etcd project was created at CoreOS in 2013 and later joined the CNCF in 2018, from which it [graduated](https://www.cncf.io/announcements/2020/11/24/cloud-native-computing-foundation-announces-etcd-graduation/) in 2020. It is an open source, strongly consistent, distributed key-value store to reliably store data that a distributed system or cluster of machines needs to be accessed. It also provides highly desirable features like Watches to monitor changes to keys. etcd is a critical component of Kubernetes where it is used as the primary data store for cluster data such as the clusters state data and its data related to its desired state. Besides being a key component in Kubernetes, etcd is also used by [many other distributed systems](https://etcd.io/docs/v3.5/integrations/#projects-using-etcd). Because of its wide usage, etcd is a critical part of the open source ecosystem to fuzz for reliability bugs and security vulnerabilities. [The CNCF annual survey of 2021](https://www.cncf.io/reports/cncf-annual-survey-2021) found that 96% of companies are either using or evaluating Kubernetes, and etcd’s performance and security are important to continued business operations of these users. | ||
|
||
## What is fuzzing? | ||
|
||
Fuzzing is a technique used to automate parts of the software testing process by way of a form of stress testing. The key idea is to write a fuzzing harness similar to a unit—or integration—test that will execute the application under test with some arbitrary input. The fuzzing engine that will run the fuzzing harness then uses genetic algorithms to extrapolate inputs that will cause the code under test to execute uniquely, i.e., generate inputs that trigger new code execution paths. The goal is then to observe if the code under test misbehaves in the event of any of the generated inputs. Fuzzing has been effective in uncovering reliability bugs and vulnerabilities in software for more than two decades, and open source software is increasingly adopting the technique. | ||
|
||
## Etcd fuzzing overview | ||
|
||
In this engagement, the goal was to write a set of fuzzers that would cover a lot of the etcd codebase and integrate the setup into the open source fuzzing service OSS-Fuzz. OSS-Fuzz is a free service offered by Google for critical open source projects to run their fuzzers continuously and report any crashes. Continuous analysis is important due to fuzzing relying on genetic algorithms, which effectively means the fuzzers will improve over time, and OSS-Fuzz will run the fuzzers daily indefinitely. In addition to this, continuous analysis is crucial for capturing any regressions. | ||
|
||
Etcd is written in the Go programming language, making it safe from memory-corruptions. Fuzzing Go will find panics such as slice/index out of range, nil-pointer dereferences, invalid type assertions, timeouts, out of memory. At the end of this engagement, eight issues were found, all of which were fixed. They are broken down as such: | ||
|
||
![figure-1](../etcd-integrates-continuous-fuzzing/etcd-fuzzing-found-bugs.png "The fuzzing engagement found 2 nil-pointer dereference, 2 slice/index out of range, 2 panics from invalid utf-8 strings, and 2 type confusions.") | ||
|
||
At the end of this engagement, the fuzzers provide significant coverage of the etcd project, including critical parts such as the etcd server, WAL, the auth store, and the raft package. | ||
During the engagement, Ada Logics found that only a few of the critical parts of etcd would be accessible with a byte slice or string but instead accepted complex types such as structs. An example of this is [the fuzzers for the etcd server](https://github.com/cncf/cncf-fuzzing/blob/main/projects/etcd/etcdserver_fuzzer.go), which configures and sets up an etcd server and then creates a series of pseudo-randomized structs representing different requests sent to the server. To write these fuzzers, Ada Logics used [go-fuzz-headers](https://github.com/AdaLogics/go-fuzz-headers) to deterministically create pseudo-random structs from the data provided by libFuzzer. | ||
|
||
## Closing thoughts | ||
|
||
The etcd team is thankful to CNCF and Chris Aniszczyk for providing the opportunity to work with Ada Logics to develop new fuzzers for etcd. The software security is taken seriously by the CNCF, and it earlier funded the etcd project for a [third-party security audit](https://www.cncf.io/blog/2020/08/05/etcd-security-audit/). We also want to thank all the etcd maintainers and reviewers, especially Marek Siarkowicz, Piotr Tabor, and Benjamin Wang, for their quick reviews of the fixes. The fuzzing findings and fixes are valuable add-ons to the previous conclusions of the security audit. etcd project has efficient test suites, and code changes are backed by tests, but the newly developed fuzzers and findings have provided significant value to the project. During the fuzzing, only eight issues were found, which revalidated the high quality of the etcd code. The etcd team should maintain the newly developed fuzzers and build on them to continue code quality and security. |
Binary file added
BIN
+48.2 KB
...ent/en/blog/2022/etcd-integrates-continuous-fuzzing/etcd-fuzzing-found-bugs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need
spelling:
but it's okay, will remove it later if needed.