Fuzzer. The competing fuzzer must link against a libFuzzer-style function called
int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size);
which accepts an array Data of size Size. The objective of the fuzzer is to generate as many unique crashes as possible. The fuzzer can leverage any type of feedback from the executed program as guidance toward this objective.
For our competition, we use Google’s FuzzBench fuzzer evaluation platform [ESEC/FSE’21]. FuzzBench provides an easy API for integrating fuzzers to contribute to painless, rigorous fuzzing research evaluations against a set of benchmark programs selected from real-world open-source projects and a set of state-of-the-art fuzzers. FuzzBench allows conducting experiments locally and publicly. Once an experiment is concluded, it generates an evaluation report with graphs and statistical tests to measure and visualize the effect size and statistical significance of the comparison across the chosen fuzzers and benchmark programs.
Fuzzer Integration. From the perspective of the fuzzer, FuzzBench provides generic access to:
LLVMFuzzerTestOneInput into executions of the programs,Related work
Registration (by 20 January).
\documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options) describing the technology and ideas behind the fuzzer you want to submit to the competition. The tool report should be sent to sbft23fuzzcomp@googlegroups.com as part of the registration. Please format your email subject as ‘[Report submission] {Fuzzer Name}’.Preliminaries. First, we encourage every participant to get familiar with the FuzzBench infrastructure. Create a private fork of the FuzzBench repository. Install prerequisites. You can now run one fuzzer (e.g., AFL) on one program (e.g., libpng-1.2.56) using make run-afl-libpng-1.2.56. Get familiar with fuzzer integration to integrate your own fuzzer into FuzzBench. Get familiar with the local setup to run multiple fuzzers, including yours, on multiple programs and get coverage and bug-finding reports.
Preparation. The participants are encouraged to use the programs and infrastructure available at FuzzBench for their local evaluation. The participants can submit new fuzzers, or extend existing fuzzers with their own novel ideas. The participants are allowed to modify the compilation of the programs to inject execution feedback for the fuzzer. During development, regularly make sure that make presubmit succeeds and it can build on benchmark programs. It can be quite pedantic and you don’t want to be failed due to trivial errors.
Pull Request Deadline (on 17 February 2023).
The participants publicly integrate their fuzzers into FuzzBench by submitting a Pull Request and getting it approved for integration. We will only consider fuzzers that passed all CI tests in the PR and that run for at least 30 minutes.
The primary contact of the participants’ team submits a short tool report (up to 2 pages in IEEE conference format; \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options) describing the technology and ideas behind the submitted fuzzer and the experience of integrating into FuzzBench.
Results. Finally, the winners will be announced live during the workshop in May.
Top-performing participants will be eligible to be considered for a new OSS-Fuzz FuzzBench integration reward (up to $11,337, depending on impact).
The fuzzing competition is organized jointly by Abhishek Arya (Google), Oliver Chang (Google), Jonathan Metzman (Google), Dongge Liu (Google), and Marcel Böhme (MPI-SP).
Abhishek Arya, Oliver Chang, Jonathan Metzman, and Dongge Liu are with the Google Open Source Security Team (GOSST). The team works to improve the security of open-source software that Google and the world rely on. The team’s FuzzBench provides a free service for researchers to rigorously evaluate fuzzers on a wide variety of real-world benchmarks at Google’s scale. The team’s ClusterFuzz and OSS-Fuzz projects combine modern fuzzing techniques with scalable, distributed execution to make widely used open source software more secure and stable. GOSST also contributes to hardening the open-source supply chain with SLSA, Sigstore, and projects in collaboration with the Open Source Security Foundation.
Marcel Böhme leads the Software Security research group at the Max Planck Institute for Security and Privacy (MPI-SP) in Germany. His current research interest is the automatic discovery of security flaws at a very large scale. One part of his group develops the probabilistic foundations of automatic software testing (i.e., finding bugs by generating executions) to elucidate fundamental limitations of existing techniques and to explore the assurances that software testing provides when no bugs are found. The other part of his group develops practical vulnerability discovery tools that are widely used in software security practice.