Fuzzers: from

John Regehr talk

Fuzzer Types

generative fuzzers
comes up with these things out of nowhere to try test inputs
mutation fuzzers
tweak an existing corpus to develop new examples.

The structure of programs is often very ill-defined and vague.

Fuzzing is useful because it allows you to rigorously build a justified sense of confidence that your code doesn't suck.

Code Differences

what's different about approaching the code?

for one compiler, the generated code could stack overflow. forces you to be super tight about catching corner cases earlier in the start rather than actually just putting up with them without the fuzzing testing. gets you to dig into these test cases more easily.

outgrowth of test driven development - it's about being able to define classes of test cases you care about and iterate on testing those cases

you want to look for anything - crashes, things going wrong, etc… as you're writing the code, you embed test oracles (assertions, primarily) within the code to dynamically check this! the weaker the language the more you need.

property testing and fuzzing are related; property testing is thought of as fuzzing with more oracles - it allows you to describe the properties, whereas fuzzing is thought of as more 'black box'

adjusting to languages with fewer guardrails, or type safety; ships the responsibility to the testing, mostly this is why gradual typing are so exciting! you can shift towards a fully typed environment and ensure that you add the types later, saving you a ton of the trouble of testing.

don't invest a lot of work into generating test cases if you already have a super rich corpus of text to have your fuzzer draw from

c had lots of undefined behavior, so a very strong property had to be enforced on the test code by csmith to ensure that the code generated was from a defined subset of the c specification.

need to impose structure on the testing - valid c++ is one of the most difficult file formats.generators are so hard to write for not much benefit!

what should be fuzzed? what doesn't matter? ideal workflow with fuzzing involved

don't fuzz if your code doesn't yet have the maturity to do it!

fuzzing is just weak verificatio: it's a weak quantifier over all possible inputs in the space of inputs. it's one of hte closest ways we can get to formal verification without the effort to do it! most things are mostly wrong and you need to fuzz them before proving them correct! only after fuzzing does verification make any sense. always fuzz with assertions, too, now have a decent specification for the system already after doing this fuzzing campaign!

almost everything he does is about people and making things better for people! great programming language trend towards interactivity and ways to work with you the way you want to work in code!

Insight that makes creduce works across languages - alcohol and lisp? shared syntax of block structure language - understands all of the braces at some rudimentary level. has tokenizatio npass based on c tokenizer, but it works half decently for other things?


DSL to specify what you care about - take advantage of what you care about in testing! in general this is an incredible view on this program

best advice for producing qulaity software, in general: use modern languages, test rigorously, write proper fuzzer generators


https://concuerror.com/: stateless model checker for erlang https://github.com/csmith-project/creduce/blob/master/creduce/creduce.in the creduce code https://www.carolemieux.com/rlcheck_preprint.pdf reinforcemenet learning for testing inputs!

https://gamozolabs.github.io/fuzzing/2020/12/06/fuzzos.html OS for fuzzing optimization?? https://embed.cs.utah.edu/csmith/ best fuzzer for C

https://www.fuzzingbook.org/html/Grammars.html fuzzing with grammars https://blog.trailofbits.com/2018/12/31/fuzzing-like-its-1989/ a decent fuzzing article delta debugging was mentioned as an interesting tactic to use

built for C, but works pretty well on lots of languages! The code can be found here
a fuzzer written in OCaml