This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit https://owasp.org

Difference between revisions of "Fuzzing"

From OWASP
Jump to: navigation, search
(Fuzzing tools (mostly open sourced))
m (Added BFuzz)
 
(86 intermediate revisions by 10 users not shown)
Line 1: Line 1:
''Fuzz testing'' or ''Fuzzing'' is a software testing technique, which basically consists in finding implementation bugs using malformed/semi-malformed data injection: it's a type of ''Black Box Testing'', which means testing a closed system on the comportemental point of view.
+
''Fuzz testing'' or ''Fuzzing'' is a Black Box software testing technique, which basically consists in finding implementation bugs using malformed/semi-malformed data injection in an automated fashion.
  
== A trivial example: ==
+
== A trivial example ==
  
Lets's say there's a integer variable in a program, which results of a user's choice between 3 questions. When the user picks one choice, the choice will be transmitted to the program under a variable, say 0, 1 or 2. Which makes 3 practical cases. But what if we transmit 3, or 255 ? We can, because integers are stored in values which allow them to take a lot of cases (often more than necessary).
+
Lets's consider an integer in a program, which stores the result of a user's choice between 3 questions. When the user picks one, the choice will be 0, 1 or 2. Which makes three practical cases. But what if we transmit 3, or 255 ? We can, because integers are stored a static size variable. If the default switch case hasn't been implemented securely, the program may crash and lead to "classical" security issues: (un)exploitable buffer overflows, DoS, ...
  
If the programmer was lazy, he may have taken no/less attention to the default switch case.
+
Fuzzing is the art of automatic bug finding, and it's role is to find software implementation faults, and identify them if possible.
  
If so, the program may crash, which leads to "classical" security issues: (un)exploitable buffer overflows, simple DoS, ...
+
== History ==
  
In these perspectives, Fuzzing is the "art" of automatically finding this type of implementation failures, eventually being able to measure the found vulnerability dangerosity, and of course locate it (in order to exploit it).
+
Fuzz testing was developed at the University of Wisconsin Madison in 1989 by Professor Barton Miller and his students. Their (continued) work can be found at http://www.cs.wisc.edu/~bart/fuzz/ ; it's mainly oriented towards command-line and UI fuzzing, and shows that modern operating systems are vulnerable to even simple fuzzing.
  
 
== Fuzzer implementations ==
 
== Fuzzer implementations ==
  
A fuzzer is a program which will inject semi-random data into a program/stack and detect problems.
+
A fuzzer is a program which injects automatically semi-random data into a program/stack and detect bugs.
  
The data-generation part is made of ''generators'', and the error detection relies on debugging tools.
+
The data-generation part is made of generators, and vulnerability identification relies on debugging tools. Generators usually use combinations of static fuzzing vectors (known-to-be-dangerous values), or totally random data. New generation fuzzers use genetic algorithms to link injected data and observed impact. Such tools are not public yet.
  
A quite "efficient" fuzzer is:
+
== Comparison with cryptanalysis ==
  
- protocol/file-format dependant
+
The number of possible tryable solutions is ''the explorable solutions space''. The aim of cryptanalysis is to reduce this space, which means finding a way of having less keys to try than pure bruteforce to decrypt something.
  
- data-type dependant
+
Most of the fuzzers are:
  
Why?
+
* protocol/file-format dependant
Because a program only understands structured-enough data. If you connect to a web server in a raw way, it will only respond to listed commands such as GET (or eventually crash). If you try totally random inputs, it will in most cases respond with a "non-implemented method" error.
 
  
Take a 10 chars long string, supposed to fuzz a webserver: we generate all the possible strings doable within this string. It makes a lot of possibilities, right? What are the odds that you find the ones that start with "GET " ?
+
* data-type dependant
  
That's why a fuzzer has to understand at least the basics of a protocol / file format.
+
Why?
  
== About explorable solutions space: comparison with Cryptanalysis ==
+
* First, because the fuzzer has to connect to the input channel, which is bound to the target.
  
The number of possible tryable solutions is ''the explorable solutions space''.  
+
* Second, because a program only understands structured-enough data. If you connect to a web server in a raw way, it will only respond to listed commands such as GET (or eventually crash). It will take less time to start the string with "GET ", and fuzz the rest, but the drawback is that you'll skip all the tests on the first verb.
  
The aim of Cryptanalysis is to reduce this so called space, which means finding a way of having less keys to try than pure bruteforce to decrypt something.  
+
In this regard, Fuzzers try to reduce the number of unuseful tests, i.e. the values we already know that there's little chance they'll work: you reduce unpredictability, in favor of speed.
  
In this regard, Fuzzers try to reduce the number of unuseful tests: the values we already know that here's little chance they'll work.  
+
== Attack types ==
 +
 
 +
A fuzzer would try combinations of attacks on:
 +
 
 +
* numbers (signed/unsigned integers/float...)
 +
 
 +
* chars (urls, command-line inputs)
 +
 
 +
* metadata : user-input text (id3 tag)
 +
 
 +
* pure binary sequences
 +
 
 +
A common approach to fuzzing is to define lists of "known-to-be-dangerous values" (''fuzz vectors'') for each type, and to inject them or recombinations.
 +
 
 +
* for integers: zero, possibly negative or very big numbers
 +
 
 +
* for chars: escaped, interpretable characters / instructions (ex: For SQL Requests, quotes / commands...)
 +
 
 +
* for binary: random ones
 +
 
 +
Please refer to [[OWASP_Testing_Guide_Appendix_C:_Fuzz_Vectors|OWASP's Fuzz Vector's resource]] for real-life fuzzing vectors examples and methodology.
 +
 
 +
Protocols and file formats imply norms, which are sometimes blurry, very complicated or badly implemented : that's why developers sometimes mess up in the implementation process (because of time/cost constraints). That's why it can be interesting to take the opposite approach: take a norm, look at all mandatory features and constraints, and try all of them; forbidden/reserved values, linked parameters, field sizes. That would be ''conformance testing oriented'' fuzzing.
 +
 
 +
== Application fuzzing ==
 +
 
 +
Whatever the fuzzed system is, the attack vectors are within it's I/O. For a desktop app:
 +
 
 +
* the UI (testing all the buttons sequences / text inputs)
 +
 
 +
* the command-line options
 +
 
 +
* the import/export capabilities (see file format fuzzing below)
 +
 
 +
For a web app: urls, forms, user-generated content, RPC requests, ...
 +
 
 +
== Protocol fuzzing ==
 +
 
 +
A protocol fuzzer sends forged packets to the tested application, or eventually acts as a proxy, modifying requests on the fly and replaying them.
 +
 
 +
== File format fuzzing ==
 +
 
 +
A file format fuzzer generates multiple malformed samples, and opens them sequentially. When the program crashes, debug information is kept for further investigation.
 +
 
 +
One can attack:
 +
 
 +
* the parser layer (container layer): file format constraints, structure, conventions, field sizes, flags, ...
  
There's a big difference with Crypto though: the more "intelligent" your fuzzer is, the less weird errors it will find: you reduce impredictibility, in favor of speed. That's a compromise.
+
* the codec/application layer: lower-level attacks, aiming at the program's deeper internals
  
== Attack types ==
+
One example of file format related vulnerabilities: [http://www.microsoft.com/technet/security/bulletin/ms04-028.mspx MS04-028] (KB833987) Microsoft's JPEG GDI+ vulnerability was a zero sized comment field, without content.
  
A fuzzer would try combinations of attacks on:
+
Surprisingly, file format fuzzers are not that common, but tend to appear these days; some examples:
  
- numbers (signed/unsigned integers/float...)
+
* A generic file format fuzzer : [http://www.digitaldwarf.be/products/mangle.c Ilja van Sprundel's mangle.c]; "it's usage is very simple, it takes a filename and headersize as input. it will then change approximatly between 0 and 10% of the header with random bytes." (from the author)
  
- metadata : text-valuable information (mostly text)
+
* Zzuf can act as a fuzzed file generator, http://sam.zoy.org/zzuf/
  
- the protocol/file format itself (field size attacks...)
+
* One may use tools like [http://hachoir.org/ Hachoir] as a generic parser for file format fuzzer development.
  
A quick way (and common approach) to fuzzing is, for each types of data defined here, to define lists of "known-to-be-dangerous values" (''fuzz vectors''), and to inject them in place of the classical values, see [http://www.owasp.org/index.php/OWASP_Testing_Guide_Appendix_C:_Fuzz_Vectors] for real-life examples.
+
== Fuzzers advantages ==
  
Protocols and file formats imply norms, which are sometimes blurry, very complicated or badly implemented : that's why developers sometimes mess up in the implementation process (because of time/cost constraints).  
+
''The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior'' (from Wikipedia http://en.wikipedia.org/wiki/Fuzz_testing).
  
Take a norm, look at all mandatory features and constraints, and try all of them: forbidden/reserved values, linked parameters, field sizes (a lot of implementations are said not to verify field sizes). That would be ''hand-fuzzing''.
+
The systematical/random approach allows this method to find bugs that would have often been missed by human eyes. Plus, when the tested system is totally closed (say, a SIP phone), fuzzing is one of the only means of reviewing it's quality.
  
 
== Fuzzers limitations ==
 
== Fuzzers limitations ==
  
Fuzzers usually tend to try one-level-imbrication-level attacks, which means changing only one parameter at a time. A highly intelligent fuzzer would detect/know linked parameters, and play on them.
+
Fuzzers usually tend to find simple bugs; plus, the more a fuzzer is protocol-aware, the less weird errors it will find. This is why the exhaustive / random approach is still popular among the fuzzing community.
 +
 
 +
Another problem is that when you do some black-box-testing, you usually attack a closed system, which increases difficulty to evaluate the dangerosity/impact of the found vulnerability (no debugging possibilities).
  
Fuzzing tools can detect trivial errors quite easily, but are less gifted with complex ones (in terms of imbrication level).
+
== Why Fuzz? ==
 +
 
 +
The purpose of fuzzing relies on the assumption that there are bugs within every program, which are waiting to be discovered. Therefore, a systematical approach should find them sooner or later.
 +
 
 +
Fuzzing can add another point of view to classical software testing techniques (hand code review, debugging) because of it's non-human approach. It doesn't replace them,  but is a reasonable complement, thanks to the limited work needed to put the procedure in place.
 +
 
 +
Recent fuzzing initiatives:
 +
 
 +
* The [http://projects.info-pull.com/mokb/ Month of Kernel Bugs], which revealed an Apple Wireless flaw; see http://kernelfun.blogspot.com/; mainly used file system fuzzing tools
 +
 
 +
* The [http://browserfun.blogspot.com/ Month of Browser Bugs] ([http://osvdb.org/blog/?p=127 review here]); number of bugs found: MSIE: 25 Apple Safari: 2 Mozilla: 2 Opera: 1 Konqueror: 1; used DHTML, Css, DOM, ActiveX fuzzing tools
 +
 
 +
== Fuzzers from OWASP ==
 +
 
 +
Fuzzing with [[WebScarab]]: a framework for analysing applications that communicate using the HTTP and HTTPS protocols
 +
 
 +
[[JBroFuzz]]: a web application fuzzer
 +
 
 +
[[WSFuzzer]]: real-world manual SOAP pen testing tool
  
 
== Technical resources on OWASP  ==
 
== Technical resources on OWASP  ==
  
Fuzzing vectors [http://www.owasp.org/index.php/OWASP_Testing_Guide_Appendix_C:_Fuzz_Vectors]
+
[[OWASP Testing Guide Appendix C: Fuzz Vectors]]
 +
 
 +
== References ==
 +
 
 +
Wikipedia article - http://en.wikipedia.org/wiki/Fuzz_testing
 +
 
 +
Fuzzing-related papers - http://www.threatmind.net/secwiki/FuzzingPapers
 +
 
 +
[https://web.archive.org/web/20090202043027/http://www.whitestar.linuxbox.org/pipermail/fuzzing/ The fuzzing mailing list archive]
 +
 
 +
== Fuzzing tools ==
 +
 
 +
=== Open Source ===
 +
 
 +
==== Mutational Fuzzers ====
 +
[https://en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer) american fuzzy lop]
 +
 
 +
[https://github.com/aoh/radamsa Radamsa - a flock of fuzzers]
 +
 
 +
==== Fuzzing Frameworks ====
 +
[https://github.com/OpenRCE/sulley Sulley Fuzzing Framework]
 +
 
 +
[https://github.com/jtpereyda/boofuzz boofuzz]
 +
 
 +
[https://github.com/RootUp/BFuzz BFuzz]
 +
 
 +
==== Domain-Specific Fuzzers ====
 +
 
 +
[http://www.microsoft.com/download/en/details.aspx?id=21769 Microsoft SDL MiniFuzz File Fuzzer]
  
== Sources ==
+
[http://www.microsoft.com/download/en/details.aspx?id=20095 Microsoft SDL Regex Fuzzer]
  
Wikipedia article [http://en.wikipedia.org/wiki/Fuzz_testing]
+
[https://github.com/nradov/abnffuzzer ABNF Fuzzer]
  
Fuzzing-related papers [http://www.threatmind.net/secwiki/FuzzingPapers]
+
=== Commercial products ===
  
== Fuzzing tools (mostly open sourced) ==
+
[http://www.codenomicon.com/products/all.shtml Codenomicon's product suite]
  
The ultimate fuzzers list @ infosec [http://www.infosecinstitute.com/blog/2005/12/fuzzers-ultimate-list.html]
+
[http://peachfuzzer.com Peach Fuzzing Platform]
  
Another list @ hacksafe
+
[http://www.spirent.com/Products/AvalancheNEXT Spirent Avalanche NEXT]
[http://www.hacksafe.com.au/blog/2006/08/21/fuzz-testing-tools-and-techniques/]
 
  
Hachoir, a generic parser: could be quite a good starting point [http://www.hachoir.org]
+
[http://www.beyondsecurity.com/bestorm_overview.html Beyond Security's beSTORM product]
 +
[[Category:Externally Linked Page]]
  
== Commercial products ==
+
=== Deprecated Tools ===
  
Codenomicon's product suite: [http://www.codenomicon.com/products/all.shtml]
+
[http://www.bonsai-sec.com/en/research/untidy-xml-fuzzer.php Untidy - XML Fuzzer] (Now integrated into Peach)

Latest revision as of 14:27, 9 May 2018

Fuzz testing or Fuzzing is a Black Box software testing technique, which basically consists in finding implementation bugs using malformed/semi-malformed data injection in an automated fashion.

A trivial example

Lets's consider an integer in a program, which stores the result of a user's choice between 3 questions. When the user picks one, the choice will be 0, 1 or 2. Which makes three practical cases. But what if we transmit 3, or 255 ? We can, because integers are stored a static size variable. If the default switch case hasn't been implemented securely, the program may crash and lead to "classical" security issues: (un)exploitable buffer overflows, DoS, ...

Fuzzing is the art of automatic bug finding, and it's role is to find software implementation faults, and identify them if possible.

History

Fuzz testing was developed at the University of Wisconsin Madison in 1989 by Professor Barton Miller and his students. Their (continued) work can be found at http://www.cs.wisc.edu/~bart/fuzz/ ; it's mainly oriented towards command-line and UI fuzzing, and shows that modern operating systems are vulnerable to even simple fuzzing.

Fuzzer implementations

A fuzzer is a program which injects automatically semi-random data into a program/stack and detect bugs.

The data-generation part is made of generators, and vulnerability identification relies on debugging tools. Generators usually use combinations of static fuzzing vectors (known-to-be-dangerous values), or totally random data. New generation fuzzers use genetic algorithms to link injected data and observed impact. Such tools are not public yet.

Comparison with cryptanalysis

The number of possible tryable solutions is the explorable solutions space. The aim of cryptanalysis is to reduce this space, which means finding a way of having less keys to try than pure bruteforce to decrypt something.

Most of the fuzzers are:

  • protocol/file-format dependant
  • data-type dependant

Why?

  • First, because the fuzzer has to connect to the input channel, which is bound to the target.
  • Second, because a program only understands structured-enough data. If you connect to a web server in a raw way, it will only respond to listed commands such as GET (or eventually crash). It will take less time to start the string with "GET ", and fuzz the rest, but the drawback is that you'll skip all the tests on the first verb.

In this regard, Fuzzers try to reduce the number of unuseful tests, i.e. the values we already know that there's little chance they'll work: you reduce unpredictability, in favor of speed.

Attack types

A fuzzer would try combinations of attacks on:

  • numbers (signed/unsigned integers/float...)
  • chars (urls, command-line inputs)
  • metadata : user-input text (id3 tag)
  • pure binary sequences

A common approach to fuzzing is to define lists of "known-to-be-dangerous values" (fuzz vectors) for each type, and to inject them or recombinations.

  • for integers: zero, possibly negative or very big numbers
  • for chars: escaped, interpretable characters / instructions (ex: For SQL Requests, quotes / commands...)
  • for binary: random ones

Please refer to OWASP's Fuzz Vector's resource for real-life fuzzing vectors examples and methodology.

Protocols and file formats imply norms, which are sometimes blurry, very complicated or badly implemented : that's why developers sometimes mess up in the implementation process (because of time/cost constraints). That's why it can be interesting to take the opposite approach: take a norm, look at all mandatory features and constraints, and try all of them; forbidden/reserved values, linked parameters, field sizes. That would be conformance testing oriented fuzzing.

Application fuzzing

Whatever the fuzzed system is, the attack vectors are within it's I/O. For a desktop app:

  • the UI (testing all the buttons sequences / text inputs)
  • the command-line options
  • the import/export capabilities (see file format fuzzing below)

For a web app: urls, forms, user-generated content, RPC requests, ...

Protocol fuzzing

A protocol fuzzer sends forged packets to the tested application, or eventually acts as a proxy, modifying requests on the fly and replaying them.

File format fuzzing

A file format fuzzer generates multiple malformed samples, and opens them sequentially. When the program crashes, debug information is kept for further investigation.

One can attack:

  • the parser layer (container layer): file format constraints, structure, conventions, field sizes, flags, ...
  • the codec/application layer: lower-level attacks, aiming at the program's deeper internals

One example of file format related vulnerabilities: MS04-028 (KB833987) Microsoft's JPEG GDI+ vulnerability was a zero sized comment field, without content.

Surprisingly, file format fuzzers are not that common, but tend to appear these days; some examples:

  • A generic file format fuzzer : Ilja van Sprundel's mangle.c; "it's usage is very simple, it takes a filename and headersize as input. it will then change approximatly between 0 and 10% of the header with random bytes." (from the author)
  • One may use tools like Hachoir as a generic parser for file format fuzzer development.

Fuzzers advantages

The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior (from Wikipedia http://en.wikipedia.org/wiki/Fuzz_testing).

The systematical/random approach allows this method to find bugs that would have often been missed by human eyes. Plus, when the tested system is totally closed (say, a SIP phone), fuzzing is one of the only means of reviewing it's quality.

Fuzzers limitations

Fuzzers usually tend to find simple bugs; plus, the more a fuzzer is protocol-aware, the less weird errors it will find. This is why the exhaustive / random approach is still popular among the fuzzing community.

Another problem is that when you do some black-box-testing, you usually attack a closed system, which increases difficulty to evaluate the dangerosity/impact of the found vulnerability (no debugging possibilities).

Why Fuzz?

The purpose of fuzzing relies on the assumption that there are bugs within every program, which are waiting to be discovered. Therefore, a systematical approach should find them sooner or later.

Fuzzing can add another point of view to classical software testing techniques (hand code review, debugging) because of it's non-human approach. It doesn't replace them, but is a reasonable complement, thanks to the limited work needed to put the procedure in place.

Recent fuzzing initiatives:

  • The Month of Browser Bugs (review here); number of bugs found: MSIE: 25 Apple Safari: 2 Mozilla: 2 Opera: 1 Konqueror: 1; used DHTML, Css, DOM, ActiveX fuzzing tools

Fuzzers from OWASP

Fuzzing with WebScarab: a framework for analysing applications that communicate using the HTTP and HTTPS protocols

JBroFuzz: a web application fuzzer

WSFuzzer: real-world manual SOAP pen testing tool

Technical resources on OWASP

OWASP Testing Guide Appendix C: Fuzz Vectors

References

Wikipedia article - http://en.wikipedia.org/wiki/Fuzz_testing

Fuzzing-related papers - http://www.threatmind.net/secwiki/FuzzingPapers

The fuzzing mailing list archive

Fuzzing tools

Open Source

Mutational Fuzzers

american fuzzy lop

Radamsa - a flock of fuzzers

Fuzzing Frameworks

Sulley Fuzzing Framework

boofuzz

BFuzz

Domain-Specific Fuzzers

Microsoft SDL MiniFuzz File Fuzzer

Microsoft SDL Regex Fuzzer

ABNF Fuzzer

Commercial products

Codenomicon's product suite

Peach Fuzzing Platform

Spirent Avalanche NEXT

Beyond Security's beSTORM product

Deprecated Tools

Untidy - XML Fuzzer (Now integrated into Peach)