|
|
| (15 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | = Main =
| + | #redirect [[Benchmark]] |
| − | <div style="width:100%;height:100px;border:0,margin:0;overflow: hidden;">[[File:Incubator_big.jpg|link=OWASP_Project_Stages#tab=Incubator_Projects]]</div>
| |
| − | {| style="padding: 0;margin:0;margin-top:10px;text-align:left;" |-
| |
| − | | valign="top" style="border-right: 1px dotted gray;padding-right:25px;" |
| |
| − | | |
| − | == WebGoat Benchmark Edition ==
| |
| − | | |
| − | The OWASP WebGoat Benchmark Edition (WBE) is a test suite designed to evaluate the speed, coverage, and accuracy of vulnerability detection tools. Without the ability to measure these tools, it is difficult to understand their value or interpret vendor claims. The WBE contains over 20,000 test cases that are fully runnable and exploitable.
| |
| − | | |
| − | You can use WBE with Static Application Security Testing (SAST) and Interactive Application Security Testing (IAST) tools. A future goal is to support the evaluation of Dynamic Application Security Testing (DAST) tools like OWASP [[ZAP]]. The current version is implemented in Java. Future versions may expand to include other languages.
| |
| − | | |
| − | ==Project Philosophy==
| |
| − | | |
| − | Security tools (SAST, DAST, and IAST) are amazing when they find a complex vulnerability in your code. But they can drive everyone crazy with complexity, false alarms, and missed vulnerabilities. Using these tools without understanding their strengths and weaknesses can lead to a dangerous false sense of security.
| |
| − | | |
| − | We are on a quest to measure just how good these tools are at discovering and properly diagnosing security problems in applications. We rely on the [http://en.wikipedia.org/wiki/Receiver_operating_characteristic long history] of military and medical evaluation of detection technology as a foundation for our research. Therefore, the test suite tests both real and fake vulnerabilities.
| |
| − | | |
| − | There are four possible test outcomes in the WBE:
| |
| − | | |
| − | # Tool correctly identifies a real vulnerability (True Positive - TP)
| |
| − | # Tool fails to identify a real vulnerability (False Negative - FN)
| |
| − | # Tool correctly ignores a false alarm (True Negative - TN)
| |
| − | # Tool fails to ignore a false alarm (False Positive - FP) | |
| − | | |
| − | We can learn a lot about a tool from these four metrics. A tool that simply flags every line of code as vulnerable will perfectly identify all vulnerabilities in an application, but will also have 100% false positives. Similarly, a tool that reports nothing will have zero false positives, but will also identify zero real vulnerabilities. Imagine a tool that flips a coin to decide whether to report each vulnerability for every test case. The result would be 50% true positives and 50% false positives. We need a way to distinguish valuable security tools from these trivial ones.
| |
| − | | |
| − | If you imagine the line that connects all these points, from 0,0 to 100,100 establishes a line that roughly translates to "random guessing." The ultimate measure of a security tool is how much better it can do than this line. The diagram below shows how we will evaluate security tools against the WBE.
| |
| − | | |
| − | [[File:Wbe guide.png]] | |
| − | | |
| − | We recognize that the WBE test cases are not as complex as real code. These tests are designed to test various capabilities of security tools, but the results for real code may be different or worse than the performance on the WBE. We also recognize that different tools may be strong in some areas and weak in others. That's why we don't simply report a single score for each tool, but provide details about how they perform in each category. If you are curious about whether tools can detect certain issues, please submit examples of code so that we can improve the fidelity of the WBE.
| |
| − | | |
| − | ==Scoring and Reporting Results==
| |
| − | | |
| − | We encourage both vendors, open source tools, and end users to verify their application security tools using the WBE. We encourage everyone to contribute their results to the project. In order to ensure that the results are fair and useful, we ask that you follow a few simple rules when publishing results.
| |
| − | | |
| − | # Provide an easily reproducible procedure (script preferred) to run the tool on the WBE, including:
| |
| − | ## A description of the default “out-of-the-box” installation, version numbers, etc…
| |
| − | ## All configuration, tailoring, onboarding, etc… performed to make the tool run
| |
| − | ## All changes to default security rules, tests, or checks to achieve the results
| |
| − | ## Easily reproducible steps for achieving the result
| |
| − | # Results should be in the following table format
| |
| − |
| |
| − | {| class="wikitable nowraplinks"
| |
| − | |-
| |
| − | ! Security Category
| |
| − | ! TP
| |
| − | ! FN
| |
| − | ! TN
| |
| − | ! FP
| |
| − | ! Total
| |
| − | ! TPR
| |
| − | ! FPR
| |
| − | ! Score
| |
| − | |-
| |
| − | | valign="top" | General security category for test cases
| |
| − | | valign="top" | '''True Positives''': Tests with real vulnerabilities that were correctly reported as vulnerable by the tool
| |
| − | | valign="top" | '''False Negative''': Tests with real vulnerabilities that were not correctly reported as vulnerable by the tool
| |
| − | | valign="top" | '''True Negative''': Tests with fake vulnerabilities that were correctly not reported as vulnerable by the tool
| |
| − | | valign="top" | '''False Positive''':Tests with fake vulnerabilities that were incorrectly reported as vulnerable by the tool
| |
| − | | valign="top" | Total test cases for this category
| |
| − | | valign="top" width="250pt" | '''True Positive Rate''': TP / ( TP + FN )
| |
| − | | valign="top" width="250pt" | '''False Positive Rate''': FP / ( FP + TN )
| |
| − | | valign="top" | Normalized distance from the “guess line” TPR - FPR
| |
| − | |}
| |
| − | | |
| − | ==Code Repo==
| |
| − | | |
| − | The code for this project is hosted at the [https://github.com/OWASP/webgoat-benchmark OWASP Git repository]. Along with the code comes a Maven pom.xml file so you can download all the dependencies and build the entire project with ease using Maven.
| |
| − | | |
| − | Using the pom, it should be easy to verify all the code compiles correctly. To download and build everything, if you already have git and maven installed, all you have to do is:
| |
| − | | |
| − | $ git clone https://github.com/OWASP/webgoat-benchmark
| |
| − | $ cd webgoat-benchmark
| |
| − | $ mvn compile
| |
| − | | |
| − | ==Licensing==
| |
| − | | |
| − | The OWASP WebGoat Benchmark is free to use under the [http://choosealicense.com/licenses/gpl-2.0/ GNU General Public License v2.0].
| |
| − | | |
| − | == Mailing List ==
| |
| − | | |
| − | [https://lists.owasp.org/mailman/listinfo/owasp-webgoat-benchmark-project OWASP WebGoat Benchmark Mailing List]
| |
| − | | |
| − | == Project Leaders ==
| |
| − | | |
| − | [https://www.owasp.org/index.php/User:Wichers Dave Wichers] [mailto:[email protected] @] | |
| − | | |
| − | == Related Projects ==
| |
| − | | |
| − | * [[WebGoat]]
| |
| − | * [http://samate.nist.gov/SARD/testsuite.php NSA's Juliet for Java]
| |
| − | * [https://code.google.com/p/wavsep/ WAVESEP]
| |
| − | | |
| − | | valign="top" style="padding-left:25px;width:200px;" |
| |
| − | | |
| − | == Quick Download ==
| |
| − | | |
| − | All test code and project files can be downloaded from [https://github.com/OWASP/webgoat-benchmark OWASP GitHub].
| |
| − | | |
| − | == News and Events ==
| |
| − | | |
| − | * April 15, 2015 - First Version Released
| |
| − | | |
| − | ==Classifications==
| |
| − | | |
| − | {| width="200" cellpadding="2"
| |
| − | |-
| |
| − | | align="center" valign="top" width="50%" rowspan="2"| [[File:Owasp-incubator-trans-85.png|link=https://www.owasp.org/index.php/OWASP_Project_Stages#tab=Incubator_Projects]]
| |
| − | | align="center" valign="top" width="50%"| [[File:Owasp-builders-small.png|link=]]
| |
| − | |-
| |
| − | | align="center" valign="top" width="50%"| [[File:Owasp-defenders-small.png|link=]]
| |
| − | |-
| |
| − | | colspan="2" align="center" | [http://choosealicense.com/licenses/gpl-2.0/ GNU General Public License v2.0]
| |
| − | |-
| |
| − | | colspan="2" align="center" | [[File:Project_Type_Files_CODE.jpg|link=]]
| |
| − | |}
| |
| − | | |
| − | |}
| |
| − | | |
| − | = Test Cases =
| |
| − | | |
| − | This initial release of the WBE has 20,983 test cases. The test case areas and quantities for the April 15, 2015 release are:
| |
| − | | |
| − | {| class="wikitable nowraplinks"
| |
| − | |-
| |
| − | ! Vulnerability Area
| |
| − | ! Number of Tests
| |
| − | ! CWE Number
| |
| − | |-
| |
| − | | [[Command Injection]]
| |
| − | | 2902
| |
| − | | [https://cwe.mitre.org/data/definitions/78.html 78]
| |
| − | |-
| |
| − | | Weak Cryptography
| |
| − | | 668
| |
| − | | [https://cwe.mitre.org/data/definitions/327.html 327]
| |
| − | |-
| |
| − | | Weak Hashing
| |
| − | | 689
| |
| − | | [https://cwe.mitre.org/data/definitions/328.html 328]
| |
| − | |-
| |
| − | | Header Injection
| |
| − | | 1069
| |
| − | | [https://cwe.mitre.org/data/definitions/113.html 113]
| |
| − | |-
| |
| − | | [[LDAP injection | LDAP Injection]]
| |
| − | | 890
| |
| − | | [https://cwe.mitre.org/data/definitions/90.html 90]
| |
| − | |-
| |
| − | | [[Path Traversal]]
| |
| − | | 2974
| |
| − | | [https://cwe.mitre.org/data/definitions/22.html 22]
| |
| − | |-
| |
| − | | Secure Cookie Flag
| |
| − | | 450
| |
| − | | [https://cwe.mitre.org/data/definitions/614.html 614]
| |
| − | |-
| |
| − | | [[SQL Injection]]
| |
| − | | 4105
| |
| − | | [https://cwe.mitre.org/data/definitions/89.html 89]
| |
| − | |-
| |
| − | | [[Trust Boundary Violation]]
| |
| − | | 838
| |
| − | | [https://cwe.mitre.org/data/definitions/501.html 501]
| |
| − | |-
| |
| − | | Weak Randomness
| |
| − | | 1923
| |
| − | | [https://cwe.mitre.org/data/definitions/330.html 330]
| |
| − | |-
| |
| − | | [[XPATH Injection]]
| |
| − | | 436
| |
| − | | [https://cwe.mitre.org/data/definitions/643.html 643]
| |
| − | |-
| |
| − | | [[XSS]] (Cross-Site Scripting)
| |
| − | | 4039
| |
| − | | [https://cwe.mitre.org/data/definitions/79.html 79]
| |
| − | |-
| |
| − | | Total Test Cases
| |
| − | | 20,983
| |
| − | |}
| |
| − | | |
| − | To download a spreadsheet that lists every test case, the vulnerability category, the CWE number, and the expected result (true finding/false positive), click [https://github.com/OWASP/webgoat-benchmark/blob/master/benchmarktestcases.xls?raw=true here].
| |
| − | | |
| − | Every test case is:
| |
| − | * a servlet or JSP (currently they are all servlets, but we plan to add JSPs soon)
| |
| − | * either a true vulnerability or a false positive for a single issue
| |
| − | | |
| − | The tool analyzes a broad array of application and framework behavior:
| |
| − | | |
| − | * HTTP request and response problems?
| |
| − | * Simple and complex data flow?
| |
| − | * Simple and complex control flow?
| |
| − | * Popular frameworks?
| |
| − | * Inversion of control?
| |
| − | * Reflection?
| |
| − | * Class loading?
| |
| − | * Annotations?
| |
| − | * Popular UI technologies (particularly JavaScript frameworks)
| |
| − | | |
| − | Future enhancements could cover:
| |
| − | * All vulnerability types in the [[Top10 | OWASP Top 10]]
| |
| − | * Does the tool find flaws in libraries?
| |
| − | * Does the tool find flaws spanning custom code and libraries?
| |
| − | * Does tool handle web services? REST, XML, GWT, etc…
| |
| − | * Does tool work with different app servers? Java platforms?
| |
| − | | |
| − | == Example Test Case ==
| |
| − | | |
| − | Each test case is a simple Java EE servlet. BenchmarkTest00001 is an LDAP Injection test with the following metadata in the accompanying BenchmarkTest00001.xml file:
| |
| − | | |
| − | <test-metadata>
| |
| − | <category>ldapi</category>
| |
| − | <test-number>00001</test-number>
| |
| − | <vulnerability>true</vulnerability>
| |
| − | <cwe>90</cwe>
| |
| − | </test-metadata>
| |
| − | | |
| − | BenchmarkTest00001.java simply reads in all the cookie values, looks for a cookie named "foo" and uses the value of this cookie when performing an LDAP query. Here's the code for BenchmarkTest00001.java:
| |
| − | | |
| − | package org.owasp.webgoat.benchmark.testcode;
| |
| − |
| |
| − | import java.io.IOException;
| |
| − |
| |
| − | import javax.servlet.ServletException;
| |
| − | import javax.servlet.annotation.WebServlet;
| |
| − | import javax.servlet.http.HttpServlet;
| |
| − | import javax.servlet.http.HttpServletRequest;
| |
| − | import javax.servlet.http.HttpServletResponse;
| |
| − |
| |
| − | @WebServlet("/BenchmarkTest00001")
| |
| − | public class BenchmarkTest00001 extends HttpServlet {
| |
| − |
| |
| − | private static final long serialVersionUID = 1L;
| |
| − |
| |
| − | @Override
| |
| − | public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
| |
| − | doPost(request, response);
| |
| − | }
| |
| − |
| |
| − | @Override
| |
| − | public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
| |
| − | // some code
| |
| − |
| |
| − | javax.servlet.http.Cookie[] cookies = request.getCookies();
| |
| − |
| |
| − | String param = null;
| |
| − | boolean foundit = false;
| |
| − | if (cookies != null) {
| |
| − | for (javax.servlet.http.Cookie cookie : cookies) {
| |
| − | if (cookie.getName().equals("foo")) {
| |
| − | param = cookie.getValue();
| |
| − | foundit = true;
| |
| − | }
| |
| − | }
| |
| − | if (!foundit) {
| |
| − | // no cookie found in collection
| |
| − | param = "";
| |
| − | }
| |
| − | } else {
| |
| − | // no cookies
| |
| − | param = "";
| |
| − | }
| |
| − |
| |
| − | try {
| |
| − | javax.naming.directory.DirContext dc = org.owasp.webgoat.benchmark.helpers.Utils.getDirContext();
| |
| − | Object[] filterArgs = {"a","b"};
| |
| − | dc.search("name", param, filterArgs, new javax.naming.directory.SearchControls());
| |
| − | } catch (javax.naming.NamingException e) {
| |
| − | throw new ServletException(e);
| |
| − | }
| |
| − | }
| |
| − | }
| |
| − | | |
| − | = Tool Results =
| |
| − | | |
| − | As of this initial release, we don't have any vulnerability detection tool results to publish. We are working on generating results for Findbugs as our first example, and then plan to work on more after that. If you would like to contribute to this project by running a tool against the benchmark and producing a set of results in the format described in the --Scoring and Reporting Results-- section on the main project tab, please contact the project lead.
| |
| − | | |
| − | Our vision for this project is that we will develop automated test harnesses for lots of vulnerability detection tools where we can repeatably run the tools against each version of the benchmark and automatically produce results in our desired format.
| |
| − | | |
| − | We want to test the WBE against as many tools as possible. If you are:
| |
| − | | |
| − | * A tool vendor and want to participate in the project
| |
| − | * Someone who wants to help score a free tool agains the project
| |
| − | * Someone who has a license to a commercial tool and the terms of the license allow you to publish tool results, and you want to participate
| |
| − | | |
| − | | |
| − | | |
| − | = Acknowledgements =
| |
| − | | |
| − | The following people have contributed to this project and their contributions are much appreciated!
| |
| − | | |
| − | * Juan Gama - Development of initial release and continued support
| |
| − | * Ken Prole - Assistance with automated score card development using CodeDx
| |
| − | * Nick Sanidas - Development of initial release
| |
| − | | |
| − | We are looking for volunteers. Please contact [mailto:[email protected] Dave Wichers] if you are interested in contributing new test cases, tool results run against the benchmark, or anything else. | |
| − | | |
| − | __NOTOC__ <headertabs />
| |
| − | | |
| − | [[Category:OWASP_Project]]
| |
