OWASP WebGoat Benchmark Edition (WBE) - Revision history

Wichers: Undo revision 222698 by Wichers (talk)

2016-10-25T21:19:19Z

Undo revision 222698 by Wichers (talk)

Wichers: Blanked the page

2016-10-25T21:16:55Z

Blanked the page

Wichers: Redirected page to OWASP Benchmark Project

2015-06-30T03:32:41Z

Redirected page to OWASP Benchmark Project

Wichers: Redirected page to Benchmark

2015-06-30T03:28:45Z

Redirected page to Benchmark

Show changes

Wichers at 03:26, 30 June 2015

2015-06-30T03:26:48Z

Wichers at 01:31, 30 June 2015

2015-06-30T01:31:17Z

Wichers: /* Project References */

2015-06-06T16:51:26Z

‎Project References

Wichers at 14:56, 1 June 2015

2015-06-01T14:56:15Z

Wichers: /* Test Cases */

2015-05-23T21:31:42Z

‎Test Cases

Wichers: /* News and Events */

2015-05-23T21:23:13Z

‎News and Events

← Older revision	Revision as of 21:19, 25 October 2016
Line 1:	Line 1:
−	+	#redirect [[Benchmark]]

← Older revision		Revision as of 21:16, 25 October 2016
Line 1:		Line 1:
−	~~#redirect [[OWASP_Benchmark_Project]]~~	+

← Older revision		Revision as of 03:32, 30 June 2015
Line 1:		Line 1:
−	#redirect [[~~Benchmark~~]]	+	#redirect [[OWASP_Benchmark_Project]]

@@ Line 6: / Line 6: @@
 == OWASP Benchmark Project ==
-The OWASP Benchmark for Security Automation (OWASP Benchmark) is a test suite designed to evaluate the speed, coverage, and accuracy of vulnerability detection tools. Without the ability to measure these tools, it is difficult to understand their value or interpret vendor claims. The OWASP Benchmark contains over 20,000 test cases that are fully runnable and exploitable.
+The OWASP Benchmark for Security Automation (OWASP Benchmark) is a test suite designed to evaluate the speed, coverage, and accuracy of automated vulnerability detection tools and services (henceforth simply referred to as 'tools'). Without the ability to measure these tools, it is difficult to understand their value or interpret vendor claims. The OWASP Benchmark contains over 20,000 test cases that are fully runnable and exploitable.
-You can use the OWASP Benchmark with Static Application Security Testing (SAST) tools. A future goal is to support the evaluation of Dynamic Application Security Testing (DAST) tools like OWASP [[ZAP]] and Interactive Application Security Testing (IAST) Tools. The current version is implemented in Java.  Future versions may expand to include other languages.
+You can use the OWASP Benchmark with Static Application Security Testing (SAST) tools. A future goal is to support the evaluation of Dynamic Application Security Testing (DAST) tools like OWASP [[ZAP]] and Interactive Application Security Testing (IAST) tools. The current version of the Benchmark is implemented in Java.  Future versions may expand to include other languages.
 ==Benchmark Project Philosophy==
@@ Line 18: / Line 18: @@
 There are four possible test outcomes in the Benchmark:
-# Tool/Service correctly identifies a real vulnerability (True Positive - TP)
+# Tool correctly identifies a real vulnerability (True Positive - TP)
-# Tool/Service fails to identify a real vulnerability (False Negative - FN)
+# Tool fails to identify a real vulnerability (False Negative - FN)
-# Tool/Service correctly ignores a false alarm (True Negative - TN)
+# Tool correctly ignores a false alarm (True Negative - TN)
-# Tool/Service fails to ignore a false alarm (False Positive - FP)
+# Tool fails to ignore a false alarm (False Positive - FP)
-We can learn a lot about a tool or service from these four metrics. A tool/service that simply flags every line of code as vulnerable will perfectly identify all vulnerabilities in an application, but will also have 100% false positives.  Similarly, a tool/service that reports nothing will have zero false positives, but will also identify zero real vulnerabilities.  Imagine a tool that flips a coin to decide whether to report each vulnerability for every test case. The result would be 50% true positives and 50% false positives.  We need a way to distinguish valuable security tools from these trivial ones.
+We can learn a lot about a tool from these four metrics. A tool that simply flags every line of code as vulnerable will perfectly identify all vulnerabilities in an application, but will also have 100% false positives.  Similarly, a tool that reports nothing will have zero false positives, but will also identify zero real vulnerabilities.  Imagine a tool that flips a coin to decide whether to report each vulnerability for every test case. The result would be 50% true positives and 50% false positives.  We need a way to distinguish valuable security tools from these trivial ones.
 If you imagine the line that connects all these points, from 0,0 to 100,100 establishes a line that roughly translates to "random guessing." The ultimate measure of a security tool is how much better it can do than this line.  The diagram below shows how we will evaluate security tools against the Benchmark.
@@ Line 37: / Line 37: @@
 ==Benchmark Scoring and Reporting Results==
-We encourage both vendors, open source tools, and end users to verify their application security tools using the WBE.  We encourage everyone to contribute their results to the project.  In order to ensure that the results are fair and useful, we ask that you follow a few simple rules when publishing results. We won't recognize any results that aren't easily reproducible.
+We encourage both vendors, open source tools, and end users to verify their application security tools against the Benchmark.  We encourage everyone to contribute their results to the project.  In order to ensure that the results are fair and useful, we ask that you follow a few simple rules when publishing results. We won't recognize any results that aren't easily reproducible.
 # Provide an easily reproducible procedure (script preferred) to run the tool on the Benchmark, including:
@@ Line 116: / Line 116: @@
 == Mailing List ==
-[https://lists.owasp.org/mailman/listinfo/owasp-webgoat-benchmark-project OWASP Benchmark Mailing List]
+[https://lists.owasp.org/mailman/listinfo/owasp-benchmark-project OWASP Benchmark Mailing List]
 == Project Leaders ==
@@ Line 160: / Line 160: @@
 = Test Cases =
-Version 1.0 of the WBE was published on April 15, 2015 and had 20,983 test cases. On May 23, 2015, version 1.1 of the WBE was released. The 1.1 release improves on the previous version by making sure that there are both true positives and false positives in every vulnerability area. The test case areas and quantities for the 1.1 release are:
+Version 1.0 of the Benchmark was published on April 15, 2015 and had 20,983 test cases. On May 23, 2015, version 1.1 of the Benchmark was released. The 1.1 release improves on the previous version by making sure that there are both true positives and false positives in every vulnerability area. The test case areas and quantities for the 1.1 release are:
 {| class="wikitable nowraplinks"
@@ Line 234: / Line 234: @@
 * Popular UI technologies (particularly JavaScript frameworks)
-Not all of these are yet tested by the WBE but future enhancements intend to provide more coverage of these issues.
+Not all of these are yet tested by the Benchmark but future enhancements intend to provide more coverage of these issues.
 Additional future enhancements could cover:
@@ Line 245: / Line 245: @@
 == Example Test Case ==
-Each test case is a simple Java EE servlet. BenchmarkTest00001 in version 1.0 of the WBE was an LDAP Injection test with the following metadata in the accompanying BenchmarkTest00001.xml file:
+Each test case is a simple Java EE servlet. BenchmarkTest00001 in version 1.0 of the Benchmark was an LDAP Injection test with the following metadata in the accompanying BenchmarkTest00001.xml file:
    <test-metadata>
@@ Line 254: / Line 254: @@
    </test-metadata>
-BenchmarkTest00001.java in WBE 1.0 simply reads in all the cookie values, looks for a cookie named "foo", and uses the value of this cookie when performing an LDAP query. Here's the code for BenchmarkTest00001.java:
+BenchmarkTest00001.java in the OWASP Benchmark 1.0 simply reads in all the cookie values, looks for a cookie named "foo", and uses the value of this cookie when performing an LDAP query. Here's the code for BenchmarkTest00001.java:
-   package org.owasp.webgoat.benchmark.testcode;
+   package org.owasp.benchmark.testcode;
    import java.io.IOException;
@@ Line 301: / Line 301: @@
    		try {
-   			javax.naming.directory.DirContext dc = org.owasp.webgoat.benchmark.helpers.Utils.getDirContext();
+   			javax.naming.directory.DirContext dc = org.owasp.benchmark.helpers.Utils.getDirContext();
    			Object[] filterArgs = {"a","b"};
    			dc.search("name", param, filterArgs, new javax.naming.directory.SearchControls());
@@ Line 316: / Line 316: @@
 Our vision for this project is that we will develop automated test harnesses for lots of vulnerability detection tools where we can repeatably run the tools against each version of the benchmark and automatically produce results in our desired format.
-We want to test the WBE against as many tools as possible. If you are:
+We want to test as many tools as possible against the Benchmark. If you are:
 * A tool vendor and want to participate in the project

@@ Line 4: / Line 4: @@
 | valign="top"  style="border-right: 1px dotted gray;padding-right:25px;" |
-== WebGoat Benchmark Edition ==
+== OWASP Benchmark Project ==
-The OWASP WebGoat Benchmark Edition (WBE) is a test suite designed to evaluate the speed, coverage, and accuracy of vulnerability detection tools. Without the ability to measure these tools, it is difficult to understand their value or interpret vendor claims. The WBE contains over 20,000 test cases that are fully runnable and exploitable.
+The OWASP Benchmark for Security Automation (OWASP Benchmark) is a test suite designed to evaluate the speed, coverage, and accuracy of vulnerability detection tools. Without the ability to measure these tools, it is difficult to understand their value or interpret vendor claims. The OWASP Benchmark contains over 20,000 test cases that are fully runnable and exploitable.
-You can use WBE with Static Application Security Testing (SAST) and Interactive Application Security Testing (IAST) tools. A future goal is to support the evaluation of Dynamic Application Security Testing (DAST) tools like OWASP [[ZAP]]. The current version is implemented in Java.  Future versions may expand to include other languages.
+You can use the OWASP Benchmark with Static Application Security Testing (SAST) tools. A future goal is to support the evaluation of Dynamic Application Security Testing (DAST) tools like OWASP [[ZAP]] and Interactive Application Security Testing (IAST) Tools. The current version is implemented in Java.  Future versions may expand to include other languages.
-==WBE Project Philosophy==
+==Benchmark Project Philosophy==
 Security tools (SAST, DAST, and IAST) are amazing when they find a complex vulnerability in your code.  But they can drive everyone crazy with complexity, false alarms, and missed vulnerabilities.  Using these tools without understanding their strengths and weaknesses can lead to a dangerous false sense of security.
@@ Line 16: / Line 16: @@
 We are on a quest to measure just how good these tools are at discovering and properly diagnosing security problems in applications. We rely on the [http://en.wikipedia.org/wiki/Receiver_operating_characteristic long history] of military and medical evaluation of detection technology as a foundation for our research. Therefore, the test suite tests both real and fake vulnerabilities.
-There are four possible test outcomes in the WBE:
+There are four possible test outcomes in the Benchmark:
-# Tool correctly identifies a real vulnerability (True Positive - TP)
+# Tool/Service correctly identifies a real vulnerability (True Positive - TP)
-# Tool fails to identify a real vulnerability (False Negative - FN)
+# Tool/Service fails to identify a real vulnerability (False Negative - FN)
-# Tool correctly ignores a false alarm (True Negative - TN)
+# Tool/Service correctly ignores a false alarm (True Negative - TN)
-# Tool fails to ignore a false alarm (False Positive - FP)
+# Tool/Service fails to ignore a false alarm (False Positive - FP)
-We can learn a lot about a tool from these four metrics. A tool that simply flags every line of code as vulnerable will perfectly identify all vulnerabilities in an application, but will also have 100% false positives.  Similarly, a tool that reports nothing will have zero false positives, but will also identify zero real vulnerabilities.  Imagine a tool that flips a coin to decide whether to report each vulnerability for every test case. The result would be 50% true positives and 50% false positives.  We need a way to distinguish valuable security tools from these trivial ones.
+We can learn a lot about a tool or service from these four metrics. A tool/service that simply flags every line of code as vulnerable will perfectly identify all vulnerabilities in an application, but will also have 100% false positives.  Similarly, a tool/service that reports nothing will have zero false positives, but will also identify zero real vulnerabilities.  Imagine a tool that flips a coin to decide whether to report each vulnerability for every test case. The result would be 50% true positives and 50% false positives.  We need a way to distinguish valuable security tools from these trivial ones.
-If you imagine the line that connects all these points, from 0,0 to 100,100 establishes a line that roughly translates to "random guessing." The ultimate measure of a security tool is how much better it can do than this line.  The diagram below shows how we will evaluate security tools against the WBE.
+If you imagine the line that connects all these points, from 0,0 to 100,100 establishes a line that roughly translates to "random guessing." The ultimate measure of a security tool is how much better it can do than this line.  The diagram below shows how we will evaluate security tools against the Benchmark.
 [[File:Wbe guide.png]]
-==WBE Validity==
+==Benchmark Validity==
-The WBE tests are not exactly like real applications. The tests are derived from coding patterns observed in real applications, but many of them are considerably simpler than real applications. Other tests may have coding patterns that don't occur frequently in real code.  It's best to imagine the WBE as a continuum of tests from very simple all the way up to pretty difficult.
+The Benchmark tests are not exactly like real applications. The tests are derived from coding patterns observed in real applications, but many of them are considerably simpler than real applications. Other tests may have coding patterns that don't occur frequently in real code.  It's best to imagine the Benchmark as a continuum of tests from very simple all the way up to pretty difficult.
 Remember, we are trying to test the capabilities of the tools and make them explicit, so that *users* can make informed decisions about what tools to use, how to use them, and what results to expect.  This is exactly aligned with the OWASP mission to make application security visible.
-==WBE Scoring and Reporting Results==
+==Benchmark Scoring and Reporting Results==
 We encourage both vendors, open source tools, and end users to verify their application security tools using the WBE.  We encourage everyone to contribute their results to the project.  In order to ensure that the results are fair and useful, we ask that you follow a few simple rules when publishing results. We won't recognize any results that aren't easily reproducible.
-# Provide an easily reproducible procedure (script preferred) to run the tool on the WBE, including:
+# Provide an easily reproducible procedure (script preferred) to run the tool on the Benchmark, including:
 ## A description of the default “out-of-the-box” installation, version numbers, etc…
 ## All configuration, tailoring, onboarding, etc… performed to make the tool run
@@ Line 112: / Line 112: @@
 ==Licensing==
-The OWASP WebGoat Benchmark is free to use under the [http://choosealicense.com/licenses/gpl-2.0/ GNU General Public License v2.0].
+The OWASP Benchmark is free to use under the [http://choosealicense.com/licenses/gpl-2.0/ GNU General Public License v2.0].
 == Mailing List ==
-[https://lists.owasp.org/mailman/listinfo/owasp-webgoat-benchmark-project OWASP WebGoat Benchmark Mailing List]
+[https://lists.owasp.org/mailman/listinfo/owasp-webgoat-benchmark-project OWASP Benchmark Mailing List]
 == Project Leaders ==
@@ Line 128: / Line 128: @@
 == Related Projects ==
 * [http://samate.nist.gov/SARD/testsuite.php NSA's Juliet for Java]
 * [https://code.google.com/p/wavsep/ WAVESEP]
@@ Line 140: / Line 139: @@
 == News and Events ==
-* April 15, 2015 - WBE Version 1.0 Released
+* April 15, 2015 - Benchmark Version 1.0 Released
-* May 23, 2015 - WBE Version 1.1 Released
+* May 23, 2015 - Benchmark Version 1.1 Released
 ==Classifications==

← Older revision		Revision as of 16:51, 6 June 2015
Line 123:		Line 123:

	== Project References ==		== Project References ==
		+	* [https://www.mir-swamp.org/#packages/public Software Assurance Marketplace (SWAMP) - set of curated packages to test tools against]
	* [http://samate.nist.gov/Other_Test_Collections.html SAMATE List of Test Collections]		* [http://samate.nist.gov/Other_Test_Collections.html SAMATE List of Test Collections]

← Older revision		Revision as of 14:56, 1 June 2015
Line 121:		Line 121:

	[https://www.owasp.org/index.php/User:Wichers Dave Wichers] [mailto:dave.wichers@owasp.org @]		[https://www.owasp.org/index.php/User:Wichers Dave Wichers] [mailto:dave.wichers@owasp.org @]
		+
		+	== Project References ==
		+	* [http://samate.nist.gov/Other_Test_Collections.html SAMATE List of Test Collections]

	== Related Projects ==		== Related Projects ==