Benchmark

• Main
• Test Cases
• Tool Support/Results
• Quick Start
• RoadMap
• Test Case Details/Challenges
• Acknowledgements

OWASP Benchmark Project The OWASP Benchmark for Security Automation (OWASP Benchmark) is a test suite designed to evaluate the speed, coverage, and accuracy of automated vulnerability detection tools and services (henceforth simply referred to as 'tools'). Without the ability to measure these tools, it is difficult to understand their strengths and weaknesses, and compare them to each other. The OWASP Benchmark contains over 20,000 test cases that are fully runnable and exploitable. You can use the OWASP Benchmark with Static Application Security Testing (SAST) tools, Dynamic Application Security Testing (DAST) tools like OWASP ZAP and Interactive Application Security Testing (IAST) tools. The current version of the Benchmark is implemented in Java. Future versions may expand to include other languages. Benchmark Project Philosophy Security tools (SAST, DAST, and IAST) are amazing when they find a complex vulnerability in your code. But with widespread misunderstanding of the specific vulnerabilities automated tools cover, end users are often left with a false sense of security. We are on a quest to measure just how good these tools are at discovering and properly diagnosing security problems in applications. We rely on the long history of military and medical evaluation of detection technology as a foundation for our research. Therefore, the test suite tests both real and fake vulnerabilities. There are four possible test outcomes in the Benchmark: Tool correctly identifies a real vulnerability (True Positive - TP) Tool fails to identify a real vulnerability (False Negative - FN) Tool correctly ignores a false alarm (True Negative - TN) Tool fails to ignore a false alarm (False Positive - FP) We can learn a lot about a tool from these four metrics. Consider a tool that simply flags every line of code as vulnerable. This tool will perfectly identify all vulnerabilities! But it will also have 100% false positives and thus adds no value. Similarly, consider a tool that reports absolutely nothing. This tool will have zero false positives, but will also identify zero real vulnerabilities and is also worthless. You can even imagine a tool that flips a coin to decide whether to report whether each test case contains a vulnerability. The result would be 50% true positives and 50% false positives. We need a way to distinguish valuable security tools from these trivial ones. If you imagine the line that connects all these points, from 0,0 to 100,100 establishes a line that roughly translates to "random guessing." The ultimate measure of a security tool is how much better it can do than this line. The diagram below shows how we will evaluate security tools against the Benchmark. Benchmark Validity The Benchmark tests are not exactly like real applications. The tests are derived from coding patterns observed in real applications, but the majority of them are considerably simpler than real applications. That is, most real world applications will be considerably harder to successfully analyze than the OWASP Benchmark Test Suite. Although the tests are based on real code, it is possible that some tests may have coding patterns that don't occur frequently in real code. Remember, we are trying to test the capabilities of the tools and make them explicit, so that users can make informed decisions about what tools to use, how to use them, and what results to expect. This is exactly aligned with the OWASP mission to make application security visible. Benchmark Scoring Anyone can use this Benchmark to evaluate their tools. The basic steps are: Download the Benchmark from github Run your tools against the Benchmark Run the BenchmarkScore tool on the reports from your tools That's it! We encourage both vendors, open source tools, and end users to verify their application security tools against the Benchmark. In order to ensure that the results are fair and useful, we ask that you follow a few simple rules when publishing results. We won't recognize any results that aren't easily reproducible: A description of the default “out-of-the-box” installation, version numbers, etc… Any and all configuration, tailoring, onboarding, etc… performed to make the tool run Any and all changes to default security rules, tests, or checks used to achieve the results Easily reproducible steps to run the tool Reporting Format The Benchmark includes tools to interpret raw tool output, compare it to the expected results, and generate summary charts and graphs. We use the following table format in order to capture all the information generated during the evaluation. Security Category TP FN TN FP Total TPR FPR Score General security category for test cases. True Positives: Tests with real vulnerabilities that were correctly reported as vulnerable by the tool. False Negative: Tests with real vulnerabilities that were not correctly reported as vulnerable by the tool. True Negative: Tests with fake vulnerabilities that were correctly not reported as vulnerable by the tool. False Positive:Tests with fake vulnerabilities that were incorrectly reported as vulnerable by the tool. Total test cases for this category. True Positive Rate: TP / ( TP + FN ) - Also referred to as Precision, as defined at Wikipedia. False Positive Rate: FP / ( FP + TN ). Normalized distance from the “guess line” TPR - FPR. Command Injection ... ... ... ... ... ... ... ... Etc... ... ... ... ... ... ... ... ... Total TP Total FN Total TN Total FP Total TC Average TPR Average FPR Average Score Code Repo and Build/Run Instructions See the Getting Started and Getting, Building, and Running the Benchmark sections on the Documentation tab. Licensing The OWASP Benchmark is free to use under the GNU General Public License v2.0. Mailing List OWASP Benchmark Mailing List Project Leaders Dave Wichers @ Project References Software Assurance Marketplace (SWAMP) - set of curated packages to test tools against SAMATE List of Test Collections Related Projects NSA's Juliet for Java WAVESEP

Quick Download All test code and project files can be downloaded from OWASP GitHub. News and Events Aug 27, 2015 - The U.S. Dept. of Homeland Security (DHS) is financially supporting the Benchmark project. Aug 15, 2015 - DAST Support with Benchmark Version 1.2beta Released. Checkmarx and ZAP scorecard generators also released. July 10, 2015 - Benchmark Scorecard generator and open source scorecards released May 23, 2015 - Benchmark Version 1.1 Released April 15, 2015 - Benchmark Version 1.0 Released Classifications GNU General Public License v2.0

Version 1.0 of the Benchmark was published on April 15, 2015 and had 20,983 test cases. On May 23, 2015, version 1.1 of the Benchmark was released. The 1.1 release improves on the previous version by making sure that there are both true positives and false positives in every vulnerability area. Version 1.2beta was released on August 15, 2015.

Version 1.2 and forward of the Benchmark is a fully executable web application, which means it is scannable by any kind of vulnerability detection tool. The 1.2beta has been limited to slightly less than 3,000 test cases, to make it easier for DAST tools to scan it (so it doesn't take so long and they don't run out of memory, or blow up the size of their database). The final 1.2 release is expected to be at least 5,000 or possibly 10,000 test cases, after we determine that the popular DAST scanners can handle that size. The 1.2beta release covers the same vulnerability areas that 1.1 covers. We added a few Spring database SQL Injection tests, but thats it. The bulk of the work was turning each test case into something that actually runs correctly AND is fully exploitable, and then generating a UI on top of it that works in order to turn the test cases into a real running application.

Given 1.2beta is temporary, we aren't updating the chart below. You can still download the version 1.1 release of the Benchmark by cloning the release marked with the GIT tag '1.1'.

The test case areas and quantities for the 1.1 release are:

Vulnerability Area	Number of Tests	CWE Number
Command Injection	2708	78
Weak Cryptography	1440	327
Weak Hashing	1421	328
LDAP Injection	736	90
Path Traversal	2630	22
Secure Cookie Flag	416	614
SQL Injection	3529	89
Trust Boundary Violation	725	501
Weak Randomness	3640	330
XPATH Injection	347	643
XSS (Cross-Site Scripting)	3449	79
Total Test Cases	21,041

Each Benchmark version comes with a spreadsheet that lists every test case, the vulnerability category, the CWE number, and the expected result (true finding/false positive). Look for the file: expectedresults-VERSION#.csv in the project root directory.

Every test case is:

• a servlet or JSP (currently they are all servlets, but we plan to add JSPs)
• either a true vulnerability or a false positive for a single issue

The benchmark is intended to help determine how well analysis tools correctly analyze a broad array of application and framework behavior, including:

• HTTP request and response problems
• Simple and complex data flow
• Simple and complex control flow
• Popular frameworks
• Inversion of control
• Reflection
• Class loading
• Annotations
• Popular UI technologies (particularly JavaScript frameworks)

Not all of these are yet tested by the Benchmark but future enhancements intend to provide more coverage of these issues.

Additional future enhancements could cover:

• All vulnerability types in the OWASP Top 10
• Does the tool find flaws in libraries?
• Does the tool find flaws spanning custom code and libraries?
• Does tool handle web services? REST, XML, GWT, etc…
• Does tool work with different app servers? Java platforms?

Example Test Case

Each test case is a simple Java EE servlet. BenchmarkTest00001 in version 1.0 of the Benchmark was an LDAP Injection test with the following metadata in the accompanying BenchmarkTest00001.xml file:

 <test-metadata>
   <category>ldapi</category>
   <test-number>00001</test-number>
   <vulnerability>true</vulnerability>
   <cwe>90</cwe>
 </test-metadata>

BenchmarkTest00001.java in the OWASP Benchmark 1.0 simply reads in all the cookie values, looks for a cookie named "foo", and uses the value of this cookie when performing an LDAP query. Here's the code for BenchmarkTest00001.java:

 package org.owasp.benchmark.testcode;
 
 import java.io.IOException;
 
 import javax.servlet.ServletException;
 import javax.servlet.annotation.WebServlet;
 import javax.servlet.http.HttpServlet;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 
 @WebServlet("/BenchmarkTest00001")
 public class BenchmarkTest00001 extends HttpServlet {
 	
 	private static final long serialVersionUID = 1L;
 	
 	@Override
 	public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
 		doPost(request, response);
 	}
 
 	@Override
 	public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
 		// some code
 
 		javax.servlet.http.Cookie[] cookies = request.getCookies();
 		
 		String param = null;
 		boolean foundit = false;
 		if (cookies != null) {
 			for (javax.servlet.http.Cookie cookie : cookies) {
 				if (cookie.getName().equals("foo")) {
 					param = cookie.getValue();
 					foundit = true;
 				}
 			}
 			if (!foundit) {
 				// no cookie found in collection
 				param = "";
 			}
 		} else {
 			// no cookies
 			param = "";
 		}
 		
 		try {
 			javax.naming.directory.DirContext dc = org.owasp.benchmark.helpers.Utils.getDirContext();
 			Object[] filterArgs = {"a","b"};
 			dc.search("name", param, filterArgs, new javax.naming.directory.SearchControls());
 		} catch (javax.naming.NamingException e) {
 			throw new ServletException(e);
 		}
 	}
 }

The results for 4 free tools, PMD, FIndBugs, FindBugs with the FindSecBugs plugin, and ZAP are now available here against version 1.2beta of the Benchmark: https://rawgit.com/OWASP/Benchmark/master/scorecard/OWASP_Benchmark_Home.html

We have computed results for all the following tools, but are prohibited from publicly releasing the results for any commercial tools due to their licensing restrictions against such publication.

The Benchmark can generate results for the following tools:

Free Dynamic Analysis Tools (DAST):

• Arachni
• OWASP ZAP

Commercial Dynamic Analysis Tools (DAST):

• Burp Pro
• IBM AppScan

If you have access to other DAST Tools, PLEASE RUN THEM FOR US against the Benchmark, and send us the results file so we can build a scorecard generator for that tool.

Free Static Analysis Tools (SAST):

• PMD (which really has no security rules)
• Findbugs
• FindBugs with the FindSecurityBugs plugin
• SonarQube

Note: We looked into supporting Checkstyle and Error Prone but neither of these free static analysis tools have any security rules, so they would both score all zeroes, just like PMD.

Commercial Static Analysis Tools (SAST):

• Checkmarx CxSAST
• Coverity Code Advisor (On-Demand and stand-alone versions)
• HP Fortify (On-Demand and stand-alone versions)
• IBM AppScan Source
• Parasoft Jtest
• Veracode SAST

We are looking for results for other commercial static analysis tools like: Grammatech CodeSonar, Klocwork, etc. If you have a license for any static analysis tool not already listed above and can run it on the Benchmark and send us the results file that would be very helpful.

The free SAST tools come bundled with the Benchmark so you can run them yourselves. If you have a license for any commercial SAST tool, you can also run them against the Benchmark. Just put your results files in the /results folder of the project, and then run the BenchmarkScore script for your platform (.sh / .bat) and it will generate a scorecard in the /scorecard directory for all the tools you have results for that are currently supported.

WARNING: If you generate results for a commercial tool, be careful who you distribute it to. Each tool has its own license defining when any results it produces can be released/made public. It is likely to be against the terms of a commercial tool's license to publicly release that tool's score against the OWASP Benchmark. The OWASP Benchmark project takes no responsibility if someone else releases such results. It is for just this reason that the Benchmark project isn't releasing such results itself.

The project has automated test harnesses for these vulnerability detection tools, so we can repeatably run the tools against each version of the Benchmark and automatically produce scorecards in our desired format.

We want to test as many tools as possible against the Benchmark. If you are:

• A tool vendor and want to participate in the project
• Someone who wants to help score a free tool agains the project
• Someone who has a license to a commercial tool and the terms of the license allow you to publish tool results, and you want to participate

please let me know!

What is in the Benchmark?

The Benchmark is a Java Maven project. Its primary component is thousands of test cases (e.g., BenchmarkTest00001.java) , each of which is a single Java servlet that contains a single vulnerability (either a true positive or false positive). The vulnerabilities span about a dozen different types currently and are expected to expand significantly in the future.

An expectedresults.csv is published with each version of the Benchmark (e.g., expectedresults-1.1.csv) and it specifically lists the expected results for each test case. Here’s what the first two rows in this file looks like for version 1.1 of the Benchmark:

# test name		category	real vulnerability	CWE	Benchmark version: 1.1	2015-05-22
BenchmarkTest00001	crypto		TRUE			327

This simply means that the first test case is a crypto test case (use of weak cryptographic algorithms), this is a real vulnerability (as opposed to a false positive), and this issue maps to CWE 327. It also indicates this expected results file is for Benchmark version 1.1 (produced May 22, 2015). There is a row in this file for each of the tens of thousands of test cases in the Benchmark. Each time a new version of the Benchmark is published, a new corresponding results file is generated and each test case can be completely different from one version to the next.

The Benchmark also comes with a bunch of different utilities, commands, and prepackaged open source security analysis tools, all of which can be executed through Maven goals, including:

• Open source vulnerability detection tools to be run against the Benchmark
• A scorecard generator, which computes a scorecard for each of the tools you have results files for.

What Can You Do With the Benchmark?

• Compile all the software in the Benchmark project (e.g., mvn compile)
• Run a static vulnerability analysis tool (SAST) against the Benchmark test case code

• Scan a running version of the Benchmark with a dynamic application security testing tool (DAST)
  • Instructions on how to run it are provided below

• Generate scorecards for each of the tools you have results files for
  • See the Tool Support/Results page for the list of tools the Benchmark supports generating scorecards for

Getting Started

Before downloading or using the Benchmark make sure you have the following installed and configured properly:

GIT: http://git-scm.com/ or https://github.com/
Maven: https://maven.apache.org/  (Version: 3.2.3 or newer works. We heard that 3.0.5 throws an error.)
Java: http://www.oracle.com/technetwork/java/javase/downloads/index.html (Java 7 or 8) (64-bit) - Takes ALOT of memory to compile the Benchmark.

Getting, Building, and Running the Benchmark

To download and build everything:

$ git clone https://github.com/OWASP/benchmark 
$ cd benchmark
$ mvn compile   (This compiles it)
$ runBenchmark.sh/.bat - This compiles and runs it.

Then navigate to: https://localhost:8443/benchmark/ to go to its home page. It uses a self signed SSL certificate, so you'll get a security warning when you hit the home page.

Note: We have set the Benchmark app to use up to 6 Gig of RAM, which it may need when it is fully scanned by a DAST scanner. The DAST tool probably also requires 3+ Gig of RAM. As such, we recommend having a 16 Gig machine if you are going to try to run a full DAST scan. And at least 4 or ideally 8 Gig if you are going to play around with the running Benchmark app.

Running Free Static Analysis Tools Against the Benchmark

There are scripts for running each of the free SAST vulnerability detection tools included with the Benchmark against the Benchmark test cases. On Linux, you might have to make them executable (e.g., chmod 755 *.sh) before you can run them.

Generating Test Results for PMD:

$ runPMD.sh (Linux) or runPMD.bat (Windows)

Generating Test Results for FindBugs:

$ runFindBugs.sh (Linux) or runFindBugs.bat (Windows)

Generating Test Results for FindBugs with the FindSecBugs plugin:

$ runFindSecBugs.sh (Linux) or runFindSecBugs.bat (Windows)

In each case, the script will generate a results file and put it in the /results directory. For example:

Benchmark_1.1-findbugs-v3.0.1-1026.xml

This results file name is carefully constructed to mean the following: It's a results file against the OWASP Benchmark version 1.1, FindBugs was the analysis tool, it was version 3.0.1 of FindBugs, and it took 1026 seconds to run the analysis.

NOTE: If you create a results file yourself, by running a commercial tool for example, you can add the version # and the compute time to the filename just like this and the Benchmark Scorecard generator will pick this information up and include it in the generated scorecard. If you don't, depending on what metadata is included in the tool results, the Scorecard generator might do this automatically anyway.

Generating Scorecards

The application BenchmarkScore is included with the Benchmark. It parses the output files generated by security tools run against the Benchmark and compares them against the expected results, and produces a set of web pages that detail the accuracy and speed of the tools involved.

The following command will compute a BenchmarkScore for all the results files in the /results directory:

createScorecard.sh (Linux) or createScorecard.bat (Windows)

We recommend including the Benchmark version number in any results file name, in order to help prevent mismatches between the expected results and the actual results files. A tool will not score well against the wrong expected results.

WARNING: If you generate results for a commercial tool, be careful who you distribute it to. Each tool has its own license defining when any results it produces can be released/made public. It is likely to be against the terms of a commercial tool's license to publicly release that tool's score against the OWASP Benchmark. The OWASP Benchmark project takes no responsibility if someone else releases such results. It is for just this reason that the Benchmark project isn't releasing such results itself.

Benchmark v1.0 - Released April 15, 2015 - This initial release included over 20,000 test cases in 11 different vulnerability categories. As this initial version was not a runnable application, it was only suitable for assessing static analysis tools (SAST).

Benchmark v1.1 - Released May 23, 2015 - This update fixed some inaccurate test cases, and made sure that every vulnerability area included both True Positives and False Positives.

Benchmark Scorecard Generator - Released July 10, 2015 - The ability to automatically and repeatably produce a scorecard of how well tools do against the Benchmark was released for most of the SAST tools supported by the Benchmark. Scorecards present graphical as well as statistical data on how well a tool does against the Benchmark down to the level of detail of how exactly it did against each individual test in the Benchmark. The latest public scorecards are available at: [1] Support for producing scorecards for additional tools is being added frequently and the current full set is documented on the Tool Results and Documentation tabs of this wiki.

Benchmark v1.2beta - Released Jul 10, 2015 - The 1st release of a fully runnable version of the Benchmark to support assessing all types of vulnerability detection and prevention technologies, including DAST, IAST, RASP, WAFs, etc. (This version has been updated with minor fixes numerous times since its initial release). This involved creating a user interface for every test case, and enhancing each test case to make sure its actually exploitable, not just uses something that is theoretically weak. This release is under 3,000 test cases to make it practical to scan the entire Benchmark with a DAST tool in a reasonable amount of time, with commodity hardware specs. The next release is expected to be larger, but we aren't sure how large. Maybe more like 5,000 or even 10,000 test cases.

Next Release --> Benchmark 1.2 - We are currently working with a number of DAST tool developers, getting their feedback on the size, accuracy, and other characteristics of the Benchmark and making agreed to improvements. Once we get a sufficient amount of performance and accuracy data we'll be able to decide what size the Benchmark should be to support DAST and SAST assessments and release a final 1.2 version. We'll then have to rerun all the tools against this version to get a baseline set of results to compare against each other.

Post Benchmark 1.2 steps:

While we don't have hard and fast rules of exactly what we are going to do next, enhancements in the following areas are goals for the project:

• Add new vulnerability categories
• Add support for popular server side Java frameworks (since the Benchmark is written in Java)
• Add support for popular client side JavaScript frameworks
• Add support for scorecard generation for additional tools (a continuous goal)

We also plan to do a bunch of data analysis on the results in order to see if we can learn and report some interesting things about the strengths and weaknesses of vulnerability analysis tools of all different types.

General Challenges

Tool Configuration Required

All tools support various levels of configuration in order to improve their results. The Benchmark project, in general, is trying to compare out of the box capabilities of tools. However, if a few simple tweaks to a tool can be done to improve that tool's score, that's fine. We'd like to understand what those simple tweaks are, and document them here, so others can repeat those tests in exactly the same way. For example, just turn on the 'test cookies and headers' flag, which is off by default. Or turn on the 'advanced' scan, so it will work harder, find more vulnerabilities. Its simple things like this we are talking about, not an extensive effort to teach the tool about the app, or perform 'expert' configuration of the tool.

So, if you know of some simple tweaks to improve a tool's results, let us know what they are and we'll document them here so everyone can benefit and make it easier to do apples to apples comparisons.

Cookies as a Source of Attack for XSS

Benchmark v1.1 and early versions of the 1.2beta included test cases that used cookies as a source of data that flowed into XSS vulnerabilities. The Benchmark treated these tests as False Positives because the Benchmark team figured that you'd have to use an XSS vulnerability in the first place to set the cookie value, and so it wasn't fair/reasonable to consider an XSS vulnerability whose data source was a cookie value as actually exploitable. However, we got feedback from some tool vendors, like Fortify, Burp, and Arachni, that they disagreed with this analysis and felt that, in fact, cookies were a valid source of attack against XSS vulnerabilities. Given that there are good arguments on both sides of this safe vs. unsafe question, we decided on Aug 25, 2015 to simply remove those test cases from the Benchmark. If, in the future, we decide who is right, we may add such test cases back in.

Headers as a Source of Attack for XSS

Similarly, the Benchmark team believes that the names of headers aren't a valid source of XSS attack for the same reason we thought cookie values aren't a valid source. Because it would require an XSS vulnerability to be exploited in the first place to set them. In fact, we feel that this argument is much stronger for header names, than cookie values. Right now, the Benchmark doesn't include any header names as sources for XSS test cases, but we plan to add them, and mark them as false positive in the Benchmark.

We do have header values as sources for some XSS test cases in the Benchmark and we treat those as valid XSS sources (i.e., true positives) because of headers the attacker could cause to be set, like the Referer or possibly the User-Agent. We could imagine the Benchmark being more specific about exactly which header is considered a valid XSS source, but for now, we treat all of them as valid XSS sources.

Unintentional Vulnerabilities

We are sure there are vulnerabilities we didn't intend to be there and we are eliminating them as we find them. If you find some, let us know and we'll fix them too. We are primarily focused on unintentional vulnerabilities in the categories of vulnerabilities the Benchmark currently supports, since that is what is actually measured.

Right now, two types of vulnerabilities that get reported are ignored:

1. Vulnerabilities in categories not yet supported
2. Vulnerabilities of a type that is supported, but reported in test cases not of that type

In the case of #2, false positives reported in unexpected areas are also ignored, which is primarily a DAST problem. Right now those false positives are completely ignored, but we are thinking about including them in the false positive score in some fashion. We just haven't decided how yet.

DAST Scanning Challenges

Benchmark Generates Lots of Cookies

The Burp team pointed out a cookies bug in the 1.2beta Benchmark. Each Weak Randomness test case generates its own cookie, 1 per test case. This caused the creation of so many cookies that servers would eventually start returning 400 errors because there were simply too many cookies being submitted in a request. This was fixed in the Aug 27, 2015 update to the Benchmark by setting the path attribute for each of these cookies to be the path to that individual test case. Now, only at most one of these cookies should be submitted with each request, eliminating this 'too many cookies' problem. However, if a DAST tool doesn't honor this path attribute, it may continue to send too many cookies, making the Benchmark unscannable for that tool. The Burp team acknowledges that Burp Pro currently has this limitation so it can't successfully scan the full Benchmark at this time. It is on the Burp team's roadmap to address this deficiency in the future. Other DAST tools may have similar problems with these test cases.

The Benchmark is a BIG Application

Yes. It is, so you might have to give your scanner more memory than it normally uses by default in order to successfully scan the entire Benchmark. Please consult your tool vendor's documentation on how to give it more memory.

Your machine itself might not have enough memory in the first place. For example, we were not able to successfully scan the 1.2beta with OWASP ZAP with only 8 Gig of RAM. So, you might need a more powerful machine or use a cloud provided machine to successfully scan the Benchmark with certain DAST tools. You may have similar problems with SAST tools against large versions of the Benchmark, like the 1.1 release.

SAST Scanning Challenges

Static Values Passed to Unsafe (Weak) Sinks

The Benchmark has MANY test cases where unsafe data flows in from the browser, but that data is replaced with static content as it goes through the propagators in the that specific test case. This static (safe) data then flows to the sink, which may be a weak/unsafe sink, like, for example, an unsafely constructed SQL statement. The Benchmark treats those test cases as false positives because there is absolutely no way for that weakness to be exploited. The NSA Juliet SAST benchmark treats such test cases exactly the same way, as false positives. We do recognize that there are weaknesses in those test cases, even though they aren't exploitable.

Some SAST tool vendors feel it is appropriate to point out those weaknesses, and that's fine. However, if the tool points those weaknesses out, and does not distinguish them from truly exploitable vulnerabilities, then the Benchmark treats those findings as false positives. If the tool allows a user to differentiate these non-exploitable weaknesses from exploitable vulnerabilities, then the Benchmark scorecard generator can use that information to filter out these extra findings (along with any other similarly marked findings) so they don't count against that tool when calculating that tool's Benchmark score. In the real world, its important for analysts to be able to filter out such findings if they only have time to deal with the most critical, actually exploitable, vulnerabilities. If a tool doesn't make it easy for an analyst to distinguish the two situations, then they are providing a disservice to the analyst.

This issue doesn't affect DAST tools. They only report what appears to be exploitable to them. So this has no affect on them.

If you are a SAST tool vendor or user, and you believe the Benchmark scorecard generator is counting such findings against that tool, and there is a way to tell them apart, please let the project team know so the scorecard generator can be adjusted to not count those findings against the tool. The Benchmark project's goal is the generate the most fair and accurate results it can generate. If such an adjustment is made to how a scorecard is generated for that tool, we plan to document this was done for that tool, and explain how others could perform the same filtering within that tool in order to get the same focused set of results.

Dead Code

Some SAST tools point out weaknesses in dead code because they might eventually end up being used, and serve as bad coding examples (think cut/paste of code). We think this is fine/appropriate. However, there is no dead code in the OWASP Benchmark (at least not intentionally). So dead code should not be causing any tool to report unnecessary false positives.

The following people and organizations, and many others, have contributed to this project and their contributions are much appreciated!

• Juan Gama - Development of initial release and continued support
• Ken Prole - Assistance with automated scorecard development using CodeDx
• Nick Sanidas - Development of initial release
• Denim Group - Contribution of scan results to facilitate scorecard development
• Tasos Laskos - Significant feedback on the DAST version of the Benchmark.
• PortSwigger - The Burp Suite team has provided us with important feedback on the DAST version of the Benchmark.
• DHS - The U.S. Department of Homeland Security, for providing financial support to this important project.
• Ann Campbell - From SonarSource - for fixing our SonarQube results parser

We are looking for volunteers. Please contact Dave Wichers if you are interested in contributing new test cases, tool results run against the benchmark, or anything else.