Benchmark - Revision history

Jacqueline.king: Added Kiuwan to the tool scanning tips section

2019-12-20T03:05:29Z

Added Kiuwan to the tool scanning tips section

Wichers at 21:25, 2 November 2019

2019-11-02T21:25:34Z

Wichers: Add Semmle LGTM Scorecard Generation Support to Tools tab

2019-09-22T22:18:46Z

Add Semmle LGTM Scorecard Generation Support to Tools tab

Wichers at 19:20, 1 July 2019

2019-07-01T19:20:45Z

Wichers at 04:16, 12 December 2018

2018-12-12T04:16:35Z

Wichers: /* Using a VM instead */

2018-11-04T21:03:56Z

‎Using a VM instead

Wichers at 21:01, 4 November 2018

2018-11-04T21:01:22Z

Wichers: Update Docker VM creation to match scripts created to make it easier.

2018-11-04T18:24:36Z

Update Docker VM creation to match scripts created to make it easier.

Wichers: Xanitizer no longer free.

2018-10-03T20:33:18Z

Xanitizer no longer free.

Hblankenship at 13:59, 12 September 2018

2018-09-12T13:59:01Z

← Older revision		Revision as of 03:05, 20 December 2019
Line 637:		Line 637:

	[http://h3xstream.github.io/find-sec-bugs/ FindSecurityBugs] is a great plugin for FindBugs that significantly increases the ability for FindBugs to find security issues. We include this free tool in the Benchmark and its all dialed in. Simply run the script: ./script/runFindSecBugs.(sh or bat). If you want to run a different version of FindSecBugs, just change the version number of the findsecbugs-plugin artifact in the Benchmark pom.xml file.		[http://h3xstream.github.io/find-sec-bugs/ FindSecurityBugs] is a great plugin for FindBugs that significantly increases the ability for FindBugs to find security issues. We include this free tool in the Benchmark and its all dialed in. Simply run the script: ./script/runFindSecBugs.(sh or bat). If you want to run a different version of FindSecBugs, just change the version number of the findsecbugs-plugin artifact in the Benchmark pom.xml file.
		+
		+	=== Kiuwan Code Security ===
		+
		+	Kiuwan Code Security includes a predefined model for executing the OWASP benchmark. Refer to the [https://www.kiuwan.com/blog/owasp-benchmark-diy/ step-by-step instructions] on the Kiuwan website.

	=== Micro Focus (Formally HP) Fortify ===		=== Micro Focus (Formally HP) Fortify ===

@@ Line 407: / Line 407: @@
 * [https://www.ibm.com/us-en/marketplace/ibm-appscan-source IBM AppScan Source (Standalone and Cloud)] - .ozasmt or .xml results file
 * [https://juliasoft.com/solutions/julia-for-security/ Julia Analyzer] - .xml results file
 * [https://software.microfocus.com/en-us/products/static-code-analysis-sast/overview Micro Focus (Formally HPE) Fortify (On-Demand and stand-alone versions)] - .fpr results file
 * [https://www.parasoft.com/products/jtest/ Parasoft Jtest] - .xml results file

@@ Line 409: / Line 409: @@
 * [https://software.microfocus.com/en-us/products/static-code-analysis-sast/overview Micro Focus (Formally HPE) Fortify (On-Demand and stand-alone versions)] - .fpr results file
 * [https://www.parasoft.com/products/jtest/ Parasoft Jtest] - .xml results file
 * [https://www.shiftleft.io/product/ ShiftLeft SAST] - .sl results file (Benchmark specific format. Ask vendor how to generate this)
 * [https://snappycodeaudit.com/category/static-code-analysis Snappycode Audit's SnappyTick Source Edition (SAST)] - .xml results file

@@ Line 394: / Line 394: @@
 * [https://pmd.github.io/ PMD] (which really has no security rules) - .xml results file
-* [http://findbugs.sourceforge.net/ Findbugs] - .xml results file (Note: The 'new' Findbugs is now at: https://spotbugs.github.io/)
+* [http://findbugs.sourceforge.net/ FindBugs] - .xml results file (Note: FindBugs hasn't been updated since 2015. Use SpotBugs instead (see below))
 * [https://www.sonarqube.org/downloads/ SonarQube] - .xml results file
 Note: We looked into supporting [http://checkstyle.sourceforge.net/ Checkstyle] but it has no security rules, just like PMD. The [http://fb-contrib.sourceforge.net/ fb-contrib] FindBugs plugin doesn't have any security rules either. We did test [http://errorprone.info/ Error Prone], and found that it does report some use of [http://errorprone.info/bugpattern/InsecureCipherMode) insecure ciphers (CWE-327)], but that's it.
@@ Line 402: / Line 403: @@
 '''Commercial SAST Tools:'''
-* [http://www.castsoftware.com/products/application-intelligence-platform CAST Application Intelligence Platform (AIP)] - .xml results file
+* [https://www.castsoftware.com/products/application-intelligence-platform CAST Application Intelligence Platform (AIP)] - .xml results file
 * [https://www.checkmarx.com/products/static-application-security-testing/ Checkmarx CxSAST] - .xml results file
 * [https://www.ibm.com/us-en/marketplace/ibm-appscan-source IBM AppScan Source (Standalone and Cloud)] - .ozasmt or .xml results file
 * [https://juliasoft.com/solutions/julia-for-security/ Julia Analyzer] - .xml results file
 * [https://www.parasoft.com/products/jtest/ Parasoft Jtest] - .xml results file
 * [https://www.shiftleft.io/product/ ShiftLeft SAST] - .sl results file (Benchmark specific format. Ask vendor how to generate this)
 * [https://www.sourcemeter.com/features/ SourceMeter] - .txt results file of ALL results from VulnerabilityHunter
 * [https://www.defensecode.com/thunderscan.php Thunderscan SAST] - .xml results file
-* [http://www.veracode.com/products/binary-static-analysis-sast Veracode SAST] - .xml results file
+* [https://www.veracode.com/products/binary-static-analysis-sast Veracode SAST] - .xml results file
 * [https://www.rigs-it.com/xanitizer/ XANITIZER] - xml results file ([https://www.rigs-it.com/wp-content/uploads/2018/03/howtosetupxanitizerforowaspbenchmarkproject.pdf Their white paper on how to setup Xanitizer to scan Benchmark.]) (Free trial available)
-We are looking for results for other commercial static analysis tools like: [http://www.grammatech.com/codesonar Grammatech CodeSonar], [http://www.klocwork.com/products-services/klocwork Klocwork], etc. If you have a license for any static analysis tool not already listed above and can run it on the Benchmark and send us the results file that would be very helpful.
+We are looking for results for other commercial static analysis tools like: [https://www.grammatech.com/products/codesonar Grammatech CodeSonar], [https://www.roguewave.com/products-services/klocwork RogueWave's Klocwork], etc. If you have a license for any static analysis tool not already listed above and can run it on the Benchmark and send us the results file that would be very helpful.
 The free SAST tools come bundled with the Benchmark so you can run them yourselves. If you have a license for any commercial SAST tool, you can also run them against the Benchmark. Just put your results files in the /results folder of the project, and then run the BenchmarkScore script for your platform (.sh / .bat) and it will generate a scorecard in the /scorecard directory for all the tools you have results for that are currently supported.
@@ Line 432: / Line 434: @@
 * [https://www.acunetix.com/vulnerability-scanner/ Acunetix Web Vulnerability Scanner (WVS)] - .xml results file (Generated using [https://www.acunetix.com/resources/wvs7manual.pdf command line interface (see Chapter 10.)] /ExportXML switch)
-* [https://portswigger.net/burp/ Burp Pro] - .xml results file
+* [https://portswigger.net/burp Burp Pro] - .xml results file
-**You must use Burp Pro v1.6.30+ to scan the Benchmark due to a limitation fixed in v1.6.30.
+* [https://www.ibm.com/us-en/marketplace/appscan-standard IBM AppScan] - .xml results file
 * [https://software.microfocus.com/en-us/products/webinspect-dynamic-analysis-dast/overview Micro Focus (Formally HPE) WebInspect] - .xml results file
 * [https://www.netsparker.com/web-vulnerability-scanner/ Netsparker] - .xml results file
 * [https://www.rapid7.com/products/appspider/ Rapid7 AppSpider] - .xml results file
 If you have access to other DAST Tools, PLEASE RUN THEM FOR US against the Benchmark, and send us the results file so we can build a scorecard generator for that tool.

@@ Line 7: / Line 7: @@
 The OWASP Benchmark for Security Automation (OWASP Benchmark) is a free and open test suite designed to evaluate the speed, coverage, and accuracy of automated software vulnerability detection tools and services (henceforth simply referred to as 'tools'). Without the ability to measure these tools, it is difficult to understand their strengths and weaknesses, and compare them to each other. Each version of the OWASP Benchmark contains thousands of test cases that are fully runnable and exploitable, each of which maps to the appropriate CWE number for that vulnerability.
-You can use the OWASP Benchmark with [[Source_Code_Analysis_Tools | Static Application Security Testing (SAST)]] tools, [[:Category:Vulnerability_Scanning_Tools | Dynamic Application Security Testing (DAST)]] tools like OWASP [[ZAP]] and Interactive Application Security Testing (IAST) tools. The current version of the Benchmark is implemented in Java.  Future versions may expand to include other languages.
+You can use the OWASP Benchmark with [[Source_Code_Analysis_Tools | Static Application Security Testing (SAST)]] tools, [[:Category:Vulnerability_Scanning_Tools | Dynamic Application Security Testing (DAST)]] tools like OWASP [[ZAP]] and Interactive Application Security Testing (IAST) tools. Benchmark is implemented in Java.  Future versions may expand to include other languages.
 ==Benchmark Project Scoring Philosophy==
@@ Line 51: / Line 51: @@
 Anyone can use this Benchmark to evaluate vulnerability detection tools. The basic steps are:
-# Download the Benchmark from github
+# Download the Benchmark from GitHub
 # Run your tools against the Benchmark
 # Run the BenchmarkScore tool on the reports from your tools
@@ Line 447: / Line 447: @@
 * [https://www.contrastsecurity.com/interactive-application-security-testing-iast Contrast Assess] - .zip results file (You can scan Benchmark w/Contrast for free. See: https://www.contrastsecurity.com/contrast-community-edition)
 * [https://hdivsecurity.com/interactive-application-security-testing-iast Hdiv Detection (IAST)] - .hlg results file
 '''Commercial Hybrid Analysis Application Security Testing Tools:'''
@@ Line 496: / Line 497: @@
   GIT: http://git-scm.com/ or https://github.com/
-  Maven: https://maven.apache.org/  (Version: 3.2.3 or newer works. We heard that 3.0.5 throws an error.)
+  Maven: https://maven.apache.org/  (Version: 3.2.3 or newer works.)
-  Java: http://www.oracle.com/technetwork/java/javase/downloads/index.html (Java 7 or 8) (64-bit) - Takes ALOT of memory to compile the Benchmark.
+  Java: http://www.oracle.com/technetwork/java/javase/downloads/index.html (Java 7 or 8) (64-bit)
 ==Getting, Building, and Running the Benchmark==

@@ Line 516: / Line 516: @@
 * Docker: A Dockerfile is checked into the project [https://github.com/OWASP/Benchmark/blob/master/VMs/Dockerfile here]. This Docker file should automatically produce a Docker VM with the latest Benchmark project files. After you have Docker installed, cd to /VMs then run:
-  buildDockerImage.sh --> This builds the Docker Benchmark VM (This will take a WHILE)
+  ./buildDockerImage.sh --> This builds the Docker Benchmark VM (This will take a WHILE)
   docker images  --> You should see the new benchmark:latest image in the list provided
   # The Benchmark Docker Image only has to be created once.
   To run the Benchmark in your Docker VM, just run:
-    runDockerImage.sh  --> This pulls in any updates to Benchmark since the Image was built, builds everything, and starts a remotely accessible Benchmark web app.
+    ./runDockerImage.sh  --> This pulls in any updates to Benchmark since the Image was built, builds everything, and starts a remotely accessible Benchmark web app.
   If successful, you should see this at the end:
     [INFO] [talledLocalContainer] Tomcat 8.x started on port [8443]
@@ Line 528: / Line 528: @@
   Or if you want to access from a different machine:
-   run: docker-machine ls (in a different window) --> To get IP Docker VM is exporting (e.g., tcp://192.168.99.100:2376)
+   docker-machine ls (in a different terminal) --> To get IP Docker VM is exporting (e.g., tcp://192.168.99.100:2376)
-   Navigate to: https://192.168.99.100:8443/benchmark (using the above IP as an example)
+   Navigate to: https://192.168.99.100:8443/benchmark in your browser (using the above IP as an example)
 * Amazon Web Services (AWS) - Here's how you set up the Benchmark on an AWS VM:

@@ Line 515: / Line 515: @@
 We have several preconstructed VMs or instructions on how to build one that you can use instead:
-* Docker: A Dockerfile is checked into the project [https://github.com/OWASP/Benchmark/blob/master/VMs/Dockerfile here]. This Docker file should automatically produce a Docker VM with the latest Benchmark project files. After you have Docker installed, run:
+* Docker: A Dockerfile is checked into the project [https://github.com/OWASP/Benchmark/blob/master/VMs/Dockerfile here]. This Docker file should automatically produce a Docker VM with the latest Benchmark project files. After you have Docker installed, cd to /VMs then run:
-  scripts/buildDockerImage.sh --> This builds the Docker Benchmark VM (This will take a WHILE)
+  buildDockerImage.sh --> This builds the Docker Benchmark VM (This will take a WHILE)
   docker images  --> You should see the new benchmark:latest image in the list provided
   # The Benchmark Docker Image only has to be created once.
   To run the Benchmark in your Docker VM, just run:
-    scripts/runDockerImage.sh  --> This clones the latest Benchmark from github, builds everything, and starts a remotely accessible Benchmark web app.
+    runDockerImage.sh  --> This pulls in any updates to Benchmark since the Image was built, builds everything, and starts a remotely accessible Benchmark web app.
   If successful, you should see this at the end:
     [INFO] [talledLocalContainer] Tomcat 8.x started on port [8443]
@@ Line 733: / Line 733: @@
 * In Terminal 1, launch the Benchmark application and wait until it starts
-    '''$  ./runBenchmark_wContrast.sh''' (.bat on Windows)
+    '''$ cd tools/Contrast
     '''[INFO] Scanning for projects...
     '''[INFO]
@@ Line 761: / Line 762: @@
     '''Copying Contrast report to results directory'''
-* Generate scorecards in /Benchmark/scorecard
+* In Terminal 2, generate scorecards in /Benchmark/scorecard
     '''$ ./createScorecards.sh''' (.bat on Windows)
     '''Analyzing results from Benchmark_1.2-Contrast.log
@@ Line 790: / Line 791: @@
 While we don't have hard and fast rules of exactly what we are going to do next, enhancements in the following areas are planned for the next release:
-* Add new vulnerability categories (e.g., Hibernate Injection)
+* Add new vulnerability categories (e.g., XXE, Hibernate Injection)
 * Add support for popular server side Java frameworks (e.g., Spring)
 * Add web services test cases

@@ Line 1: / Line 1: @@
 = Main =
-  <div style="width:100%;height:100px;border:0,margin:0;overflow: hidden;">[[File:Incubator_big.jpg|link=OWASP_Project_Stages#tab.3DLab_Projects]]</div>
+  <div style="width:100%;height:100px;border:0,margin:0;overflow: hidden;">[[File:Lab_big.jpg|link=OWASP_Project_Stages#tab.3DLab_Projects]]</div>
 {| style="padding: 0;margin:0;margin-top:10px;text-align:left;" |-
 | valign="top"  style="border-right: 1px dotted gray;padding-right:25px;" |