Winter Code Sprint - Revision history

Kevin W. Wall: Added 4 ESAPI related project proposals

2014-10-12T03:15:06Z

Added 4 ESAPI related project proposals

Abraham Aranguren: /* OWASP OWTF - VMS - OWTF Vulnerability Management System (FREE!) */

2014-10-07T00:00:55Z

‎OWASP OWTF - VMS - OWTF Vulnerability Management System (FREE!)

Abraham Aranguren: /* Participating OWASP Projects */

2014-10-07T00:00:09Z

‎Participating OWASP Projects

Abraham Aranguren at 23:49, 6 October 2014

2014-10-06T23:49:15Z

Abraham Aranguren at 23:41, 6 October 2014

2014-10-06T23:41:28Z

Abraham Aranguren: /* OWTF JLS - JavaScript Library Sniper Project (FREE!) */

2014-10-06T23:39:37Z

‎OWTF JLS - JavaScript Library Sniper Project (FREE!)

Abraham Aranguren: /* OWTF JLS - JavaScript Library Sniper Project (FREE!) */

2014-10-06T23:38:28Z

‎OWTF JLS - JavaScript Library Sniper Project (FREE!)

Abraham Aranguren at 23:37, 6 October 2014

2014-10-06T23:37:59Z

Abraham Aranguren: /* OWASP OWTF - Off-line HTTP traffic uploader (FREE!) */

2014-10-06T23:28:40Z

‎OWASP OWTF - Off-line HTTP traffic uploader (FREE!)

Abraham Aranguren at 23:27, 6 October 2014

2014-10-06T23:27:28Z

@@ Line 915: / Line 915: @@
 '''OWASP ZAP Mentor:'''
 Simon Bennetts - Contact: psiinon@gmail.com
 == Participating Universities, Professors ==

@@ Line 101: / Line 101: @@
 Proposed solution:
-We can meta aggregate these duplicate issues into one issue that outdated software / apache / php detected. with XYZ list of issues in them.
+We can meta aggregate these duplicate issues into one issue of "outdated software / apache / php detected". with XYZ list of issues in them.
 A separate set of scripts that allows for grouping and management of vulnerabilities (i.e. think huge assessments), to be usable *both* from inside + outside of OWTF in a separate sub-repo here: https://github.com/owtf

@@ Line 91: / Line 91: @@
 == Participating OWASP Projects ==
 === OWASP OWTF - HTTP Request Translator (FREE!) ===
@@ Line 868: / Line 914: @@
 '''OWASP ZAP Mentor:'''
 Simon Bennetts - Contact: psiinon@gmail.com
 == Participating Universities, Professors ==

← Older revision		Revision as of 23:49, 6 October 2014
Line 92:		Line 92:


		+	=== OWASP OWTF - HTTP Request Translator (FREE!) ===

		+	'''Brief explanation:'''

−	=== OWASP OWTF - ~~JLS~~ - JavaScript Library Sniper (FREE!) ===	+	Problem to solve:
		+
		+	There are many situations in web app pentests where just no tool will do the job and you need to script something, or mess around with the command line (classic example: sequence of steps where each step requires input from the previous step). In these situations, translating an HTTP request or a sequence of HTTP requests, takes valuable time which the pentester might just not really have.
		+
		+	Proposed solution:
		+
		+	An HTTP request translator, a standalone tool that can:
		+
		+	1) Be used from inside OR outside of OWTF.
		+
		+	2) Translate raw HTTP requests into curl commands or bash/python/php/ruby/PowerShell scripts
		+
		+	3) Provide essential quick and dirty transforms: base64 (encode/decode), urlencode (encode/decode)
		+	* Transforms with boundary strings? (TBD)
		+	* Individually or in bulk? (TBD)
		+
		+	'''Essential Function: "--output" argument'''
		+
		+	CRITICAL: The command/script should be generated so that the request is sent as literally as possible.
		+
		+	Example: NO client specific headers are sent. IF the original request had "User-Agent: X", the generated command/script should have EXACTLY that (i.e. NOT a curl user agent, etc.). Obviously, the same applies to ALL other headers.
		+
		+	NOTE: Ideally the following should be implemented using an extensible plugin architecture (i.e. NEW plugins are EASY to add)
		+	* http request in => curl command out
		+	* http request in => bash script out
		+	* http request in => python script out
		+	* http request in => php script out
		+	* http request in => ruby script out
		+	* http request in => PowerShell script out
		+
		+	'''Basic additional arguments:'''
		+
		+	- "--proxy" argument: generates the command/script with the relevant proxy option
		+
		+	NOTE: With this the command/script may send requests through a MiTM proxy (i.e. OWTF, ZAP, Burp, etc.)
		+
		+	- "--string-search" argument: generates the command/script so that it:
		+
		+	1) performs the request
		+
		+	2) then searches for something in the response (i.e. literal match)
		+
		+	- "--regex-search" argument: generates the command/script so that it:
		+	1) performs the request
		+
		+	2) then searches for something in the response (i.e. regex match)
		+
		+	'''OWTF integration'''
		+
		+	The idea here, is to invoke this tool from:
		+
		+	1) Single HTTP transactions:
		+
		+	For example, have a button to "export http request" + then show options equivalent to the flags
		+
		+	2) Multiple HTTP transactions:
		+
		+	Same as with Single transactions, but letting the user "select a number of transactions" first (maybe a checkbox?).
		+
		+
		+	'''Desired input formats:'''
		+
		+	* Read raw HTTP request from stdin -Suggested default behaviour! :)-
		+
		+	Example: cat path/to/http_request.txt \| http-request-translator.py --output
		+
		+	* Interactive mode: read raw HTTP request from keyboard + "hit enter when ready"
		+
		+	Suggestion: This could be a "-i" (for "interactive") flag and/or the fallback option when "stdin is empty"
		+
		+	Example:
		+
		+	1) User runs tool with desired flags (i.e. "--output ruby --proxy 127.0.0.1:1234 ...", etc.)
		+
		+	2) Tool prints: "Please paste a raw HTTP request and hit enter when ready"
		+
		+	3) User pastes a raw HTTP requests + hits enter
		+
		+	4) Tool outputs whatever is relevant for the flags + http request given
		+
		+	* For bulk processing: Maybe a directory of raw http request files?
		+
		+	'''Nice to have: Transforms'''
		+
		+	In the context of translating raw HTTP requests into commands/scripts, what we want here is to provide some handy "macros" so that the relevant command/script is generated accordingly.
		+
		+	Example:
		+
		+	NOTE: Assume something like the following arguments: "--transform-boundary=@@@@@@@ --transform-language=php"
		+
		+	Step 1) The user provides a raw HTTP request like this:
		+
		+	GET /path/to/urlencode@@@@@@@abc d@@@@@@@/test
		+	Host: target.com
		+	...
		+
		+	Step 2) The tool generates a bash script like the following:
		+
		+	#!/bin/bash
		+
		+	PARAM1=$(echo 'abc d' \| php -r "echo urlencode(fgets(STDIN));")
		+	curl ...... "http://target.com/path/to/$PARAM1/test"
		+
		+
		+	OR a "curl command" like the following:
		+	PARAM1=$(echo 'abc d' \| php -r "echo urlencode(fgets(STDIN));"); curl ...... "http://target.com/path/to/$PARAM1/test"
		+
		+
		+	This feature can be valuable to shave a bit more time in script writing.
		+
		+
		+	For background on OWASP OWTF please see: [https://www.owasp.org/index.php/OWASP_OWTF https://www.owasp.org/index.php/OWASP_OWTF]
		+
		+
		+	'''Expected results:'''
		+
		+	* '''IMPORTANT: [http://legacy.python.org/dev/peps/pep-0008/ PEP-8 compliant code] in all modified code and surrounding areas.'''
		+	* '''IMPORTANT: [https://github.com/7a/owtf/wiki/Contributor%27s-README OWTF contributor README compliant code]'''
		+	* '''IMPORTANT: [http://sphinx-doc.org/ Sphinx-friendly python comments] [http://owtf.github.io/ptp/_modules/ptp/tools/w3af/parser.html#W3AFXMLParser example Sphinx-friendly python comments here]'''
		+	* CRITICAL: Excellent reliability -i.e. the Health Monitor cannot crash! :)-
		+	* Good performance
		+	* Unit tests / Functional tests
		+	* Good documentation
		+
		+
		+	'''Knowledge Prerequisite:'''
		+
		+	Python and bash experience would be beneficial, some previous exposure to security concepts and penetration testing is welcome but not strictly necessary as long as there is will to learn
		+
		+
		+	'''OWASP OWTF Mentor:'''
		+
		+	Abraham Aranguren - OWASP OWTF Project Leader - Contact: Abraham.Aranguren@owasp.org
		+
		+	=== OWASP OWTF - JavaScript Library Sniper (FREE!) ===

	'''Brief explanation:'''		'''Brief explanation:'''

← Older revision		Revision as of 23:41, 6 October 2014
Line 94:		Line 94:


−	=== OWASP OWTF - JLS - JavaScript Library Sniper ~~Project~~ (FREE!) ===	+	=== OWASP OWTF - JLS - JavaScript Library Sniper (FREE!) ===

	'''Brief explanation:'''		'''Brief explanation:'''

← Older revision		Revision as of 23:39, 6 October 2014
Line 94:		Line 94:


−	=== OWTF JLS - JavaScript Library Sniper Project (FREE!) ===	+	=== OWASP OWTF - JLS - JavaScript Library Sniper Project (FREE!) ===

	'''Brief explanation:'''		'''Brief explanation:'''

@@ Line 115: / Line 115: @@
 ) Given a [ JavaScript file OR hash OR length ], found in the database, provides:
- * JavaScript Library version
+* JavaScript Library version
- * List of vulnerabilities sorted in descending CVSS score order
+* List of vulnerabilities sorted in descending CVSS score order
 ) (very cool to have) Given a list of JavaScript files (maybe a directory), provides:

← Older revision		Revision as of 23:37, 6 October 2014
Line 91:		Line 91:
	== Participating OWASP Projects ==		== Participating OWASP Projects ==

		+
		+
		+
		+	=== OWTF JLS - JavaScript Library Sniper Project (FREE!) ===
		+
		+	'''Brief explanation:'''
		+	This is a project that tries to resolve a very common problem during penetration tests:
		+
		+	The customer is running a number of outdated JavaScript Libraries, but there is just not enough time to determine if something useful -i.e. something really bad! :)- can be done with that or not.
		+
		+
		+	To solve this problem, we propose a standalone tool that can:
		+
		+	1) Be run BOTH from inside AND outside of OWTF
		+
		+	2) Build and update a fingerprint JavaScript library database of:
		+	* Library File hashes => JavaScript Library version
		+	* Library File lengths => JavaScript Library version
		+	* (Nice to have:) As above, but for each individual github commit (possible drawback: too big?)
		+
		+	3) Build and update a vulnerability database of:
		+	* JavaScript Library version => CVE - CVSS score - Vulnerability info
		+
		+	4) Given a [ JavaScript file OR hash OR length ], found in the database, provides:
		+	* JavaScript Library version
		+	* List of vulnerabilities sorted in descending CVSS score order
		+
		+	5) (very cool to have) Given a list of JavaScript files (maybe a directory), provides:
		+	* ALL Library/vulnerability matches described on 4)
		+
		+	Once the standalone tool is built and verified to be working, OWTF should be able to:
		+
		+	Feature 1) GREP plugin improvement (Web Application Fingerprint):
		+
		+	Step 1) Lookup file lengths and hashes in the "JavaScript library database"
		+
		+	Step 2) If a match is found: provide the list of known vulnerabilities against "JavaScript library X" to the user
		+
		+	Feature 2) SEMI-PASSIVE plugin improvement (Web Application Fingerprint):
		+
		+	1) Requests all referenced BUT missing JavaScript files -i.e. scanners won't load JavaScript files! :)-
		+
		+	2) re-runs the GREP plugin on the new files (i.e. to avoid missing vulns due to unrequested JavaScript files)
		+
		+	Potential projects worth having a look for potential overlap/inspiration:
		+	* [https://owasp.org/index.php/OWASP_Dependency_Check OWASP Dependency Check?]
		+
		+	How many JavaScript libraries should be included?
		+	* As many as possible, but especially the major ones: jQuery, knockout, etc.
		+	* "Nirvana" Nice to have: ALL Individual versions of ALL JavaScript files from ALL opensource projects, (ideally) even if the project is not a JavaScript library -i.e. JavaScript files from Joomla, Wordpress, etc.-
		+
		+	Common JavaScript library fingerprinting techniques include:
		+	* Parse the JavaScript file and grab the version from there
		+	* Determine the JavaScript version based on a hash of the file
		+	* Determine the JavaScript version based on the length of the file
		+
		+	Other Challenges:
		+	* "the file" could be "the minimised file", "the expanded file" or even "a specific JavaScript file from Library X"
		+	* When the JavaScript file does not match a specific version:
		+	1) The commit that matches the closest should (ideally) be found
		+	2) The NEXT library version after that commit (if present) should be found
		+	3) From there, it is about reusing the knowledge to figure out public vulnerabilities, CVSS scores, etc. again
		+
		+	For background on OWASP OWTF please see: [https://www.owasp.org/index.php/OWASP_OWTF https://www.owasp.org/index.php/OWASP_OWTF]
		+
		+
		+	'''Expected results:'''
		+
		+	* '''IMPORTANT: [http://legacy.python.org/dev/peps/pep-0008/ PEP-8 compliant code] in all modified code and surrounding areas.'''
		+	* '''IMPORTANT: [https://github.com/7a/owtf/wiki/Contributor%27s-README OWTF contributor README compliant code]'''
		+	* '''IMPORTANT: [http://sphinx-doc.org/ Sphinx-friendly python comments] [http://owtf.github.io/ptp/_modules/ptp/tools/w3af/parser.html#W3AFXMLParser example Sphinx-friendly python comments here]'''
		+	* CRITICAL: Excellent reliability -i.e. the Health Monitor cannot crash! :)-
		+	* Good performance
		+	* Unit tests / Functional tests
		+	* Good documentation
		+
		+
		+	'''Knowledge Prerequisite:'''
		+
		+	Python and bash experience would be beneficial, some previous exposure to security concepts and penetration testing is welcome but not strictly necessary as long as there is will to learn
		+
		+
		+	'''OWASP OWTF Mentor:'''
		+
		+	Abraham Aranguren - OWASP OWTF Project Leader - Contact: Abraham.Aranguren@owasp.org

	=== OWASP OWTF - Off-line HTTP traffic uploader (FREE!) ===		=== OWASP OWTF - Off-line HTTP traffic uploader (FREE!) ===

@@ Line 97: / Line 97: @@
 Although it is awesome that OWTF runs a lot of tools on behalf of the user, there are situations where uploading the HTTP traffic of another tool off-line can be very interesting for OWTF, for example:
-- Tools that OWTF has trouble proxying right now: skipfish, hoppy
-- Tools that the user may have run manually OR even from a tool aggregator -very common! :)-
+* Tools that OWTF has trouble proxying right now: skipfish, hoppy
-- Tools that we just don't run from OWTF: ZAP, Burp, Fiddler
+* Tools that the user may have run manually OR even from a tool aggregator -very common! :)-
 This project is about implementing an off-line utility able to parse HTTP traffic:
 ) Figure out how to read output files from various tools like:
 skipfish, hoppy, w3af, arachni, etc.
 Nice to have: ZAP database, Burp database
 ) Translate that into the following clearly defined fields:
 HTTP request
-HTTP response status code
+* HTTP request
-HTTP response headers
+* HTTP response status code
-HTTP response body
+* HTTP response headers
 ) IMPORTANT: Implement a plugin-based uploader system
 ) IMPORTANT: Implement ONE plugin, that uploads that into the OWTF database
 ) IMPORTANT: OWTF should ideally be able to invoke the uploader right after running a tool
 	Example: OWTF runs skipfish, skipfish finishes, OWTF runs the HTTP traffic uploader, all skipfish data is pushed to the OWTF DB.
 ) CRITICAL: The off-line HTTP traffic uploader should be smart enough to read + push 1-by-1 instead of *stupidly* trying to load everything into memory first, you have been warned! :)
 	Why? Because in a huge assessment, the output of "tool X" can be "10 GB", which is *stupid* to load into memory, this is OWTF, we *really* try to foresee the crash before it happens! ;)
 CRITICAL: It is important to implement a plugin-based uploader system, so that other projects can benefit from this work (i.e. to be able to import third-party tool data to ZAP, Burp, and other tools in a similar fashion), and hence hopefully join us in maintaining this project moving forward.
@@ Line 142: / Line 152: @@
 Abraham Aranguren - OWASP OWTF Project Leader - Contact: Abraham.Aranguren@owasp.org
 === OWASP OWTF - Health Monitor (FREE!) ===

← Older revision		Revision as of 23:27, 6 October 2014
Line 90:		Line 90:

	== Participating OWASP Projects ==		== Participating OWASP Projects ==
		+
		+
		+	=== OWASP OWTF - Off-line HTTP traffic uploader (FREE!) ===
		+
		+	'''Brief explanation:'''
		+
		+	Although it is awesome that OWTF runs a lot of tools on behalf of the user, there are situations where uploading the HTTP traffic of another tool off-line can be very interesting for OWTF, for example:
		+	- Tools that OWTF has trouble proxying right now: skipfish, hoppy
		+	- Tools that the user may have run manually OR even from a tool aggregator -very common! :)-
		+	- Tools that we just don't run from OWTF: ZAP, Burp, Fiddler
		+
		+	This project is about implementing an off-line utility able to parse HTTP traffic:
		+	1) Figure out how to read output files from various tools like:
		+	skipfish, hoppy, w3af, arachni, etc.
		+	Nice to have: ZAP database, Burp database
		+	2) Translate that into the following clearly defined fields:
		+	HTTP request
		+	HTTP response status code
		+	HTTP response headers
		+	HTTP response body
		+	3) IMPORTANT: Implement a plugin-based uploader system
		+	4) IMPORTANT: Implement ONE plugin, that uploads that into the OWTF database
		+	5) IMPORTANT: OWTF should ideally be able to invoke the uploader right after running a tool
		+	Example: OWTF runs skipfish, skipfish finishes, OWTF runs the HTTP traffic uploader, all skipfish data is pushed to the OWTF DB.
		+	6) CRITICAL: The off-line HTTP traffic uploader should be smart enough to read + push 1-by-1 instead of stupidly trying to load everything into memory first, you have been warned! :)
		+	Why? Because in a huge assessment, the output of "tool X" can be "10 GB", which is stupid to load into memory, this is OWTF, we really try to foresee the crash before it happens! ;)
		+
		+	CRITICAL: It is important to implement a plugin-based uploader system, so that other projects can benefit from this work (i.e. to be able to import third-party tool data to ZAP, Burp, and other tools in a similar fashion), and hence hopefully join us in maintaining this project moving forward.
		+
		+
		+	For background on OWASP OWTF please see: [https://www.owasp.org/index.php/OWASP_OWTF https://www.owasp.org/index.php/OWASP_OWTF]
		+
		+
		+	'''Expected results:'''
		+
		+	* '''IMPORTANT: [http://legacy.python.org/dev/peps/pep-0008/ PEP-8 compliant code] in all modified code and surrounding areas.'''
		+	* '''IMPORTANT: [https://github.com/7a/owtf/wiki/Contributor%27s-README OWTF contributor README compliant code]'''
		+	* '''IMPORTANT: [http://sphinx-doc.org/ Sphinx-friendly python comments] [http://owtf.github.io/ptp/_modules/ptp/tools/w3af/parser.html#W3AFXMLParser example Sphinx-friendly python comments here]'''
		+	* CRITICAL: Excellent reliability -i.e. the Health Monitor cannot crash! :)-
		+	* Good performance
		+	* Unit tests / Functional tests
		+	* Good documentation
		+
		+
		+	'''Knowledge Prerequisite:'''
		+
		+	Python and bash experience would be beneficial, some previous exposure to security concepts and penetration testing is welcome but not strictly necessary as long as there is will to learn
		+
		+
		+	'''OWASP OWTF Mentor:'''
		+
		+	Abraham Aranguren - OWASP OWTF Project Leader - Contact: Abraham.Aranguren@owasp.org