OWASP - User contributions [en]

OWASP Code Kids 2015 Ideas

2014-11-09T14:20:33Z

Abdelhadi AZOUNI: /* OWASP ZAP */

=Task Categories=

The tasks are grouped into the categories described below. '''Please make sure each task is assigned a category.'''

'''Code:''' Tasks related to writing or refactoring code.

'''Documentation/Training:''' Tasks related to creating/editing documents and helping others learn more

'''Outreach/Research:''' Tasks related to community management, outreach/marketing, or studying problems and recommending solutions

'''Quality Assurance:''' Tasks related to testing and ensuring code is of high quality

'''User Interface:''' Tasks related to user experience research or user interface design and interaction

== OWASP ZAP ==

=== OWASP ZAP Task 1 ===

'''Brief description:'''

Write a blogpost about CMS-scnanning techniques, this will include web-apps fingerprinting methods, vulnerability checking using on-line databases and a survey of existing tools.

'''Task Category:'''

Documentation

'''Expected Results:'''

A professional blogpost that will be published on OWASP website

'''Knowledge Prerequisites:'''

Good understanding of web security basics, Knowledge on how do CMSs work and good writing skills.

'''Mentors:''' Abdelhadi

=== OWASP ZAP Task 2 ===

'''Brief description:'''

Rebuild the CMSscanner GUI including a progress bar that shows scanning progress and a textzone to display tried strings and paths used by the scanner in real time

'''Task Category:'''

Code/Design

'''Expected Results:'''

New GUI with progress bar and displaying paths when scanning

'''Knowledge Prerequisites:'''

Java language, GUI design.

'''Mentors:''' Abdelhadi

== OWASP OWTF ==

=== OWASP OWTF Task 1 ===

'''Brief Explanation:'''

Task description

'''Task Category:'''

Eg. Code Category

'''Expected Results:'''

Describe the expected results of the task

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' XXXXXX

== OWASP WIKI ==

=== Task 1: Latam Tour 2015 logo ===

'''Brief Explanation:'''

Design a new logo for the Latam Tour 2015. The logo must resemble previous editions of the Tour and represent the Latin America region. It would be better if the new logo is based on the OWASP logo. As a reference, here is the Latam Tour 2014 Logo:

https://www.owasp.org/images/f/f3/OWASP_Latam_Tour_Logo_2014.png

'''Task Category:'''

Design

'''Expected Results:'''

Latam Tour 2015 logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Fabio Cerullo

== OWASP WebGoatPHP ==

=== Task 1: Implement "remember me" feature ===

'''Brief Explanation:'''

Implement a secure "Remember me" feature in user login form using cookies. At present the remember me check box is present in the form but it does nothing.

'''Task Category:'''

Code

'''Expected Results:'''

If user checks the "remember me" check box when logging in, then the user will not be required to login every time he visits the application within X days.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/45

'''Code:'''

app/control/user/login.php

'''Mentors:''' Shivam Dixit

=== Task 2: Make workshop mode dashboard responsive ===

'''Brief Explanation:'''

In workshop mode of the application, the side panel of admin dashboard is not responsive i.e it does not fits well in smaller size screen resolutions. If the screen size is small the side panel should shrink into a smaller panel preferably at the bottom of the application.

'''Task Category:'''

Code

'''Expected Results:'''

Panel perfectly adjusts on small screen resolutions.

'''Knowledge Prerequisites:'''

CSS (media queries), HTML

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/26

'''Code:'''

style/dashboard.css

'''Mentors:''' Shivam Dixit

=== Task 3: WebGoatPHP logo ===

'''Brief Explanation:'''

Design a new logo for the application. The logo must resemble various aspects of the application. It would be better if the new logo is based on the OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

WebGoatPHP logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Shivam Dixit

=== Task 4: WebGoatPHP deployment screencast ===

'''Brief Explanation:'''

Deploy the application on the local server without using vagrant and record a screencast of the process. Upload to a video streaming service and comment link on the melange for mentor to review.

'''Task Category:'''

Code

'''Expected Results:'''

The screencast should clearly contain all the steps required for the deployment and how to troubleshoot most common errors in the whole process.

'''Knowledge Prerequisites:'''

Familiarity with an operating system (Linux/Windows)

'''Mentors:''' Shivam Dixit

=== Task 5: Create a SQL injection challenge ===

'''Brief Explanation:'''

Single user mode of WebGoatPHP consist of set of challenges. These challenges simulate various real world security vulnerabilities in web applications. You have to add a challenge under category "Injection Attacks" which simulates a SQL injection vulnerability in single user mode. The input data must be of type string and the challenge should mimic some real world scenario.

'''Task Category:'''

Code

'''Expected Results:'''

A challenge which helps user understand SQLi vulnerability by allowing him to exploit the vulnerability.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://www.owasp.org/index.php/SQL_Injection

https://github.com/shivamdixit/WebGoatPHP/blob/master/README.md#adding-a-lessonchallenge

'''Mentors:''' Shivam Dixit

=== Task 6-20: WebGoatPHP challenges screencast series ===

'''Brief Explanation:'''

In this task you are required to record screencast of how to solve a particular single user mode challenge. The screencast should start by providing an overview of the vulnerability that will be exploited, then step by step instructions on how to exploit the vulnerability. The screencast should conclude on a note that how to avoid this vulnerability in your application. The length of the screencast would vary according to the challenge but it should neither be too long nor too short.

Task - Screencast of challenge.....

Task 6 - HTTP Basic

Task 7 - Using Access Control Matrix

Task 8 - Business Layer Access Control

Task 9 - Path Based Access Control

Task 10 - Same Origin Policy Protection

Task 11 - Forgot Password

Task 12 - Discover clues in HTML

Task 13 - JS Obfuscation

Task 14 - XSS 1 (Reflected)

Task 15 - XSS 2 (Stored)

Task 16 - XSS 3 (DOM)

Task 17 - Fail Open Authentication

Task 18 - Log Spoofing

Task 19 - Numeric SQL Injection

Task 20 - XPATH injection

'''Task Category:'''

Code

'''Expected Results:'''

A screencast explaining the vulnerability involved in a particular challenge.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' Shivam Dixit

== OWASP CSRF Protector ==

=== Task 1-2: CSRF Protector logo ===

'''Brief Explanation:'''

Design logos for the for CSRF Protector Project, possibly two versions one for php library and another one for Apache module.
Both of them should resemble OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

OWASP CSRF Protector logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Minhaz

=== Task 3: Porting CSRF Protector PHP Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for CSRF Protector php library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''Reference'''

[https://github.com/mebjas/CSRF-Protector-PHP/wiki Github wiki for CSRF Protector php]

'''Mentors:''' Minhaz

=== Task 4: Porting mod_csrfprotector Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for mod_csrfprotector library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''References'''

[https://github.com/mebjas/mod_csrfprotector/wiki Github wiki for mod_csrfprotector]

'''Mentors:''' Minhaz

=== Task 5-6: Create screencasts on how to deploy both version of CSRF Protector individually ===
'''Brief Explanation:'''

Create two screencasts, one for each, which explains how to deploy CSRF Protector in your existing web application.

'''Task Category:'''

Screencast

'''Expected Results:'''

Screencasts explaining how to use CSRF Protector with existing web applications.

'''Knowledge Prerequisites:'''

Experience with php, HTML, and Apache (for mod_csrfprotector)

'''Mentors:''' Minhaz

OWASP Code Kids 2015 Ideas

2014-11-09T14:14:01Z

Abdelhadi AZOUNI: /* OWASP ZAP Task 1 */

=Task Categories=

The tasks are grouped into the categories described below. '''Please make sure each task is assigned a category.'''

'''Code:''' Tasks related to writing or refactoring code.

'''Documentation/Training:''' Tasks related to creating/editing documents and helping others learn more

'''Outreach/Research:''' Tasks related to community management, outreach/marketing, or studying problems and recommending solutions

'''Quality Assurance:''' Tasks related to testing and ensuring code is of high quality

'''User Interface:''' Tasks related to user experience research or user interface design and interaction

== OWASP ZAP ==

=== OWASP ZAP Task 1 ===

'''Brief description:'''

Write a blogpost about CMS-scnanning techniques, this will include web-apps fingerprinting methods, vulnerability checking using on-line databases and a survey of existing tools.

'''Task Category:'''

Documentation

'''Expected Results:'''

A professional blogpost that will be published on OWASP website

'''Knowledge Prerequisites:'''

Good understanding of web security basics, Knowledge on how do CMSs work and good writing skills.

'''Mentors:''' Abdelhadi

== OWASP OWTF ==

=== OWASP OWTF Task 1 ===

'''Brief Explanation:'''

Task description

'''Task Category:'''

Eg. Code Category

'''Expected Results:'''

Describe the expected results of the task

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' XXXXXX

== OWASP WIKI ==

=== Task 1: Latam Tour 2015 logo ===

'''Brief Explanation:'''

Design a new logo for the Latam Tour 2015. The logo must resemble previous editions of the Tour and represent the Latin America region. It would be better if the new logo is based on the OWASP logo. As a reference, here is the Latam Tour 2014 Logo:

https://www.owasp.org/images/f/f3/OWASP_Latam_Tour_Logo_2014.png

'''Task Category:'''

Design

'''Expected Results:'''

Latam Tour 2015 logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Fabio Cerullo

== OWASP WebGoatPHP ==

=== Task 1: Implement "remember me" feature ===

'''Brief Explanation:'''

Implement a secure "Remember me" feature in user login form using cookies. At present the remember me check box is present in the form but it does nothing.

'''Task Category:'''

Code

'''Expected Results:'''

If user checks the "remember me" check box when logging in, then the user will not be required to login every time he visits the application within X days.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/45

'''Code:'''

app/control/user/login.php

'''Mentors:''' Shivam Dixit

=== Task 2: Make workshop mode dashboard responsive ===

'''Brief Explanation:'''

In workshop mode of the application, the side panel of admin dashboard is not responsive i.e it does not fits well in smaller size screen resolutions. If the screen size is small the side panel should shrink into a smaller panel preferably at the bottom of the application.

'''Task Category:'''

Code

'''Expected Results:'''

Panel perfectly adjusts on small screen resolutions.

'''Knowledge Prerequisites:'''

CSS (media queries), HTML

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/26

'''Code:'''

style/dashboard.css

'''Mentors:''' Shivam Dixit

=== Task 3: WebGoatPHP logo ===

'''Brief Explanation:'''

Design a new logo for the application. The logo must resemble various aspects of the application. It would be better if the new logo is based on the OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

WebGoatPHP logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Shivam Dixit

=== Task 4: WebGoatPHP deployment screencast ===

'''Brief Explanation:'''

Deploy the application on the local server without using vagrant and record a screencast of the process. Upload to a video streaming service and comment link on the melange for mentor to review.

'''Task Category:'''

Code

'''Expected Results:'''

The screencast should clearly contain all the steps required for the deployment and how to troubleshoot most common errors in the whole process.

'''Knowledge Prerequisites:'''

Familiarity with an operating system (Linux/Windows)

'''Mentors:''' Shivam Dixit

=== Task 5: Create a SQL injection challenge ===

'''Brief Explanation:'''

Single user mode of WebGoatPHP consist of set of challenges. These challenges simulate various real world security vulnerabilities in web applications. You have to add a challenge under category "Injection Attacks" which simulates a SQL injection vulnerability in single user mode. The input data must be of type string and the challenge should mimic some real world scenario.

'''Task Category:'''

Code

'''Expected Results:'''

A challenge which helps user understand SQLi vulnerability by allowing him to exploit the vulnerability.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://www.owasp.org/index.php/SQL_Injection

https://github.com/shivamdixit/WebGoatPHP/blob/master/README.md#adding-a-lessonchallenge

'''Mentors:''' Shivam Dixit

=== Task 6-20: WebGoatPHP challenges screencast series ===

'''Brief Explanation:'''

In this task you are required to record screencast of how to solve a particular single user mode challenge. The screencast should start by providing an overview of the vulnerability that will be exploited, then step by step instructions on how to exploit the vulnerability. The screencast should conclude on a note that how to avoid this vulnerability in your application. The length of the screencast would vary according to the challenge but it should neither be too long nor too short.

Task - Screencast of challenge.....

Task 6 - HTTP Basic

Task 7 - Using Access Control Matrix

Task 8 - Business Layer Access Control

Task 9 - Path Based Access Control

Task 10 - Same Origin Policy Protection

Task 11 - Forgot Password

Task 12 - Discover clues in HTML

Task 13 - JS Obfuscation

Task 14 - XSS 1 (Reflected)

Task 15 - XSS 2 (Stored)

Task 16 - XSS 3 (DOM)

Task 17 - Fail Open Authentication

Task 18 - Log Spoofing

Task 19 - Numeric SQL Injection

Task 20 - XPATH injection

'''Task Category:'''

Code

'''Expected Results:'''

A screencast explaining the vulnerability involved in a particular challenge.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' Shivam Dixit

== OWASP CSRF Protector ==

=== Task 1-2: CSRF Protector logo ===

'''Brief Explanation:'''

Design logos for the for CSRF Protector Project, possibly two versions one for php library and another one for Apache module.
Both of them should resemble OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

OWASP CSRF Protector logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Minhaz

=== Task 3: Porting CSRF Protector PHP Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for CSRF Protector php library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''Reference'''

[https://github.com/mebjas/CSRF-Protector-PHP/wiki Github wiki for CSRF Protector php]

'''Mentors:''' Minhaz

=== Task 4: Porting mod_csrfprotector Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for mod_csrfprotector library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''References'''

[https://github.com/mebjas/mod_csrfprotector/wiki Github wiki for mod_csrfprotector]

'''Mentors:''' Minhaz

=== Task 5-6: Create screencasts on how to deploy both version of CSRF Protector individually ===
'''Brief Explanation:'''

Create two screencasts, one for each, which explains how to deploy CSRF Protector in your existing web application.

'''Task Category:'''

Screencast

'''Expected Results:'''

Screencasts explaining how to use CSRF Protector with existing web applications.

'''Knowledge Prerequisites:'''

Experience with php, HTML, and Apache (for mod_csrfprotector)

'''Mentors:''' Minhaz

OWASP Code Kids 2015 Ideas

2014-11-09T14:13:05Z

Abdelhadi AZOUNI: /* OWASP ZAP Task 1 */

=Task Categories=

The tasks are grouped into the categories described below. '''Please make sure each task is assigned a category.'''

'''Code:''' Tasks related to writing or refactoring code.

'''Documentation/Training:''' Tasks related to creating/editing documents and helping others learn more

'''Outreach/Research:''' Tasks related to community management, outreach/marketing, or studying problems and recommending solutions

'''Quality Assurance:''' Tasks related to testing and ensuring code is of high quality

'''User Interface:''' Tasks related to user experience research or user interface design and interaction

== OWASP ZAP ==

=== OWASP ZAP Task 1 ===

'''Task description:'''

Write a blogpost about CMS-scnanning techniques, this will include web-apps fingerprinting methods, vulnerability checking using on-line databases and a survey of existing tools.

'''Task Category:'''

Documentation

'''Expected Results:'''

A professional blogpost that will be published on OWASP website

'''Knowledge Prerequisites:'''

Good understanding of web security basics, Knowledge on how do CMSs work and good writing skills.

'''Mentors:''' Abdelhadi

== OWASP OWTF ==

=== OWASP OWTF Task 1 ===

'''Brief Explanation:'''

Task description

'''Task Category:'''

Eg. Code Category

'''Expected Results:'''

Describe the expected results of the task

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' XXXXXX

== OWASP WIKI ==

=== Task 1: Latam Tour 2015 logo ===

'''Brief Explanation:'''

Design a new logo for the Latam Tour 2015. The logo must resemble previous editions of the Tour and represent the Latin America region. It would be better if the new logo is based on the OWASP logo. As a reference, here is the Latam Tour 2014 Logo:

https://www.owasp.org/images/f/f3/OWASP_Latam_Tour_Logo_2014.png

'''Task Category:'''

Design

'''Expected Results:'''

Latam Tour 2015 logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Fabio Cerullo

== OWASP WebGoatPHP ==

=== Task 1: Implement "remember me" feature ===

'''Brief Explanation:'''

Implement a secure "Remember me" feature in user login form using cookies. At present the remember me check box is present in the form but it does nothing.

'''Task Category:'''

Code

'''Expected Results:'''

If user checks the "remember me" check box when logging in, then the user will not be required to login every time he visits the application within X days.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/45

'''Code:'''

app/control/user/login.php

'''Mentors:''' Shivam Dixit

=== Task 2: Make workshop mode dashboard responsive ===

'''Brief Explanation:'''

In workshop mode of the application, the side panel of admin dashboard is not responsive i.e it does not fits well in smaller size screen resolutions. If the screen size is small the side panel should shrink into a smaller panel preferably at the bottom of the application.

'''Task Category:'''

Code

'''Expected Results:'''

Panel perfectly adjusts on small screen resolutions.

'''Knowledge Prerequisites:'''

CSS (media queries), HTML

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/26

'''Code:'''

style/dashboard.css

'''Mentors:''' Shivam Dixit

=== Task 3: WebGoatPHP logo ===

'''Brief Explanation:'''

Design a new logo for the application. The logo must resemble various aspects of the application. It would be better if the new logo is based on the OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

WebGoatPHP logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Shivam Dixit

=== Task 4: WebGoatPHP deployment screencast ===

'''Brief Explanation:'''

Deploy the application on the local server without using vagrant and record a screencast of the process. Upload to a video streaming service and comment link on the melange for mentor to review.

'''Task Category:'''

Code

'''Expected Results:'''

The screencast should clearly contain all the steps required for the deployment and how to troubleshoot most common errors in the whole process.

'''Knowledge Prerequisites:'''

Familiarity with an operating system (Linux/Windows)

'''Mentors:''' Shivam Dixit

=== Task 5: Create a SQL injection challenge ===

'''Brief Explanation:'''

Single user mode of WebGoatPHP consist of set of challenges. These challenges simulate various real world security vulnerabilities in web applications. You have to add a challenge under category "Injection Attacks" which simulates a SQL injection vulnerability in single user mode. The input data must be of type string and the challenge should mimic some real world scenario.

'''Task Category:'''

Code

'''Expected Results:'''

A challenge which helps user understand SQLi vulnerability by allowing him to exploit the vulnerability.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://www.owasp.org/index.php/SQL_Injection

https://github.com/shivamdixit/WebGoatPHP/blob/master/README.md#adding-a-lessonchallenge

'''Mentors:''' Shivam Dixit

=== Task 6-20: WebGoatPHP challenges screencast series ===

'''Brief Explanation:'''

In this task you are required to record screencast of how to solve a particular single user mode challenge. The screencast should start by providing an overview of the vulnerability that will be exploited, then step by step instructions on how to exploit the vulnerability. The screencast should conclude on a note that how to avoid this vulnerability in your application. The length of the screencast would vary according to the challenge but it should neither be too long nor too short.

Task - Screencast of challenge.....

Task 6 - HTTP Basic

Task 7 - Using Access Control Matrix

Task 8 - Business Layer Access Control

Task 9 - Path Based Access Control

Task 10 - Same Origin Policy Protection

Task 11 - Forgot Password

Task 12 - Discover clues in HTML

Task 13 - JS Obfuscation

Task 14 - XSS 1 (Reflected)

Task 15 - XSS 2 (Stored)

Task 16 - XSS 3 (DOM)

Task 17 - Fail Open Authentication

Task 18 - Log Spoofing

Task 19 - Numeric SQL Injection

Task 20 - XPATH injection

'''Task Category:'''

Code

'''Expected Results:'''

A screencast explaining the vulnerability involved in a particular challenge.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' Shivam Dixit

== OWASP CSRF Protector ==

=== Task 1-2: CSRF Protector logo ===

'''Brief Explanation:'''

Design logos for the for CSRF Protector Project, possibly two versions one for php library and another one for Apache module.
Both of them should resemble OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

OWASP CSRF Protector logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Minhaz

=== Task 3: Porting CSRF Protector PHP Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for CSRF Protector php library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''Reference'''

[https://github.com/mebjas/CSRF-Protector-PHP/wiki Github wiki for CSRF Protector php]

'''Mentors:''' Minhaz

=== Task 4: Porting mod_csrfprotector Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for mod_csrfprotector library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''References'''

[https://github.com/mebjas/mod_csrfprotector/wiki Github wiki for mod_csrfprotector]

'''Mentors:''' Minhaz

=== Task 5-6: Create screencasts on how to deploy both version of CSRF Protector individually ===
'''Brief Explanation:'''

Create two screencasts, one for each, which explains how to deploy CSRF Protector in your existing web application.

'''Task Category:'''

Screencast

'''Expected Results:'''

Screencasts explaining how to use CSRF Protector with existing web applications.

'''Knowledge Prerequisites:'''

Experience with php, HTML, and Apache (for mod_csrfprotector)

'''Mentors:''' Minhaz

OWASP Code Kids 2015 Ideas

2014-11-09T14:12:12Z

Abdelhadi AZOUNI: /* OWASP ZAP Task 1 */

=Task Categories=

The tasks are grouped into the categories described below. '''Please make sure each task is assigned a category.'''

'''Code:''' Tasks related to writing or refactoring code.

'''Documentation/Training:''' Tasks related to creating/editing documents and helping others learn more

'''Outreach/Research:''' Tasks related to community management, outreach/marketing, or studying problems and recommending solutions

'''Quality Assurance:''' Tasks related to testing and ensuring code is of high quality

'''User Interface:''' Tasks related to user experience research or user interface design and interaction

== OWASP ZAP ==

=== OWASP ZAP Task 1 ===

Write a blogpost about CMS-scnanning techniques, this will include web-apps fingerprinting methods, vulnerability checking using on-line databases and a survey of existing tools.

Task description

'''Task Category:'''

Documentation

'''Expected Results:'''

A professional blogpost that will be published on OWASP website

'''Knowledge Prerequisites:'''

Good understanding of web security basics, Knowledge on how do CMSs work and good writing skills.

'''Mentors:''' Abdelhadi

== OWASP OWTF ==

=== OWASP OWTF Task 1 ===

'''Brief Explanation:'''

Task description

'''Task Category:'''

Eg. Code Category

'''Expected Results:'''

Describe the expected results of the task

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' XXXXXX

== OWASP WIKI ==

=== Task 1: Latam Tour 2015 logo ===

'''Brief Explanation:'''

Design a new logo for the Latam Tour 2015. The logo must resemble previous editions of the Tour and represent the Latin America region. It would be better if the new logo is based on the OWASP logo. As a reference, here is the Latam Tour 2014 Logo:

https://www.owasp.org/images/f/f3/OWASP_Latam_Tour_Logo_2014.png

'''Task Category:'''

Design

'''Expected Results:'''

Latam Tour 2015 logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Fabio Cerullo

== OWASP WebGoatPHP ==

=== Task 1: Implement "remember me" feature ===

'''Brief Explanation:'''

Implement a secure "Remember me" feature in user login form using cookies. At present the remember me check box is present in the form but it does nothing.

'''Task Category:'''

Code

'''Expected Results:'''

If user checks the "remember me" check box when logging in, then the user will not be required to login every time he visits the application within X days.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/45

'''Code:'''

app/control/user/login.php

'''Mentors:''' Shivam Dixit

=== Task 2: Make workshop mode dashboard responsive ===

'''Brief Explanation:'''

In workshop mode of the application, the side panel of admin dashboard is not responsive i.e it does not fits well in smaller size screen resolutions. If the screen size is small the side panel should shrink into a smaller panel preferably at the bottom of the application.

'''Task Category:'''

Code

'''Expected Results:'''

Panel perfectly adjusts on small screen resolutions.

'''Knowledge Prerequisites:'''

CSS (media queries), HTML

'''Reference:'''

https://github.com/shivamdixit/WebGoatPHP/issues/26

'''Code:'''

style/dashboard.css

'''Mentors:''' Shivam Dixit

=== Task 3: WebGoatPHP logo ===

'''Brief Explanation:'''

Design a new logo for the application. The logo must resemble various aspects of the application. It would be better if the new logo is based on the OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

WebGoatPHP logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Shivam Dixit

=== Task 4: WebGoatPHP deployment screencast ===

'''Brief Explanation:'''

Deploy the application on the local server without using vagrant and record a screencast of the process. Upload to a video streaming service and comment link on the melange for mentor to review.

'''Task Category:'''

Code

'''Expected Results:'''

The screencast should clearly contain all the steps required for the deployment and how to troubleshoot most common errors in the whole process.

'''Knowledge Prerequisites:'''

Familiarity with an operating system (Linux/Windows)

'''Mentors:''' Shivam Dixit

=== Task 5: Create a SQL injection challenge ===

'''Brief Explanation:'''

Single user mode of WebGoatPHP consist of set of challenges. These challenges simulate various real world security vulnerabilities in web applications. You have to add a challenge under category "Injection Attacks" which simulates a SQL injection vulnerability in single user mode. The input data must be of type string and the challenge should mimic some real world scenario.

'''Task Category:'''

Code

'''Expected Results:'''

A challenge which helps user understand SQLi vulnerability by allowing him to exploit the vulnerability.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Reference:'''

https://www.owasp.org/index.php/SQL_Injection

https://github.com/shivamdixit/WebGoatPHP/blob/master/README.md#adding-a-lessonchallenge

'''Mentors:''' Shivam Dixit

=== Task 6-20: WebGoatPHP challenges screencast series ===

'''Brief Explanation:'''

In this task you are required to record screencast of how to solve a particular single user mode challenge. The screencast should start by providing an overview of the vulnerability that will be exploited, then step by step instructions on how to exploit the vulnerability. The screencast should conclude on a note that how to avoid this vulnerability in your application. The length of the screencast would vary according to the challenge but it should neither be too long nor too short.

Task - Screencast of challenge.....

Task 6 - HTTP Basic

Task 7 - Using Access Control Matrix

Task 8 - Business Layer Access Control

Task 9 - Path Based Access Control

Task 10 - Same Origin Policy Protection

Task 11 - Forgot Password

Task 12 - Discover clues in HTML

Task 13 - JS Obfuscation

Task 14 - XSS 1 (Reflected)

Task 15 - XSS 2 (Stored)

Task 16 - XSS 3 (DOM)

Task 17 - Fail Open Authentication

Task 18 - Log Spoofing

Task 19 - Numeric SQL Injection

Task 20 - XPATH injection

'''Task Category:'''

Code

'''Expected Results:'''

A screencast explaining the vulnerability involved in a particular challenge.

'''Knowledge Prerequisites:'''

Comfortable in PHP, HTML and possibly Javascript. Good understanding of Application Security and related vulnerabilities.

'''Mentors:''' Shivam Dixit

== OWASP CSRF Protector ==

=== Task 1-2: CSRF Protector logo ===

'''Brief Explanation:'''

Design logos for the for CSRF Protector Project, possibly two versions one for php library and another one for Apache module.
Both of them should resemble OWASP logo.

'''Task Category:'''

Design

'''Expected Results:'''

OWASP CSRF Protector logo in either psd or jpeg format.

'''Knowledge Prerequisites:'''

Familiarity with Photoshop/GIMP or any other designing software.

'''Mentors:''' Minhaz

=== Task 3: Porting CSRF Protector PHP Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for CSRF Protector php library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''Reference'''

[https://github.com/mebjas/CSRF-Protector-PHP/wiki Github wiki for CSRF Protector php]

'''Mentors:''' Minhaz

=== Task 4: Porting mod_csrfprotector Wiki (from Github) to OWASP Wiki ===

'''Brief Explanation:'''

Currently we have wiki on how to use and deploy, at github. The task is to port them to OWASP Wiki as well so that it can be accessed directly.

'''Task Category:'''

Documentation

'''Expected Results:'''

Wiki for mod_csrfprotector library in OWASP.ORG .

'''Knowledge Prerequisites:'''

Familiarity with wiki.

'''References'''

[https://github.com/mebjas/mod_csrfprotector/wiki Github wiki for mod_csrfprotector]

'''Mentors:''' Minhaz

=== Task 5-6: Create screencasts on how to deploy both version of CSRF Protector individually ===
'''Brief Explanation:'''

Create two screencasts, one for each, which explains how to deploy CSRF Protector in your existing web application.

'''Task Category:'''

Screencast

'''Expected Results:'''

Screencasts explaining how to use CSRF Protector with existing web applications.

'''Knowledge Prerequisites:'''

Experience with php, HTML, and Apache (for mod_csrfprotector)

'''Mentors:''' Minhaz

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:53:22Z

Abdelhadi AZOUNI:

Status: in writing (24-07-2013)
WebApp Scanner on GIT: https://github.com/abdelhadi-azouni/zap-cmss-extension

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Aggressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerabilities Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Transition to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

==== '''Used techniques''' ====

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

==== '''existing tools''' ====

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

==== '''our implementation''' ====

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

=== '''aggressive search''' ===

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

==== '''used techniques''' ====

File and Folder Presence (HTTP response codes): This approach doesn't download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

[[File:404.png]]

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

==== '''existing tools''' ====

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutely Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

==== '''our implementation''' ====

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we chosen to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

[[File:BlindElephant.png]]
As described by author at its home page, The Static File Fingerprinting Approach in One Picture

dependencies : jdom-2.0.5: un parser XML,

== '''Component enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used techniques''' ====

==== '''existent tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used techniques''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existent tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detection and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

WebApp Scanner on GIT : https://github.com/abdelhadi-azouni/zap-cmss-extension

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:45:36Z

Abdelhadi AZOUNI:

Status: in writing (24-07-2013)
WebApp Scanner on GIT: https://github.com/abdelhadi-azouni/zap-cmss-extension

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Transition to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

==== '''Used techniques''' ====

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

==== '''existing tools''' ====

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

==== '''our implementation''' ====

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

=== '''aggressive search''' ===

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

==== '''used technics''' ====

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

[[File:404.png]]

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

==== '''existing tools''' ====

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

==== '''our implementation''' ====

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we chosen to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

[[File:BlindElephant.png]]
As described by author at its home page, The Static File Fingerprinting Approach in One Picture

dependencies : jdom-2.0.5: un parser XML,

== '''Component enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used technics''' ====

==== '''existant tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used technics''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existant tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

WebApp Scanner on GIT : https://github.com/abdelhadi-azouni/zap-cmss-extension

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:42:49Z

Abdelhadi AZOUNI:

Status: in writing (24-07-2013)
WebApp Scanner on GIT: https://github.com/abdelhadi-azouni/zap-cmss-extension

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Transition to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

==== '''Used techniques''' ====

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

==== '''existing tools''' ====

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

==== '''our implementation''' ====

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

=== '''aggressive search''' ===

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

==== '''used technics''' ====

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

[[File:404.png]]

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

==== '''existing tools''' ====

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

==== '''our implementation''' ====

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we choosed to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

[[File:BlindElephant.png]]
As described by author at its home page, The Static File Fingerprinting Approach in One Picture

dependancies : jdom-2.0.5: un parser XML,

== '''Componant enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used technics''' ====

==== '''existant tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used technics''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existant tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

WebApp Scanner on GIT : https://github.com/abdelhadi-azouni/zap-cmss-extension

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:41:31Z

Abdelhadi AZOUNI:

Status: in writing (24-07-2013)
WebApp Scanner on GIT: https://github.com/abdelhadi-azouni/zap-cmss-extension

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Transition to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

==== '''Used techniques''' ====

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

==== '''existing tools''' ====

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

==== '''our implementation''' ====

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

=== '''aggressive search''' ===

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

==== '''used technics''' ====

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

[[File:404.jpg]]

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

==== '''existing tools''' ====

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

==== '''our implementation''' ====

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we choosed to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

[[File:BlindElephant.jpg]]
As described by author at its home page, The Static File Fingerprinting Approach in One Picture

dependancies : jdom-2.0.5: un parser XML,

== '''Componant enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used technics''' ====

==== '''existant tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used technics''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existant tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

WebApp Scanner on GIT : https://github.com/abdelhadi-azouni/zap-cmss-extension

File:BlindElephant.png

2013-07-24T23:41:10Z

Abdelhadi AZOUNI:

File:404.png

2013-07-24T23:40:40Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:33:16Z

Abdelhadi AZOUNI:

Status: in writing (24-07-2013)
WebApp Scanner on GIT: https://github.com/abdelhadi-azouni/zap-cmss-extension

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Transition to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

==== '''Used techniques''' ====

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

==== '''existing tools''' ====

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

==== '''our implementation''' ====

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

=== '''aggressive search''' ===

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

==== '''used technics''' ====

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

==== '''existing tools''' ====

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

==== '''our implementation''' ====

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we choosed to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

dependancies : jdom-2.0.5: un parser XML,

== '''Componant enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used technics''' ====

==== '''existant tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used technics''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existant tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

WebApp Scanner on GIT : https://github.com/abdelhadi-azouni/zap-cmss-extension

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:26:16Z

Abdelhadi AZOUNI:

Status: in writing (24-07-2013)
WebApp Scanner on GIT: https://github.com/abdelhadi-azouni/zap-cmss-extension

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Passing to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

==== '''Used techniques''' ====

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

==== '''existing tools''' ====

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

==== '''our implementation''' ====

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

=== '''aggressive search''' ===

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

==== '''used technics''' ====

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

==== '''existing tools''' ====

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

==== '''our implementation''' ====

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we choosed to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

dependancies : jdom-2.0.5: un parser XML,

== '''Componant enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used technics''' ====

==== '''existant tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used technics''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existant tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

WebApp Scanner on GIT : https://github.com/abdelhadi-azouni/zap-cmss-extension

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:22:57Z

Abdelhadi AZOUNI:

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Passing to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

==== '''Used techniques''' ====

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

==== '''existing tools''' ====

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

==== '''our implementation''' ====

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

=== '''aggressive search''' ===

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

==== '''used technics''' ====

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

==== '''existing tools''' ====

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

==== '''our implementation''' ====

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we choosed to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

dependancies : jdom-2.0.5: un parser XML,

== '''Componant enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used technics''' ====

==== '''existant tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used technics''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existant tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:21:00Z

Abdelhadi AZOUNI:

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Passing to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

=== '''Used techniques''' ===

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

=== '''existing tools''' ===

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

=== '''our implementation''' ===

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

== '''aggressive search''' ==

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

=== '''used technics''' ===

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

=== '''existing tools''' ===

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

=== '''our implementation''' ===

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we choosed to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

dependancies : jdom-2.0.5: un parser XML,

== '''Componant enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

=== '''passive search''' ===

pending

==== '''used technics''' ====

==== '''existant tools''' ====

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

==== '''used technics''' ====

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

==== '''existant tools''' ====

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

==== '''our implementation''' ====

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

=== '''passive search''' ===

==== '''used techniques''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

=== '''aggressive search''' ===

==== '''used techniques ''' ====

==== '''existing tools''' ====

==== '''our implementation''' ====

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:17:29Z

Abdelhadi AZOUNI:

== '''Introduction''' ==

Latest Stats Show that the usage of CMS has grown in the last 5 years Just WordPress and Joomla occupy more than 6% of the top 1 million site
The Usage Has Grown in Both Corporate and Personal sites And with the Chaotic Development of plugins and Components in those CMSs The risk of vulnerabilities and flows increase more and more

[[File:statCMS.jpg]]

OWASP ZAP CMS SCANNER (ZAP CMSS) is a Scanner with More specified search methods

== '''Functionalities''' ==

- Enumerating Plugins and Components and Themes in the CMS (Passive and Aggressive Search methods)

- Enumerating from page content

- Enumerating from lists (or database)

- Version Fingerprinting (with multiple methods)

- Labor intensive to add signatures

- Manually locate version in files or build regexes for headers

- Built-in options to remove identifiers (eg, meta generator)

[[File:CMSSFunctions.jpg]]

- Enumerating Vulnerable plugins , themes and Components

- Enumerating from a well-known list

- Enumerating from web search

- Enumerating using the ZAP api

- Enumerating Security measures (firewalls, security plugins ...)

[[File:CMSSModules.jpg]]

== '''Matches''' ==

Matches are made with:

- Text strings (case sensitive)

- Regular expressions

- Google Hack Database queries (limited set of keywords)

- MD5 hashes

- URL recognition

- HTML tag patterns

- Custom java code for passive and aggressive operations

== '''Features''' ==

- Control the trade off between speed/stealth and reliability

- Plugins include example URLs

- Performance tuning. Control how many websites to scan concurrently

- Result certainty awareness

- Fast

- Low resource usage

- Accurate (Low FP/FN)

- Resistant to hardening/banner removal

- Super easy to support new versions/apps

== '''ZAP CMSS modules''' ==

ZAP CMS scanner extension consists of four main modules:

=== '''1- CMS detection module''' ===

mains to indicate what CMS uses the application of given url, two methods are used with CMS detector :

==== '''a- Passive search'''====

based on page content analysis :

1 - using text string (case sensitive)

2 - using regex

these two methods are used to recognize html tags, eg: meta tag Generator, or to extract texts from the page showing the tool with which this application is created

3 - using google hacks (from a list of predefined keywords)

==== '''b- Agressive search''' ====

Is to try known and unique URLs from a predefined list, the presence of these paths indicates with certainty the CMS used, the problem with this method is that it will not be very effective in if several CMSs are supported, this because of the absence of a single file in this case

=== '''2- Plugin enumerating module''' ===

The purpose of this module is to list the plugins used in the application of the given url, both passive and active methods are always possible

'''a- Passive search:''' analyzing the page content using regex

'''b- Aggressive research:''' using a list of names of plugins, which will be compared with any names found in specific URLs, for example, WordPress plugins in : url + / wp-content/plugins /

=== '''3- Fingerprinting module''' ===

Used to identify with controllable accuracy - overlooked used method and processing time - the version of CMS and / or plugins used, the joint research passive / aggressive is used, using

- Content analysis of some specific file, example: a readme file contains the following information: "package to Version 3.0.x"

- A list / database that contains unique file-links / paths that determines version of CMS / plugin

- A list / database that contains the correspondence : plugin version / filePath / hashDegest, so after you have verified the presence of a unique name file, but not unique content file, comparing its digest with that present in the database, the result indicates the component version
here is an example of WordPress versions database :

[[File:WP_hashDB_Xml.jpg]]

=== '''4- Vulnerability Enumeration module''' ===

Used to give a list of vulnerabilities that contains a given plugin, this module is called after the steps of sensing the CMS and plugin enumerating, this module use :

1- database that contains the correspondence between name-version-plugin / vulnerability-list

2- web search based on a list of links to useful sites

== '''Flowsheet''' ==

[[File:flowsheet.jpg]]

== '''Passing to WebApp Scanner''' ==

after pushing the research of fingerprinting techniques, and advanced in detailed design, it is apparent that is more appropriate and useful to go wider, and work on web applications in general and not only CMSs .
I spoke with Simon about it, and he was pleased with the proposal, so I started the implementation of the web application finger printer core methods based on existing tools such as BlindElephant and Wappalyzer.

== '''Application fingerprinting''' ==

=== '''Passive search''' ===

is to look in the target application (HTML, HTTP headers content ...) patterns that determine how likely or definite name and version of the technology used to make this application, passive research does not change anything in requests nor in content.

=== '''Used techniques''' ===

web page content analysis: is to look for patterns in the HTML document, the scan tool will download the web page and perform patterns search using regex, usually from predefined lists that are updated. Several patterns may indicate the technology used to make the application: Here are the indicators:

'''- The HTML content itself:''' by performing specific regex patterns on the page content, we can determine with high probability the technology used to build the application, for example:the pattern : "Powered by (: <a href=[^>] + cs-cart \ \ com | CS-Cart?)"..

'''- The meta tag Generator:''' for a given app, if the following regex pattern performed : "? WordPress ([. \ \ D] +) \ \; Version: \ \ 1" on the content (“content”) of the meta tag named : "generator", then if successful research therefore the application is, with high probability, carried out with the indicated version of WordPress.

most WordPress websites can be identified by the meta HTML tag, e.g. <meta name="generator" content="WordPress 2.6.5">, but a minority of WordPress websites remove this identifying tag but this does not thwart our scanner, other tests are implemented to reveal a Wordpress application.

- The script tag: its contents may contain the name of the implementation technology, the search is always done with regex patterns.

This method is suitable for most applications and gives good results in a very acceptable time, but it is sensitive to changes in the code of the page code sensitive, as it is always possible to remove these indicators.

=== '''existing tools''' ===

among tools the most efficient and best known tools that uses passive research: Wappalazer, it was originally a Firefox and Chrome extension in javascript, but It was then rewritten in several languages like python. Wappalyzer uses a long list of regex structured by type of application and type of pattern (name tag, headers ...) in a JSON file.

home page : http://wappalyzer.com/
GIT repo: https://github.com/ElbertF/Wappalyzer

WhatWeb another webapps fingerprinting tool which also uses the passive research, but not so deep as Wappalyzer.

home page: http://www.morningstarsecurity.com/research/whatweb
GIT repo: https://github.com/urbanadventurer/WhatWeb
Here is a brief comparison to other fingerprinting tools:
https://github.com/urbanadventurer/WhatWeb/wiki/Related-Projects

=== '''our implementation''' ===

we implemented a similar code to Wappalyzer and it uses the same JSON list-of-regex file, our tool connect firstly to the target’s URL, then it downloads a web DOM document, it contain the HTML content of the page, than the program applies regex patterns (by type), by reading them once from the JSON file into an object of JsonObject class.

== '''perspectives:''' ==
implement other methods of passive search, those present in Whatweb: access to htaccess

dependencies : json-simple-1.1.1 : JSON parser,
jsoup1.7.2 : HTML parser

== '''aggressive search''' ==

consists on Bruteforcing target host in search of indicators files, the presence of such a file reveals a certain technology or even its version as well, but in most cases, the version is set after comparing the MD5 digest of the content of the file found on the host with the present in a pre-built database.

=== '''used technics''' ===

File and Folder Presence (HTTP response codes): This approach doesn’t download the page however it starts looking for obvious trails of an application by directly hitting the URL and in course identifying found and not found application list. In starting days of internet this was easy, just download headers and see if it’s 200 OK or 404 not found and you are done.

However in current scenario, people have been putting up custom 404 Pages and are actually sending 200 OK in case the page is not found. This complicates the efforts and hence the new approach is as follows.
Download default page 200 OK.
Download a file which is guaranteed to be non-existing then mark it as a template for 404 and then proceed with detection logic.
Based on this assumption and knowledge this kind of tools start looking for known files and folders on a website and try to determine the exact application name and version. an example of such scenario would be
wp-login.php => wordpress
/owa/ => Microsoft outlook web frontend.

reference : http://anantshri.info/articles/web_app_finger_printing.html

Checksum Based identification :This is relatively a newer approach and the most accurate one for now.
This Technique basically works on below pattern.
1) Create checksum for files locally and store them in DB
2) Download static file from remote server
3) Create the checksum
4) Compare with the checksum stored in db and identify the version
One of the best implementation of this technique is BlindElephant

reference: http://anantshri.info/articles/web_app_finger_printing.html

=== '''existing tools''' ===

BlinElephant: among the most robust implementations Checksum Based identification. It attempts to discover the version of a (known) web application by comparing static files at known locations against precomputed hashes for versions of those files in all all available releases. The technique is fast, low-bandwidth, non-invasive, generic, and highly automatable. BlinElephant is absolutly Checksum verification oriented
home page: http://blindelephant.sourceforge.net/
Sourceforge repo: http://sourceforge.net/projects/blindelephant/

=== '''our implementation''' ===

we wrote a similar code to BlindElephant (which is in python), using the same updated database (available on its Sourceforge repo). The files of the database are PKL format which is a Python specific format , it’s a Python object serialization format. So in order to process this data in Java, we needed to convert it to a universal format, so we choosed to convert it into XML files using a small Python script.
every webapp has its own XML file, each XML file has the following format:

the resulting output (name and version of the webapp or technology) is discovered according to the following schema:

dependancies : jdom-2.0.5: un parser XML,

== '''Componant enumeration and fingerprinting''' ==

the enumeration Is to provide a list of all components, modules, plugins or themes related to the target’s application technology . The fingerprinting is to detect the version of the module already detected, the two processes (enumeration and fingerprinting) are generally parallel: to each detected component, it will be applied the fingerprinting.

why enumerate and fingerprint plugins?
the enumeration can detect the presence of vulnerable plugins and / or components, which allows the attacker, or pen-tester, directly targeting known vulnerabilities and fix them.

This process is, especially, very effective in CMSs case. In fact, most CMSs are based on plugins and extensions system,the ones they are vulnerable are generally well known and classified into vulnerability databases.

== '''passive search''' ==

pending

== '''used technics''' ==

== '''existant tools''' ==

DPscan for Drupal (https://github.com/cervoise/DPScan)
(
our implementation:

=== '''aggressive search''' ===

aggressive research (applied to components and extensions listing) is to Brute Force components and extensions paths in the URL of the target’s application, using a path lists. The result (presence or absence of such a component) is obtained by analyzing the returned HTTP code.
after you extract the name of the existing component in the application, we need to identify the version of the component, this process is not really based on general technique, we are going to use a variety of techniques like fetching in the readme file.
Example: if the README file is present and accessible in a WordPress component, you can easily open and read the Stable tag part, obtained by applying the regex pattern "Stable tag: (+.)" on this file.

=== '''used technics''' ===

bruteforce site content by using a list of paths of known components. This is for the detection of plugin itself. To detect its version, varied methods are possible and depends on the component and the webapp technology.

=== '''existant tools''' ===

OWASP ODZ MultiCMS Scanner (https://www.owasp.org/index.php/OWASP_Odz_MultiCMSScanner), (https://github.com/islamoc/odz)
the majority of enumeration tools are specific, eg Joomscan OWASP for the Joomla CMS (http://sourceforge.net/projects/joomscan/), WordPress WPscan (http://wpscan.org/) and WordPress also Plecost (https://code.google.com/p/plecost/)
Some tools work on a combination of CMSs or/and webapps. Eg:

- CMS-explorer: written in Perl, designed to reveal the the specific modules, plugins, components and themes that various CMS driven web sites are running. CMS Explorer currently supports module/theme discovery with the following products:
Drupal
Wordpress
Joomla!
Mambo

Project home on Sourceforge: https://code.google.com/p/cms-explorer/

- WebSorrow: a perl based tool for misconfiguration, version detection, enumeration, and server information scanning. It's entirely focused on Enumeration and collecting Info on the target server. Web-Sorrow is a "safe to run" program, meaning it is not designed to be an exploit or perform any harmful attacks.

Project home on Sourceforge: https://code.google.com/p/web-sorrow/

'''CMS-explorer vs WebSorrow'''

Concerning database, they use the same list of plugins, CMS-explorer also uses a list of themes and WebSorrow not. Both deal Joomla, WordPress and Drupal. WebSorrow make more detections and work. Both looks like they use the “fuzzDB” database, which is updated and contains many other files of other types of audits.

fuzzDB project home on Sourceforge: https://code.google.com/p/fuzzdb/

=== '''our implementation''' ===

passive search not implemented yet

for aggressive research, we hesitated between using existing databases and creating our own. Then, we decided to use the existing ones, but combining files between various existing tools, so we decided to implement a modular system: for each CMS or webapp, we create a module (of component detection) because we figured out that the effort to unify the formats of databases is more than creating a module for each Webapp , in addition to the specific treatment in the extraction of each Webapp version.

== '''Listing vulnerabilities''' ==

== '''passive search''' ==

=== '''used techniques''' ===

=== '''existing tools''' ===

=== '''our implementation''' ===

== '''aggressive search''' ==

=== '''used techniques ''' ===

=== '''existing tools''' ===

=== '''our implementation''' ===

Other sources
very useful :
https://github.com/urbanadventurer/WhatWeb/wiki/How-to-develop-WhatWeb-plugins
http://resources.infosecinstitute.com/prototype-model-web-application-fingerprinting/

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T23:03:47Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T19:07:23Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-24T19:06:38Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-06T18:57:41Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-06T18:57:10Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-06T16:55:11Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:31:33Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:30:18Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:28:47Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:19:54Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:17:52Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:15:09Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:09:19Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:02:12Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:01:16Z

Abdelhadi AZOUNI:

File:Flowsheet.jpg

2013-07-04T18:00:56Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T18:00:26Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:57:44Z

Abdelhadi AZOUNI:

File:WP hashDB Xml.jpg

2013-07-04T17:54:10Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:52:57Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:48:32Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:36:41Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:33:39Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:31:43Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:22:00Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:12:45Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-07-04T17:11:55Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-06-28T17:53:09Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-06-28T17:18:10Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-06-28T17:17:36Z

Abdelhadi AZOUNI:

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-06-28T17:17:01Z

Abdelhadi AZOUNI:

File:CMSSFunctions.jpg

2013-06-28T17:14:54Z

Abdelhadi AZOUNI:

File:CMSSModules.jpg

2013-06-28T17:13:03Z

Abdelhadi AZOUNI:

File:StatCMS.jpg

2013-06-28T17:12:00Z

Abdelhadi AZOUNI: uploaded a new version of "File:StatCMS.jpg"

GSoC2013 Ideas/OWASP ZAP CMS SCANNER

2013-06-28T17:09:06Z

Abdelhadi AZOUNI: