OWASP - User contributions [en]

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2014-08-01T22:54:21Z

Cmlh: Replaced forth reference

{{Template:OWASP Testing Guide v4}}

== Summary ==
This section describes how to test the robots.txt file for information leakage of the web application's directory or folder path(s). Furthermore, the list of directories that are to be avoided by Spiders, Robots, or Crawlers can also be created as a dependency for OWASP-IG-009[https://www.owasp.org/index.php/Testing_Map_execution_paths_through_application_(OWASP-IG-009)]

== Test Objectives ==
1. Information leakage of the web application's directory or folder path(s).

2. Create the list of directories that are to be avoided by Spiders, Robots, or Crawlers.

== How to Test ==
Web Spiders, Robots, or Crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the ''Robots Exclusion Protocol'' of the robots.txt file in the web root directory [1].
 

'''robots.txt in webroot''' 
As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
<pre>
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

The ''User-Agent'' directive refers to the specific web spider/robot/crawler. For example the ''User-Agent: Googlebot'' refers to the spider from Google while "User-Agent: bingbot"[http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2010/06/28/bing-crawler-bingbot-on-the-horizon.aspx] refers to crawler from Microsoft/Yahoo!. ''User-Agent: *'' in the example above applies to all web spiders/robots/crawlers [2] as quoted below:
<pre>
User-agent: *
</pre>

The ''Disallow'' directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
<pre>
...
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

Web spiders/robots/crawlers can intentionally ignore the ''Disallow'' directives specified in a robots.txt file [3], such as those from Social Networks[https://www.htbridge.com/news/social_networks_can_robots_violate_user_privacy.html] to ensure that shared linked are still valid. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
 

'''<META> Tag''' 

<META> tags are located within the HEAD section of each HTML Document and should be consistent across a web site in the likely event that the robot/spider/crawler start point does not begin from a document link other than webroot i.e. a "deep link"[http://en.wikipedia.org/wiki/Deep_linking].

If there is no "<META NAME="ROBOTS" ... >" entry then the "Robots Exclusion Protocol" defaults to "INDEX,FOLLOW" respectively. Therefore, the other two valid entries defined by the "Robots Exclusion Protocol" are prefixed with "NO..." i.e. "NOINDEX" and "NOFOLLOW".

Web spiders/robots/crawlers can intentionally ignore the "<META NAME="ROBOTS"" tag as the robots.txt file convention is preferred. Hence, <META> Tags should not be considered the primary mechanism, rather a complementary control to robots.txt.

=== Black Box Testing ===
'''robots.txt in webroot - with "wget" or "curl" 

The robots.txt file is retrieved from the web root directory of the web server. For example, to retrieve the robots.txt from www.google.com using "wget" or "curl":
<pre>
cmlh$ wget http://www.google.com/robots.txt
--2013-08-11 14:40:36-- http://www.google.com/robots.txt
Resolving www.google.com... 74.125.237.17, 74.125.237.18, 74.125.237.19, ...
Connecting to www.google.com|74.125.237.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘robots.txt.1’

[ <=> ] 7,074 --.-K/s in 0s

2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved [7074]

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
<pre>
cmlh$ curl -O http://www.google.com/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>

'''robots.txt in webroot - with rockspider''' 
"rockspider"[https://github.com/cmlh/rockspider/releases] automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site.

For example, to create the initial scope based on the Allowed: directive from www.google.com using "rockspider"[https://github.com/cmlh/rockspider/releases]:
<pre>
cmlh$ ./rockspider.pl -www www.google.com

"Rockspider" Alpha v0.1_2

Copyright 2013 Christian Heinrich
Licensed under the Apache License, Version 2.0

1. Downloading http://www.google.com/robots.txt
2. "robots.txt" saved as "www.google.com-robots.txt"
3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080
/catalogs/about sent
/catalogs/p? sent
/news/directory sent
...
4. Done.

cmlh$
</pre>

'''<META> Tags - with Burp''' 

Based on the Disallow directive(s) listed within the robots.txt file in webroot, a regular expression search for "<META NAME="ROBOTS"" within each web page is undertaken and the result compared to the robots.txt file in webroot.

For example, the robots.txt file from facebook.com has a "Disallow: /ac.php" entry[http://facebook.com/robots.txt] and the resulting search for "<META NAME="ROBOTS"" shown below:
 
[[File:CMLH-Meta Tag Example-Facebook-Aug 2013.png]]
 

The above might be considered a fail since "INDEX,FOLLOW" is the default <META> Tag specified by the "Robots Exclusion Protocol" yet "Disallow: /ac.php" is listed in robots.txt.

'''Analyze robots.txt using Google Webmaster Tools''' 
Web site owners can use the Google "Analyze robots.txt" function to analyse the website as part of its "Google Webmaster Tools" (https://www.google.com/webmasters/tools). This tool can assist with testing and the procedure is as follows:

1. Sign into Google Webmaster Tools with a Google account. 
2. On the dashboard, write the URL for the site to be analyzed. 
3. Choose between the available methods and follow the on screen instruction. 

=== Gray Box Testing ===
The process is the same as Black Box testing above.
 

== Tools ==

* Browser (View Source function)
* curl
* wget
* rockspider[https://github.com/cmlh/rockspider/releases]

== References ==
'''Whitepapers''' 
* [1] "The Web Robots Pages" - http://www.robotstxt.org/
* [2] "Block and Remove Pages Using a robots.txt File" - https://support.google.com/webmasters/answer/156449
* [3] "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
* [4] "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2014-05-17T10:04:35Z

Cmlh: Add TODO for v5

__TOC__

== Discussion ==
It could be added that, from an attacker point of view, the robots.txt file can provide some useful information on the structure of the web server, e.g., directories that are supposed to be "private".

[[User:Marco|Marco]] 18:11, 17 August 2008 (EDT)

The intent of robots.txt is *not* to specify access control for directories. Hence to quote the wiki page "''Web spiders/robots/crawlers can intentionally ignore the Disallow directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.''".

If you believe this is not your communicated clearly or could be reworded then please amend the wiki page.

[[User:cmlh|cmlh]] 12:34, 24 August 2008 (GMT +10)

== v3 Review Comments ==

I don't see anything here about actually testing robots.txt or using Spiders/Robots/Crawlers to do anything to the web app. It's nice that we can DL the file and that it contains some interesting information and that there's a google tool that can do some analysis of it (though we haven't explained what google webmaster tools gives you or provided an example of the output), but where would that lead a tester or attacker? 
[[User:Rick.mitchell|Rick.mitchell]] 09:39, 3 September 2008 (EDT)

== Reply from @cmlh ==

Rick may have overlooked the quote "Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties. " from in the "How to Test" section of v3.

The lack of the "Google Webmaster Tools" example is due to me not being the webmaster of owasp.org. This can be resolved in v4 once the webmaster is known.

: For the first part, either that sentence wasn't part of the content I reviewed during v3 draft or it didn't seem significant enough given the lead-in.

: As for the webmaster tools stuff that sounds good, however like INFO-001 seems awfully google-centric.

:: CH - Bing/Yahoo! have been included in the roadmap for v4.

::: Perfect [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]]) 19:20, 15 August 2013 (CDT)

: This content also only covers robots.txt though the heading suggests much broader coverage. So either the content should be expanded or the heading made more specific (IMHO). [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]]) 15:07, 15 August 2013 (CDT)

:: CH - I included http://www.robotstxt.org/meta.html in the OWASP Testing Guide v3 (since I also presented on this content in 2009 and 2010). I am not sure if it was removed by subsequent edits by others (I haven't checked this) but I will include it for v4 again.

::: Sounds good. Though I'd still argue that this covers an alternative to robots.txt not any actually different or other mechanisms as implied by "web server metafiles" in the heading; which to me reads like there are other configuration, instruction or interaction governors to be discussed in this section. Just my 2 cents [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]]) 19:20, 15 August 2013 (CDT)

== TODO for v4 ==

1. Insert the "Analyze robots.txt using Google Webmaster Tools" i.e. https://support.google.com/webmasters/answer/156449?hl=en&from=35237&rd=1 with owasp.org (not applicable, since webroot doesn't contain robots.txt) as the example.

2. May need to update the reference to OWASP-IG-009 within the "Summary" section depending on the finalisation of the spidering thread (To be created).

3. Add Microsoft/Yahoo! related content. - DONE

== TODO for v5 ==

http://blog.erratasec.com/2014/05/no-mcafee-didnt-violate-ethics-scraping.html
http://blog.osvdb.org/2014/05/07/the-scraping-problem-and-ethics/
https://github.com/behindthefirewalls/Parsero

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-09-03T07:01:38Z

Cmlh: Refactor 3 as done/completed

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-09-03T06:59:35Z

Cmlh: Add BingBot

{{Template:OWASP Testing Guide v4}}

== Summary ==
This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009[https://www.owasp.org/index.php/Testing_Map_execution_paths_through_application_(OWASP-IG-009)]

== Test Objectives ==
1. Information Leakage of the web application's directory/folder path(s).

2. Create the list of directories that are to be avoided by Spiders/Robots/Crawlers

== How to Test ==
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the ''Robots Exclusion Protocol'' of the robots.txt file in the web root directory [1].
 

'''robots.txt in webroot''' 
As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
<pre>
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

The ''User-Agent'' directive refers to the specific web spider/robot/crawler. For example the ''User-Agent: Googlebot'' refers to the spider from Google while "User-Agent: bingbot"[http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2010/06/28/bing-crawler-bingbot-on-the-horizon.aspx] refers to crawler from Microsoft/Yahoo!. ''User-Agent: *'' in the example above applies to all web spiders/robots/crawlers [2] as quoted below:
<pre>
User-agent: *
</pre>

The ''Disallow'' directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
<pre>
...
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>
Web spiders/robots/crawlers can intentionally ignore the ''Disallow'' directives specified in a robots.txt file [3], such as those from Social Networks[https://www.htbridge.com/news/social_networks_can_robots_violate_user_privacy.html] to ensure that shared linked are still valid. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
 

'''<META> Tag''' 

<META> tags are located within the HEAD section of each HTML Document and should be consistent across a web site in the likely event that the robot/spider/crawler start point does not begin from a document link other than webroot i.e. a "deep link"[http://en.wikipedia.org/wiki/Deep_linking].

If there is no "<META NAME="ROBOTS"" ... >" entry than the "Robots Exclusion Protocol'' defaults to "INDEX,FOLLOW" respectively. Therefore, the other two valid entries defined by the "Robots Exclusion Protocol'' are prefixed with "NO..." i.e. "NOINDEX" and "NOFOLLOW".

Web spiders/robots/crawlers can intentionally ignore the "<META NAME="ROBOTS"" tag as the robots.txt file convention is preferred. Hence, <META> Tags should not be considered the primary mechanism, rather a complementary control to robots.txt.

=== Black Box testing and example ===
'''robots.txt in webroot - with "wget" or "curl" 
The robots.txt file is retrieved from the web root directory of the web server.

For example, to retrieve the robots.txt from www.google.com using ''wget'' or "curl":
<pre>
cmlh$ wget http://www.google.com/robots.txt
--2013-08-11 14:40:36-- http://www.google.com/robots.txt
Resolving www.google.com... 74.125.237.17, 74.125.237.18, 74.125.237.19, ...
Connecting to www.google.com|74.125.237.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘robots.txt.1’

[ <=> ] 7,074 --.-K/s in 0s

2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved [7074]

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
<pre>
cmlh$ curl -O http://www.google.com/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
'''robots.txt in webroot - with rockspider''' 
"rockspider"[https://github.com/cmlh/rockspider/releases] automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site

For example, to create the initial scope based on the Allowed: directive from www.google.com using "rockspider"[https://github.com/cmlh/rockspider/releases]:
<pre>
cmlh$ ./rockspider.pl -www www.google.com

"Rockspider" Alpha v0.1_2

Copyright 2013 Christian Heinrich
Licensed under the Apache License, Version 2.0

1. Downloading http://www.google.com/robots.txt
2. "robots.txt" saved as "www.google.com-robots.txt"
3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080
/catalogs/about sent
/catalogs/p? sent
/news/directory sent
...
4. Done.

cmlh$
</pre>

'''<META> Tags - with Burp''' 

Based on the Disallow directive(s) listed within the robots.txt file in webroot, a regular expression search for "<META NAME="ROBOTS"" within each web page is undertaken and the result compared to the robots.txt file in webroot.

For example, the robots.txt file from facebook.com has a "Disallow: /ac.php" entry[http://facebook.com/robots.txt] and the resulting search for "<META NAME="ROBOTS"" shown below:
 
[[File:CMLH-Meta Tag Example-Facebook-Aug 2013.png]]
 
The above might be considered a fail since "INDEX,FOLLOW" is the default <META> Tag specified by the "Robots Exclusion Protocol" yet "Disallow: /ac.php" is listed in robots.txt.

'''Analyze robots.txt using Google Webmaster Tools''' 
Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools", which can assist with testing [4] and the procedure is as follows:

1. Sign into Google Webmaster Tools with your Google Account. 
2. On the Dashboard, click the URL for the site you want. 
3. Click Tools, and then click Analyze robots.txt. 

=== Gray Box testing and example ===
The process is the same as Black Box testing above.
 

== Tools ==

* Browser (View Source function)
* curl
* wget
* rockspider[https://github.com/cmlh/rockspider/releases]

== References ==
'''Whitepapers''' 
* [1] "The Web Robots Pages" - http://www.robotstxt.org/
* [2] "Block and Remove Pages Using a robots.txt File" - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=40364&rd=1
* [3] "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
* [4] "Block and Remove Pages Using a robots.txt File - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1
* [5] "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-09-03T06:53:49Z

Cmlh: Add Social Network Reference from Bridge

{{Template:OWASP Testing Guide v4}}

== Summary ==
This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009[https://www.owasp.org/index.php/Testing_Map_execution_paths_through_application_(OWASP-IG-009)]

== Test Objectives ==
1. Information Leakage of the web application's directory/folder path(s).

2. Create the list of directories that are to be avoided by Spiders/Robots/Crawlers

== How to Test ==
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the ''Robots Exclusion Protocol'' of the robots.txt file in the web root directory [1].
 

'''robots.txt in webroot''' 
As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
<pre>
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

The ''User-Agent'' directive refers to the specific web spider/robot/crawler. For example the ''User-Agent: Googlebot'' refers to the ''GoogleBot'' crawler while ''User-Agent: *'' in the example above applies to all web spiders/robots/crawlers [2] as quoted below:
<pre>
User-agent: *
</pre>

The ''Disallow'' directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
<pre>
...
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>
Web spiders/robots/crawlers can intentionally ignore the ''Disallow'' directives specified in a robots.txt file [3], such as those from Social Networks[https://www.htbridge.com/news/social_networks_can_robots_violate_user_privacy.html] to ensure that shared linked are still valid. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
 

'''<META> Tag''' 

<META> tags are located within the HEAD section of each HTML Document and should be consistent across a web site in the likely event that the robot/spider/crawler start point does not begin from a document link other than webroot i.e. a "deep link"[http://en.wikipedia.org/wiki/Deep_linking].

If there is no "<META NAME="ROBOTS"" ... >" entry than the "Robots Exclusion Protocol'' defaults to "INDEX,FOLLOW" respectively. Therefore, the other two valid entries defined by the "Robots Exclusion Protocol'' are prefixed with "NO..." i.e. "NOINDEX" and "NOFOLLOW".

Web spiders/robots/crawlers can intentionally ignore the "<META NAME="ROBOTS"" tag as the robots.txt file convention is preferred. Hence, <META> Tags should not be considered the primary mechanism, rather a complementary control to robots.txt.

=== Black Box testing and example ===
'''robots.txt in webroot - with "wget" or "curl" 
The robots.txt file is retrieved from the web root directory of the web server.

For example, to retrieve the robots.txt from www.google.com using ''wget'' or "curl":
<pre>
cmlh$ wget http://www.google.com/robots.txt
--2013-08-11 14:40:36-- http://www.google.com/robots.txt
Resolving www.google.com... 74.125.237.17, 74.125.237.18, 74.125.237.19, ...
Connecting to www.google.com|74.125.237.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘robots.txt.1’

[ <=> ] 7,074 --.-K/s in 0s

2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved [7074]

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
<pre>
cmlh$ curl -O http://www.google.com/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
'''robots.txt in webroot - with rockspider''' 
"rockspider"[https://github.com/cmlh/rockspider/releases] automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site

For example, to create the initial scope based on the Allowed: directive from www.google.com using "rockspider"[https://github.com/cmlh/rockspider/releases]:
<pre>
cmlh$ ./rockspider.pl -www www.google.com

"Rockspider" Alpha v0.1_2

Copyright 2013 Christian Heinrich
Licensed under the Apache License, Version 2.0

1. Downloading http://www.google.com/robots.txt
2. "robots.txt" saved as "www.google.com-robots.txt"
3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080
/catalogs/about sent
/catalogs/p? sent
/news/directory sent
...
4. Done.

cmlh$
</pre>

'''<META> Tags - with Burp''' 

Based on the Disallow directive(s) listed within the robots.txt file in webroot, a regular expression search for "<META NAME="ROBOTS"" within each web page is undertaken and the result compared to the robots.txt file in webroot.

For example, the robots.txt file from facebook.com has a "Disallow: /ac.php" entry[http://facebook.com/robots.txt] and the resulting search for "<META NAME="ROBOTS"" shown below:
 
[[File:CMLH-Meta Tag Example-Facebook-Aug 2013.png]]
 
The above might be considered a fail since "INDEX,FOLLOW" is the default <META> Tag specified by the "Robots Exclusion Protocol" yet "Disallow: /ac.php" is listed in robots.txt.

'''Analyze robots.txt using Google Webmaster Tools''' 
Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools", which can assist with testing [4] and the procedure is as follows:

1. Sign into Google Webmaster Tools with your Google Account. 
2. On the Dashboard, click the URL for the site you want. 
3. Click Tools, and then click Analyze robots.txt. 

=== Gray Box testing and example ===
The process is the same as Black Box testing above.
 

== Tools ==

* Browser (View Source function)
* curl
* wget
* rockspider[https://github.com/cmlh/rockspider/releases]

== References ==
'''Whitepapers''' 
* [1] "The Web Robots Pages" - http://www.robotstxt.org/
* [2] "Block and Remove Pages Using a robots.txt File" - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=40364&rd=1
* [3] "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
* [4] "Block and Remove Pages Using a robots.txt File - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1
* [5] "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-25T10:03:36Z

Cmlh: Refactor rockspider item

{{Template:OWASP Testing Guide v4}}

== Summary ==
This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009[https://www.owasp.org/index.php/Testing_Map_execution_paths_through_application_(OWASP-IG-009)]

== Test Objectives ==
1. Information Leakage of the web application's directory/folder path(s).

2. Create the list of directories that are to be avoided by Spiders/Robots/Crawlers

== How to Test ==
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the ''Robots Exclusion Protocol'' of the robots.txt file in the web root directory [1].
 

'''robots.txt in webroot''' 
As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
<pre>
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

The ''User-Agent'' directive refers to the specific web spider/robot/crawler. For example the ''User-Agent: Googlebot'' refers to the ''GoogleBot'' crawler while ''User-Agent: *'' in the example above applies to all web spiders/robots/crawlers [2] as quoted below:
<pre>
User-agent: *
</pre>

The ''Disallow'' directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
<pre>
...
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>
Web spiders/robots/crawlers can intentionally ignore the ''Disallow'' directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
 

'''<META> Tag''' 

<META> tags are located within the HEAD section of each HTML Document and should be consistent across a web site in the likely event that the robot/spider/crawler start point does not begin from a document link other than webroot i.e. a "deep link"[http://en.wikipedia.org/wiki/Deep_linking].

If there is no "<META NAME="ROBOTS"" ... >" entry than the "Robots Exclusion Protocol'' defaults to "INDEX,FOLLOW" respectively. Therefore, the other two valid entries defined by the "Robots Exclusion Protocol'' are prefixed with "NO..." i.e. "NOINDEX" and "NOFOLLOW".

Web spiders/robots/crawlers can intentionally ignore the "<META NAME="ROBOTS"" tag as the robots.txt file convention is preferred. Hence, <META> Tags should not be considered the primary mechanism, rather a complementary control to robots.txt.

=== Black Box testing and example ===
'''robots.txt in webroot - with "wget" or "curl" 
The robots.txt file is retrieved from the web root directory of the web server.

For example, to retrieve the robots.txt from www.google.com using ''wget'' or "curl":
<pre>
cmlh$ wget http://www.google.com/robots.txt
--2013-08-11 14:40:36-- http://www.google.com/robots.txt
Resolving www.google.com... 74.125.237.17, 74.125.237.18, 74.125.237.19, ...
Connecting to www.google.com|74.125.237.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘robots.txt.1’

[ <=> ] 7,074 --.-K/s in 0s

2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved [7074]

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
<pre>
cmlh$ curl -O http://www.google.com/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
'''robots.txt in webroot - with rockspider''' 
"rockspider"[https://github.com/cmlh/rockspider/releases] automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site

For example, to create the initial scope based on the Allowed: directive from www.google.com using "rockspider"[https://github.com/cmlh/rockspider/releases]:
<pre>
cmlh$ ./rockspider.pl -www www.google.com

"Rockspider" Alpha v0.1_2

Copyright 2013 Christian Heinrich
Licensed under the Apache License, Version 2.0

1. Downloading http://www.google.com/robots.txt
2. "robots.txt" saved as "www.google.com-robots.txt"
3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080
/catalogs/about sent
/catalogs/p? sent
/news/directory sent
...
4. Done.

cmlh$
</pre>

'''<META> Tags - with Burp''' 

Based on the Disallow directive(s) listed within the robots.txt file in webroot, a regular expression search for "<META NAME="ROBOTS"" within each web page is undertaken and the result compared to the robots.txt file in webroot.

For example, the robots.txt file from facebook.com has a "Disallow: /ac.php" entry[http://facebook.com/robots.txt] and the resulting search for "<META NAME="ROBOTS"" shown below:
 
[[File:CMLH-Meta Tag Example-Facebook-Aug 2013.png]]
 
The above might be considered a fail since "INDEX,FOLLOW" is the default <META> Tag specified by the "Robots Exclusion Protocol" yet "Disallow: /ac.php" is listed in robots.txt.

'''Analyze robots.txt using Google Webmaster Tools''' 
Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools", which can assist with testing [4] and the procedure is as follows:

1. Sign into Google Webmaster Tools with your Google Account. 
2. On the Dashboard, click the URL for the site you want. 
3. Click Tools, and then click Analyze robots.txt. 

=== Gray Box testing and example ===
The process is the same as Black Box testing above.
 

== Tools ==

* Browser (View Source function)
* curl
* wget
* rockspider[https://github.com/cmlh/rockspider/releases]

== References ==
'''Whitepapers''' 
* [1] "The Web Robots Pages" - http://www.robotstxt.org/
* [2] "Block and Remove Pages Using a robots.txt File" - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=40364&rd=1
* [3] "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
* [4] "Block and Remove Pages Using a robots.txt File - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1
* [5] "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-25T10:02:17Z

Cmlh: Add <META> Tag Section - some markup incorrect

{{Template:OWASP Testing Guide v4}}

== Summary ==
This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009[https://www.owasp.org/index.php/Testing_Map_execution_paths_through_application_(OWASP-IG-009)]

== Test Objectives ==
1. Information Leakage of the web application's directory/folder path(s).

2. Create the list of directories that are to be avoided by Spiders/Robots/Crawlers

== How to Test ==
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the ''Robots Exclusion Protocol'' of the robots.txt file in the web root directory [1].
 

'''robots.txt in webroot''' 
As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
<pre>
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

The ''User-Agent'' directive refers to the specific web spider/robot/crawler. For example the ''User-Agent: Googlebot'' refers to the ''GoogleBot'' crawler while ''User-Agent: *'' in the example above applies to all web spiders/robots/crawlers [2] as quoted below:
<pre>
User-agent: *
</pre>

The ''Disallow'' directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
<pre>
...
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>
Web spiders/robots/crawlers can intentionally ignore the ''Disallow'' directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
 

'''<META> Tag''' 

<META> tags are located within the HEAD section of each HTML Document and should be consistent across a web site in the likely event that the robot/spider/crawler start point does not begin from a document link other than webroot i.e. a "deep link"[http://en.wikipedia.org/wiki/Deep_linking].

If there is no "<META NAME="ROBOTS"" ... >" entry than the "Robots Exclusion Protocol'' defaults to "INDEX,FOLLOW" respectively. Therefore, the other two valid entries defined by the "Robots Exclusion Protocol'' are prefixed with "NO..." i.e. "NOINDEX" and "NOFOLLOW".

Web spiders/robots/crawlers can intentionally ignore the "<META NAME="ROBOTS"" tag as the robots.txt file convention is preferred. Hence, <META> Tags should not be considered the primary mechanism, rather a complementary control to robots.txt.

=== Black Box testing and example ===
'''robots.txt in webroot - with "wget" or "curl" 
The robots.txt file is retrieved from the web root directory of the web server.

For example, to retrieve the robots.txt from www.google.com using ''wget'' or "curl":
<pre>
cmlh$ wget http://www.google.com/robots.txt
--2013-08-11 14:40:36-- http://www.google.com/robots.txt
Resolving www.google.com... 74.125.237.17, 74.125.237.18, 74.125.237.19, ...
Connecting to www.google.com|74.125.237.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘robots.txt.1’

[ <=> ] 7,074 --.-K/s in 0s

2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved [7074]

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
<pre>
cmlh$ curl -O http://www.google.com/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
'''robots.txt in webroot - with rockspider''' 
"rockspider"[https://github.com/cmlh/rockspider/releases] automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site

For example, to create the initial scope based on the Allowed: directive from www.google.com using "rockspider"[https://github.com/cmlh/rockspider/releases]:
<pre>
cmlh$ ./rockspider.pl -www www.google.com

"Rockspider" Alpha v0.1_2

Copyright 2013 Christian Heinrich
Licensed under the Apache License, Version 2.0

1. Downloading http://www.google.com/robots.txt
2. "robots.txt" saved as "www.google.com-robots.txt"
3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080
/catalogs/about sent
/catalogs/p? sent
/news/directory sent
...
4. Done.

cmlh$
</pre>

'''<META> Tags - with Burp''' 

Based on the Disallow directive(s) listed within the robots.txt file in webroot, a regular expression search for "<META NAME="ROBOTS"" within each web page is undertaken and the result compared to the robots.txt file in webroot.

For example, the robots.txt file from facebook.com has a "Disallow: /ac.php" entry[http://facebook.com/robots.txt] and the resulting search for "<META NAME="ROBOTS"" shown below:
 
[[File:CMLH-Meta Tag Example-Facebook-Aug 2013.png]]
 
The above might be considered a fail since "INDEX,FOLLOW" is the default <META> Tag specified by the "Robots Exclusion Protocol" yet "Disallow: /ac.php" is listed in robots.txt.

'''Analyze robots.txt using Google Webmaster Tools''' 
Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools", which can assist with testing [4] and the procedure is as follows:

1. Sign into Google Webmaster Tools with your Google Account. 
2. On the Dashboard, click the URL for the site you want. 
3. Click Tools, and then click Analyze robots.txt. 

=== Gray Box testing and example ===
The process is the same as Black Box testing above.
 

== Tools ==

* Browser (View Source function)
* curl
* wget
* Speculum[https://github.com/cmlh/Speculum/releases]

== References ==
'''Whitepapers''' 
* [1] "The Web Robots Pages" - http://www.robotstxt.org/
* [2] "Block and Remove Pages Using a robots.txt File" - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=40364&rd=1
* [3] "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
* [4] "Block and Remove Pages Using a robots.txt File - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1
* [5] "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html

File:CMLH-Meta Tag Example-Facebook-Aug 2013.png

2013-08-25T09:16:22Z

Cmlh: Image for <META> Tag example of https://www.owasp.org/index.php?title=Testing:_Review_Webserver_Metafiles_for_Information_Leakage_(OTG-INFO-003)

Image for <META> Tag example of https://www.owasp.org/index.php?title=Testing:_Review_Webserver_Metafiles_for_Information_Leakage_(OTG-INFO-003)

Talk:Conduct search engine discovery/reconnaissance for information leakage (OTG-INFO-001)

2013-08-20T02:33:37Z

Cmlh: Reply from @cmlh i.e. ::::: CMLH

__TOC__

== v3 Review Comments ==
This section does not cover the items stated in the "brief summary".
For v3, if the section is to remain completely google'centric I suggest we rename "Search engine discovery" to "Google searching your web application and accessing google's cache".

== Reply to "v3 Review Comments" from @cmlh ==
The roadmap was to add Yahoo! and Bing to the next release of the OWASP Testing Guide (i.e. v3 -> v4) and to not appear to promote Google over Yahoo! and Bing. It should be noted that Yahoo! and Bing might refer to the same "entity" as further research is undertaken i.e. the "Yahoo! and Microsoft Search Alliance"/"Yahoo! Bing Network".

Furthermore, the intent is *not* to promote the inferior http://www.hackersforcharity.org/ghdb/, rather a more scientific and innovative approach.

: Hi cmlh, thanks for the follow-up. That comment was really old and seems to have been migrated for the v3 > v4 draft. I think the new heading/title is more appropriate than previously, however, the content still seems awfully google'centric.

: Should we also be including some Shodan stuff? (http://www.shodanhq.com/) [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]])

:: Actually now that I'm looking at this. I'm not sure how the heading has changed since v3 was a draft (when the comment was originally made). However, again looking at this now there are a number of goals, etc stated in the summary that don't seem to be covered by the content. Also the summary seems to be written from the perspective of a app/system owner not a tester.

:: I also wonder if we should be including examples such as xssed.com and their ilk, web.archive.org, etc [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]])

Adding web services, such as xxsed.com or web.archive.org, would depend on if they an API available to the public (I believe archive.org has and API) and if there is a product available (possibly released under FOSS licenses) to provide an example.

: IMHO that only applies from a purely automated point of view. There is no reason we shouldn't be referencing such as a manual step (or steps).

:: CMLH - I am aware of archive.org, I am not sure about the xxsed example that you are referring too?

:::: xssed.com is a community site that catalogs (by submission) vulnerabilities found on public sites. If I were doing a Web App test for a client I'd look to see if anyone else had reported an issue for their site. Having vulnerabilities in your web app publicly outted seems like a serious information leak to me.

::::: CMLH - I know what xssed is already :) Instead I meant did they offer a Public API (which is not the case as far as I am aware). That stated, a loud minority of OWASP members may complain that we are promoting an unethical web site but I have no issue with including xssed (perhaps in another related section of the testing guide).

: I'm not sure how the majority of the industry ends up getting involved in Web App VA but from my perspective and experience there is usually limited targets so doing a few manual lookups isn't a major stumbling block.

:: CMLH - I believe some of these other services might be out of scope of OTG-INFO-001.

::: Shodan is a search engine specifically designed to catalog systems and version/configuration information. Finding listings for your target client/app with Shodan can provide further information about your target.

:::: CMLH - I believe some of these other services might still be out of scope of OTG-INFO-001 in light of the clarification above but could be addressed in a related section of v4.

::: Further the summary for 001 talks about " Indirect methods relate to gleaning sensitive design and configuration information by searching forums, newsgroups and tendering websites." none of which is actually covered. There is a huge open source intelligence gathering activity which should be covered based on that statement. (Finding related employees on linkedin, searching google groups for SW and HW questions, etc) [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]]) 07:23, 19 August 2013 (CDT)

:::: CMLH - That section was added *after* I had contributed to v3 i.e. http://lists.owasp.org/pipermail/owasp-testing/2013-August/002160.html

:::: Oh and if it matters somehow Shodan does have an API. Though I stick by the statement that not everything has to be automated to automate'able :) Here's one small intro for it: http://raidersec.blogspot.ca/2012/02/searching-for-devices-using-shodan.html [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]]) 07:27, 19 August 2013 (CDT)

::::: CMLH - Yes, I am aware of their API i.e. http://cmlh.id.au/tagged/shodan

Talk:Conduct search engine discovery/reconnaissance for information leakage (OTG-INFO-001)

2013-08-16T00:38:16Z

Cmlh: Response to Rick

Talk:Conduct search engine discovery/reconnaissance for information leakage (OTG-INFO-001)

2013-08-15T23:17:21Z

Cmlh: Draft Reply to Rick Mitchell

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-15T23:12:57Z

Cmlh: Draft response to Rick Mitchell

__TOC__

== Discussion ==
It could be added that, from an attacker point of view, the robots.txt file can provide some useful information on the structure of the web server, e.g., directories that are supposed to be "private".

[[User:Marco|Marco]] 18:11, 17 August 2008 (EDT)

The intent of robots.txt is *not* to specify access control for directories. Hence to quote the wiki page "''Web spiders/robots/crawlers can intentionally ignore the Disallow directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.''".

If you believe this is not your communicated clearly or could be reworded then please amend the wiki page.

[[User:cmlh|cmlh]] 12:34, 24 August 2008 (GMT +10)

== v3 Review Comments ==

I don't see anything here about actually testing robots.txt or using Spiders/Robots/Crawlers to do anything to the web app. It's nice that we can DL the file and that it contains some interesting information and that there's a google tool that can do some analysis of it (though we haven't explained what google webmaster tools gives you or provided an example of the output), but where would that lead a tester or attacker? 
[[User:Rick.mitchell|Rick.mitchell]] 09:39, 3 September 2008 (EDT)

== Reply from @cmlh ==

Rick may have overlooked the quote "Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties. " from in the "How to Test" section of v3.

The lack of the "Google Webmaster Tools" example is due to me not being the webmaster of owasp.org. This can be resolved in v4 once the webmaster is known.

: For the first part, either that sentence wasn't part of the content I reviewed during v3 draft or it didn't seem significant enough given the lead-in.

: As for the webmaster tools stuff that sounds good, however like INFO-001 seems awfully google-centric.

CH - Bing/Yahoo! have been included in the roadmap for v4.

: This content also only covers robots.txt though the heading suggests much broader coverage. So either the content should be expanded or the heading made more specific (IMHO).

CH - I included http://www.robotstxt.org/meta.html in the OWASP Testing Guide v3 (since I also presented on this content in 2009 and 2010). I am not sure if it was removed by subsequent edits by others (I haven't checked this) but I will include it for v4 again.
[[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]]) 15:07, 15 August 2013 (CDT)

== TODO for v4 ==

1. Insert the "Analyze robots.txt using Google Webmaster Tools" i.e. https://support.google.com/webmasters/answer/156449?hl=en&from=35237&rd=1 with owasp.org (not applicable, since webroot doesn't contain robots.txt) as the example.

2. May need to update the reference to OWASP-IG-009 within the "Summary" section depending on the finalisation of the spidering thread (To be created).

3. Add Bing/Yahoo! related content.

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-15T21:53:22Z

Cmlh: Update status of robots.txt within webroot of owasp.org

__TOC__

== Discussion ==
It could be added that, from an attacker point of view, the robots.txt file can provide some useful information on the structure of the web server, e.g., directories that are supposed to be "private".

[[User:Marco|Marco]] 18:11, 17 August 2008 (EDT)

The intent of robots.txt is *not* to specify access control for directories. Hence to quote the wiki page "''Web spiders/robots/crawlers can intentionally ignore the Disallow directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.''".

If you believe this is not your communicated clearly or could be reworded then please amend the wiki page.

[[User:cmlh|cmlh]] 12:34, 24 August 2008 (GMT +10)

== v3 Review Comments ==

I don't see anything here about actually testing robots.txt or using Spiders/Robots/Crawlers to do anything to the web app. It's nice that we can DL the file and that it contains some interesting information and that there's a google tool that can do some analysis of it (though we haven't explained what google webmaster tools gives you or provided an example of the output), but where would that lead a tester or attacker? 
[[User:Rick.mitchell|Rick.mitchell]] 09:39, 3 September 2008 (EDT)

== Reply from @cmlh ==

Rick may have overlooked the quote "Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties. " from in the "How to Test" section of v3.

The lack of the "Google Webmaster Tools" example is due to me not being the webmaster of owasp.org. This can be resolved in v4 once the webmaster is known.

: For the first part, either that sentence wasn't part of the content I reviewed during v3 draft or it didn't seem significant enough given the lead-in.

: As for the webmaster tools stuff that sounds good, however like INFO-001 seems awfully google-centric.

: This content also only covers robots.txt though the heading suggests much broader coverage. So either the content should be expanded or the heading made more specific (IMHO). [[User:Rick.mitchell|Rick.mitchell]] ([[User talk:Rick.mitchell|talk]]) 15:07, 15 August 2013 (CDT)

== TODO for v4 ==

1. Insert the "Analyze robots.txt using Google Webmaster Tools" i.e. https://support.google.com/webmasters/answer/156449?hl=en&from=35237&rd=1 with owasp.org (not applicable, since webroot doesn't contain robots.txt) as the example.

2. May need to update the reference to OWASP-IG-009 within the "Summary" section depending on the finalisation of the spidering thread (To be created).

3. Add Bing/Yahoo! related content.

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-15T03:04:53Z

Cmlh: Added Speculum example

{{Template:OWASP Testing Guide v4}}

== Summary ==
This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009[https://www.owasp.org/index.php/Testing_Map_execution_paths_through_application_(OWASP-IG-009)]

== Test Objectives ==
1. Information Leakage of the web application's directory/folder path(s).

2. Create the list of directories that are to be avoided by Spiders/Robots/Crawlers

== How to Test ==
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the ''Robots Exclusion Protocol'' of the robots.txt file in the web root directory [1].

As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
<pre>
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

The ''User-Agent'' directive refers to the specific web spider/robot/crawler. For example the ''User-Agent: Googlebot'' refers to the ''GoogleBot'' crawler while ''User-Agent: *'' in the example above applies to all web spiders/robots/crawlers [2] as quoted below:
<pre>
User-agent: *
</pre>

The ''Disallow'' directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
<pre>
...
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>
Web spiders/robots/crawlers can intentionally ignore the ''Disallow'' directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
 

=== Black Box testing and example ===
'''wget''' 
The robots.txt file is retrieved from the web root directory of the web server.

For example, to retrieve the robots.txt from www.google.com using ''wget'' or "curl":
<pre>
cmlh$ wget http://www.google.com/robots.txt
--2013-08-11 14:40:36-- http://www.google.com/robots.txt
Resolving www.google.com... 74.125.237.17, 74.125.237.18, 74.125.237.19, ...
Connecting to www.google.com|74.125.237.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘robots.txt.1’

[ <=> ] 7,074 --.-K/s in 0s

2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved [7074]

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
<pre>
cmlh$ curl -O http://www.google.com/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
'''Speculum''' 
"Speculum"[https://github.com/cmlh/Speculum/releases] automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site

For example, to create the initial scope based on the Allowed: directive from www.google.com using "Speculum"[https://github.com/cmlh/Speculum/releases]:
<pre>
cmlh$ ./speculum.pl -www www.google.com

"Speculum" Alpha v0.0_2

Copyright 2013 Christian Heinrich
Licensed under the Apache License, Version 2.0

1. Downloading http://www.google.com/robots.txt
2. "robots.txt" saved as "www.google.com-robots.txt"
3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080
/catalogs/about sent
/catalogs/p? sent
/news/directory sent
...
4. Done.

cmlh$
</pre>

'''Analyze robots.txt using Google Webmaster Tools''' 
Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools", which can assist with testing [4] and the procedure is as follows:

1. Sign into Google Webmaster Tools with your Google Account. 
2. On the Dashboard, click the URL for the site you want. 
3. Click Tools, and then click Analyze robots.txt. 

=== Gray Box testing and example ===
The process is the same as Black Box testing above.
 

== Tools ==

* Browser (View Source function)
* curl
* wget
* Speculum[https://github.com/cmlh/Speculum/releases]

== References ==
'''Whitepapers''' 
* [1] "The Web Robots Pages" - http://www.robotstxt.org/
* [2] "Block and Remove Pages Using a robots.txt File" - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=40364&rd=1
* [3] "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
* [4] "Block and Remove Pages Using a robots.txt File - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1
* [5] "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-15T03:04:05Z

Cmlh: Inserted example of "Speculum"

{{Template:OWASP Testing Guide v4}}

== Summary ==
This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009[https://www.owasp.org/index.php/Testing_Map_execution_paths_through_application_(OWASP-IG-009)]

== Test Objectives ==
1. Information Leakage of the web application's directory/folder path(s).

2. Create the list of directories that are to be avoided by Spiders/Robots/Crawlers

== How to Test ==
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the ''Robots Exclusion Protocol'' of the robots.txt file in the web root directory [1].

As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
<pre>
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>

The ''User-Agent'' directive refers to the specific web spider/robot/crawler. For example the ''User-Agent: Googlebot'' refers to the ''GoogleBot'' crawler while ''User-Agent: *'' in the example above applies to all web spiders/robots/crawlers [2] as quoted below:
<pre>
User-agent: *
</pre>

The ''Disallow'' directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
<pre>
...
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
...
</pre>
Web spiders/robots/crawlers can intentionally ignore the ''Disallow'' directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
 

=== Black Box testing and example ===
'''wget''' 
The robots.txt file is retrieved from the web root directory of the web server.

For example, to retrieve the robots.txt from www.google.com using ''wget'' or "curl":
<pre>
cmlh$ wget http://www.google.com/robots.txt
--2013-08-11 14:40:36-- http://www.google.com/robots.txt
Resolving www.google.com... 74.125.237.17, 74.125.237.18, 74.125.237.19, ...
Connecting to www.google.com|74.125.237.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘robots.txt.1’

[ <=> ] 7,074 --.-K/s in 0s

2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved [7074]

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
<pre>
cmlh$ curl -O http://www.google.com/robots.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312

cmlh$ head -n5 robots.txt
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
cmlh$
</pre>
'''Speculum''' 
"Speculum"[https://github.com/cmlh/Speculum/releases] automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site

For example, to create the initial scope based on the Allowed: directive from www.google.com using "Speculum"[https://github.com/cmlh/Speculum/releases]:
<pre>
cmlh$ ./speculum.pl -www www.google.com

"Speculum" Alpha v0.0_2

Copyright 2013 Christian Heinrich
Licensed under the Apache License, Version 2.0

1. Downloading http://www.google.com/robots.txt
2. "robots.txt" saved as "www.google.com-robots.txt"
3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080
/catalogs/about sent
/catalogs/p? sent
/news/directory sent
...
4. Done.

cmlh$
</pre>

'''Analyze robots.txt using Google Webmaster Tools''' 
Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools", which can assist with testing [4] and the procedure is as follows:

1. Sign into Google Webmaster Tools with your Google Account. 
2. On the Dashboard, click the URL for the site you want. 
3. Click Tools, and then click Analyze robots.txt. 

=== Gray Box testing and example ===
The process is the same as Black Box testing above.
 

== Tools ==

* Browser (View Source function)
* curl
* wget

== References ==
'''Whitepapers''' 
* [1] "The Web Robots Pages" - http://www.robotstxt.org/
* [2] "Block and Remove Pages Using a robots.txt File" - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=40364&rd=1
* [3] "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
* [4] "Block and Remove Pages Using a robots.txt File - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1
* [5] "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-11T05:19:28Z

Cmlh: Add "3. Add Bing/Yahoo! related content"

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-11T05:16:53Z

Cmlh: /* TODO for v4 */ new section

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-11T04:49:46Z

Cmlh: Add [5]

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-11T04:43:50Z

Cmlh: Initial DRAFT for v4

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-11T04:31:39Z

Cmlh: 1st DRAFT for v4

Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-11T04:27:15Z

Cmlh: 1st DRAFT for v4, might need to update OWASP-IG-009 reference in FINAL

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

2013-08-08T04:45:10Z

Cmlh: Draft reply from @cmlh

Talk:Conduct search engine discovery/reconnaissance for information leakage (OTG-INFO-001)

2013-08-08T03:58:25Z

Cmlh: Draft reply from @cmlh

OpenSAMM Adopters

2013-07-07T05:43:23Z

Cmlh: Added ISG

[[Category:Software Assurance Maturity Model]]
===List of Organizations Using OpenSAMM===

{|class="wikitable sortable" style="text-align: top;" border="1" cellpadding="2"
!width="10%" |Organization Name
!width="10%" |Contact
!width="10%" |Role
!width="10%" |Organization Type ([http://en.wikipedia.org/wiki/Vertical_market *])
!width="10%" |Region
!width="40%" |Testimonial
|-
| Dell, Inc. || Michael J. Craigue || Information Security & Compliance || Technology || US || ''OWASP.org is a valuable resource for any company involved with online payment card transactions. Dell uses OWASP’s Software Assurance Maturity Model (OpenSAMM) to help focus our resources and determine which components of our secure application development program to prioritize. Participation in OWASP’s local chapter meetings and conferences around the globe helps us build stronger networks with our colleagues.''
|-
| KBC || Johan Jacobs || ICT Department Head || Banking || Europe || -
|-
| Gotham Digital Science || Matt Bartoldus || Co-Founder & Director || Security services || Global || ''SAMM has defined the building blocks for effective software security assurance… Our clients can use the model to see what needs to be done and what skills and resources are needed to do the job. Best of all, businesses can use SAMM to quantify results and improvements by assessing practices against SAMM activities.''
|-
| Fortify Software || Brian Chess || Founder & Chief Scientist || Security services || Global || ''These days people understand that security has to be built in–it can’t be bolted on. But for many a big question remains: what does it take to build secure software? SAMM tackles that question head on with a framework for creating and growing a software security initiative. SAMM has focused the way I think about the human side of the software security problem.''
|-
| ING Insurance International || Rob Moes || IT Security Manager || Insurance || Europe || ''Within ING Insurance International we adopted SAMM as it is a practical standard which provides guidance to build an Secure Application Development organization in clear and distinctive steps.''
|-
| ISG || Christian Heinrich || Application Security Manager || Health || Australia || ''ISG has integrated both OpenSAMM and BSIMM to measure security improvement over time in addition to our overall measurement of the "Capability Maturity Model for Software Development" published by Carnegie Mellon University".
|-
|<Fill in Organisation Name> || <Fill in Contact First Name, Family Name> || <Fill in Contact role in the organisation>|| <Fill in Organisation Type: Government, Finance, Healthcare, ...>|| <Fill in Region: Continent, Country>|| ''<Fill in Contact Testimonial - OPTIONAL>''
|}

Issues Concerning The OWASP Top Ten 2013

2013-06-10T01:00:24Z

Cmlh: http://lists.owasp.org/pipermail/owasp-topten/2013-June/001108.html

INTRODUCTION

The Terms of References for the "OWASP Top Ten Code of Ethics violations and project handbook." agenda item i.e. https://www.owasp.org/index.php/June_10,_2013 are specified below.

There are several complaints made against Aspect Security by OWASP Members, including http://lists.owasp.org/pipermail/owasp-leaders/2013-June/009432.html, http://lists.owasp.org/pipermail/owasp-topten/2013-May/date.html, etc

Each numbered item has been grouped into themes based on headings below:

A9 AND SONATYPE

1. The are several external complaints from OWASP members stating that the Sonatype/Aspect Security statistics are unscientific and bias and some examples are:
* GWT i.e. https://groups.google.com/forum/?fromgroups#!topic/google-web-toolkit/Ezr6acdyZv0
* SpringSource i.e. http://www.infosecurity-magazine.com/view/30282/remote-code-vulnerability-in-spring-framework-for-java/. Furthermore, as the disclosure by Aspect Security occurred in January 2013 this conflicts with their statement that the statistics were sampled well before 2013.

2. Aspect Security have promoted both AntiSammy and ESAPI in A1 or A3 which they also hold the Project Leadership of. However, their paid research for Sonatype states that their insecure releases are still being downloaded. Therefore, OWASP is placed inm until recently, unknown catastrophic residual risk as it appears that OWASP is hypocritical in not following their own recommendation i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-June/001095.html

3. The residual risk of A9 will be accepted by the developer due to the significant cost with change i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-February/000844.html

4. A9 does not direct the reader to other related open source projects, such as https://github.com/gcmurphy/enforce-victims-rule, https://github.com/jeremylong/DependencyCheck, etc

5. The Press Release from Sonatype quotes Jeff Williams and was not approved under the OWASP Quotes process which he also championed as an OWASP Board Member. Furthermore, Aspect Security did not attempt to inform the OWASP Foundation once they were alerted to the publication of the Press Release i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001017.html

6. Aspect Security hosted a Chapter Meeting on 6 June to promote Sonatype and A9 before the actual 2013 release was accepted by the webappsec community i.e. http://www.meetup.com/OWASP-Baltimore-Chapter/events/119389612/

OTHER SOURCES OF STATISTICS

7. The statistics from both WhiteHat and HP (i.e. Fortify and WebInspect) require registration. <del>Dave Wichers of Aspect Security has *not* published the promised alternate links i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001041.html</del>

8. Statistics from either Trustwave<del>, Softek or</del> Minded Security were *not* analysed as this would have resulted in a second release of the RC or at least notification of the result i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001054.html and http://lists.owasp.org/pipermail/owasp-topten/2013-May/001080.html

9. Aspect Security have not published their statistical analysis i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-June/001096.html For comparison purposes, Minded Security were able to publish their effort within less than a month (28 January to 19 February).

2010 RELEASE

10. Softek are *not* listed as a sponsor within the pages of the 2010 deliverable as Aspect Security have taken this space for their own enlarged company logo i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001039.html

ABUSE FROM ARSHAN DABIRSIAGHI OF ASPECT SECURITY

11. The formal complaint is available from http://lists.owasp.org/pipermail/owasp-topten/2013-June/001099.html

Issues Concerning The OWASP Top Ten 2013

2013-06-09T13:57:39Z

Cmlh: Initial Draft

INTRODUCTION

The Terms of References for the "OWASP Top Ten Code of Ethics violations and project handbook." agenda item i.e. https://www.owasp.org/index.php/June_10,_2013 are specified below.

There are several complaints made against Aspect Security, including http://lists.owasp.org/pipermail/owasp-leaders/2013-June/009432.html, http://lists.owasp.org/pipermail/owasp-topten/2013-May/date.html, etc

Each numbered item has been grouped into themes based on headings below:

A9 AND SONATYPE

1. The are several external complaints stating that the Sonatype/Aspect Security statistics are unscientific and bias and some examples are:
* GWT i.e. https://groups.google.com/forum/?fromgroups#!topic/google-web-toolkit/Ezr6acdyZv0
* SpringSource i.e. http://www.infosecurity-magazine.com/view/30282/remote-code-vulnerability-in-spring-framework-for-java/. Furthermore, as the disclosure by Aspect Security occurred in January 2013 this conflicts with their statement that the statistics were sampled well before 2013.

2. Aspect Security have promoted both AntiSammy and ESAPI in A1 or A3 which they also hold the Project Leadership of. However, their paid research for Sonatype states that their insecure releases are still being downloaded. Therefore, OWASP is placed inm until recently, unknown catastrophic residual risk as it appears that OWASP is hypocritical in not following their own recommendation i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-June/001095.html

3. The residual risk of A9 will be accepted by the developer due to the significant cost with change i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-February/000844.html

4. A9 does not direct the reader to other related open source projects, such as https://github.com/gcmurphy/enforce-victims-rule, https://github.com/jeremylong/DependencyCheck, etc

5. The Press Release from Sonatype quotes Jeff Williams and was not approved under the OWASP Quotes process which he also championed as an OWASP Board Member. Furthermore, Aspect Security did not attempt to inform the OWASP Foundation once they were alerted to the publication of the Press Release i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001017.html

6. Aspect Security hosted a Chapter Meeting on 6 June to promote Sonatype and A9 before the actual 2013 release was accepted by the webappsec community i.e. http://www.meetup.com/OWASP-Baltimore-Chapter/events/119389612/

OTHER SOURCES OF STATISTICS

7. The statistics from both WhiteHat and HP (i.e. Fortify and WebInspect) require registration. Dave Wichers of Aspect Security has *not* published the promised alternate links i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001041.html

8. Statistics from either Trustwave, Softek or Minded Security were *not* analysed as this would have resulted in a second release of the RC or at least notification of the result i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001054.html and http://lists.owasp.org/pipermail/owasp-topten/2013-May/001080.html

9. Aspect Security have not published their statistical analysis i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-June/001096.html For comparison purposes, Minded Security were able to publish their effort within less than a month (28 January to 19 February).

2010 RELEASE

10. Softek are *not* listed as a sponsor within the pages of the deliverable as Aspect Security have taken this space for their own enlarged company logo i.e. http://lists.owasp.org/pipermail/owasp-topten/2013-May/001039.html

ABUSE FROM ARSHAN DABIRSIAGHI OF ASPECT SECURITY

11. The formal complaint is available from http://lists.owasp.org/pipermail/owasp-topten/2013-June/001099.html

User:Cmlh

2011-06-30T22:56:32Z

Cmlh: Added SlideShare/cmlh URL

=Contact=

In relation to OWASP matters, [mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org].

For matters not related to OWASP or as an Out of Band Communications Channel to his @owasp.org e-mail address, Christian Heinrich has listed multiple points of contact at [http://cmlh.id.au/contact http://cmlh.id.au/contact].

=Biography=

Christian Heinrich has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

=Contributions to OWASP=

Christian Heinrich's edits to the OWASP wiki are listed at: [[:Special:Contributions/Cmlh|Special:Contributions/Cmlh]].

==OWASP Projects==

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Leader of the [http://www.owasp.org/index.php/Category:OWASP_PCI_Project OWASP PCI Project] having previously lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP ESAPI Java WAF, [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

==OWASP Presentations==

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

Videos of these presentations are available from [http://www.google.com.au/search?tbm=vid&q=%22Christian+Heinrich%22+OWASP Google] and associated slides are available from [http://www.slideshare.net/cmlh/tag/owasp slideshare.net/cmlh]

=OWASP Board Candidate=

==Global==

While the candidates are either from USA or Europe and have contributed significantly to OWASP, I would like to highlight the many contributions made by Canada, EMEA and Asia Pacific, Central (America) and South America.

==Governance==

===Board===

I believe that during the term of a Board Member that they should disassociate themselves from leadership position of their Chapters and Projects of OWASP with the option to contribute during their term but not in a leadership capacity.

I also believe that funding for Board Members to travel should not be approved.

===Projects===

I believe that Project Leaders should be able to determine their own level of quality which the consumer can measure based on published peer review. As expected, those who require funding from OWASP to market their project or increase its quality should be subject to project management.

I believe that those who contribute to an OWASP Project should be credited as such irrespective of their employer.

==Significant Experience==

I have founded a number of groups in Australia, including Snort User Group and Australian Information Security Association with over 1000 members within Australia.

I also initiated the OWASP relationship with Mozilla during Hack in the Box Amsterdam in 2010.

== Commercial Independence==

I am not associated with any vendor and/or consultancy and therefore my agenda is *not* to exploit OWASP for commercial gain.

User:Cmlh

2011-06-28T22:25:52Z

Cmlh: Added Central America to Global

=Contact=

In relation to OWASP matters, [mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org].

For matters not related to OWASP or as an Out of Band Communications Channel to his @owasp.org e-mail address, Christian Heinrich has listed multiple points of contact at [http://cmlh.id.au/contact http://cmlh.id.au/contact].

=Biography=

Christian Heinrich has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

=Contributions to OWASP=

Christian Heinrich's edits to the OWASP wiki are listed at: [[:Special:Contributions/Cmlh|Special:Contributions/Cmlh]].

==OWASP Projects==

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Leader of the [http://www.owasp.org/index.php/Category:OWASP_PCI_Project OWASP PCI Project] having previously lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP ESAPI Java WAF, [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

==OWASP Presentations==

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

Videos of these presentations are available from [http://www.google.com.au/search?tbm=vid&q=%22Christian+Heinrich%22+OWASP Google]

=OWASP Board Candidate=

==Global==

While the candidates are either from USA or Europe and have contributed significantly to OWASP, I would like to highlight the many contributions made by Canada, EMEA and Asia Pacific, Central (America) and South America.

==Governance==

===Board===

I believe that during the term of a Board Member that they should disassociate themselves from leadership position of their Chapters and Projects of OWASP with the option to contribute during their term but not in a leadership capacity.

I also believe that funding for Board Members to travel should not be approved.

===Projects===

I believe that Project Leaders should be able to determine their own level of quality which the consumer can measure based on published peer review. As expected, those who require funding from OWASP to market their project or increase its quality should be subject to project management.

I believe that those who contribute to an OWASP Project should be credited as such irrespective of their employer.

==Significant Experience==

I have founded a number of groups in Australia, including Snort User Group and Australian Information Security Association with over 1000 members within Australia.

I also initiated the OWASP relationship with Mozilla during Hack in the Box Amsterdam in 2010.

== Commercial Independence==

I am not associated with any vendor and/or consultancy and therefore my agenda is *not* to exploit OWASP for commercial gain.

User:Cmlh

2011-06-28T05:04:22Z

Cmlh: Campaign Announcement

=Contact=

In relation to OWASP matters, [mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org].

For matters not related to OWASP or as an Out of Band Communications Channel to his @owasp.org e-mail address, Christian Heinrich has listed multiple points of contact at [http://cmlh.id.au/contact http://cmlh.id.au/contact].

=Biography=

Christian Heinrich has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

=Contributions to OWASP=

Christian Heinrich's edits to the OWASP wiki are listed at: [[:Special:Contributions/Cmlh|Special:Contributions/Cmlh]].

==OWASP Projects==

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Leader of the [http://www.owasp.org/index.php/Category:OWASP_PCI_Project OWASP PCI Project] having previously lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP ESAPI Java WAF, [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

==OWASP Presentations==

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

Videos of these presentations are available from [http://www.google.com.au/search?tbm=vid&q=%22Christian+Heinrich%22+OWASP Google]

=OWASP Board Candidate=

==Global==

While the candidates are either from North America or Europe and have contributed significantly to OWASP, I would like to highlight the many contributions made by South America, EMEA and Asia Pacific.

==Governance==

===Board===

I believe that during the term of a Board Member that they should disassociate themselves from leadership position of their Chapters and Projects of OWASP with the option to contribute during their term but not in a leadership capacity.

I also believe that funding for Board Members to travel should not be approved.

===Projects===

I believe that Project Leaders should be able to determine their own level of quality which the consumer can measure based on published peer review. As expected, those who require funding from OWASP to market their project or increase its quality should be subject to project management.

I believe that those who contribute to an OWASP Project should be credited as such irrespective of their employer.

==Significant Experience==

I have founded a number of groups in Australia, including Snort User Group and Australian Information Security Association with over 1000 members.

I also initiated the OWASP relationship with Mozilla during Hack in the Box Amsterdam in 2010.

== Commercial Independence==

I am not associated with any vendor and/or consultancy and therefore my agenda is *not* to exploit OWASP for commercial gain.

User:Cmlh

2011-06-16T07:15:01Z

Cmlh: Amended Headings i.e. Biography and Contributions to OWASP. Consider listing mailing list contributions at a later date.

User:Cmlh

2011-06-15T05:01:08Z

Cmlh: Added Headings

User:Cmlh

2011-06-15T04:56:27Z

Cmlh: Relocated "Christian Heinrich's edits to the OWASP wiki are listed at: Special:Contributions/Cmlh" from Top of Page by Tom Brennan

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Leader of the [http://www.owasp.org/index.php/Category:OWASP_PCI_Project OWASP PCI Project] having previously lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP ESAPI Java WAF, [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

Christian Heinrich's edits to the OWASP wiki are listed at: [[:Special:Contributions/Cmlh|Special:Contributions/Cmlh]].

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

In relation to OWASP matters, [mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org].

For matters not related to OWASP or as an Out of Band Communications Channel to his @owasp.org e-mail address, Christian Heinrich has listed multiple points of contact at [http://cmlh.id.au/contact http://cmlh.id.au/contact] and a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

=OWASP Board Candidate 2011=

TBC

User:Cmlh

2011-05-26T06:14:17Z

Cmlh: Separated owasp.org and cmlh.id.au contact information into two separate paragraphs

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Leader of the [http://www.owasp.org/index.php/Category:OWASP_PCI_Project OWASP PCI Project] having previously lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP ESAPI Java WAF, [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

In relation to OWASP matters, [mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org].

For matters not related to OWASP or as an Out of Band Communications Channel to his @owasp.org e-mail address, Christian Heinrich has listed multiple points of contact at [http://cmlh.id.au/contact http://cmlh.id.au/contact] and a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

User:Cmlh

2011-05-24T02:52:04Z

Cmlh: Corrected reference to OWASP ESAPI Java WAF

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Leader of the [http://www.owasp.org/index.php/Category:OWASP_PCI_Project OWASP PCI Project] having previously lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP ESAPI Java WAF, [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

[mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org] and has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

User:Cmlh

2011-05-24T02:50:53Z

Cmlh: Amended as Leader of OWASP PCI Project and added contribution to ESAPI Java WAF.

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Leader of the [http://www.owasp.org/index.php/Category:OWASP_PCI_Project OWASP PCI Project] having previously lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP, EASPI Java WAF, [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

[mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org] and has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

Membership/2011Election

2011-05-24T02:47:29Z

Cmlh: Indented cmlh entry for list format

=== About OWASP Foundation - [http://www.owasp.org/index.php/About_OWASP Click Here] ===
=== 2011 Candidates ===
The [http://waybackmachine.org/jsp/Interstitial.jsp?seconds=5&date=1009278073000&url=http%3A%2F%2Fwww.owasp.org%2Fabout_owasp%2Forgchart.shtml&target=http%3A%2F%2Freplay.waybackmachine.org%2F20011225110113%2Fhttp%3A%2F%2Fwww.owasp.org%2Fabout_owasp%2Forgchart.shtml OWASP Foundation, est. 2001], the Board of Directors consists of six board seats that are tasked with keeping the organization on [https://www.owasp.org/index.php/About_OWASP#Core_Values mission], helping with all global committee and project efforts for the benefit of the professional association. Every two years we rotate three of the board members. This year the following are NOT up for election this term are: [http://www.owasp.org/index.php/Eoin_Keary Eoin Keary], [http://www.owasp.org/index.php/User:Brennan Tom Brennan] and [http://www.owasp.org/index.php/Matt_Tesauro Matt Tesauro]. There will be (3) seats up for election in 2011 each voter will cast three votes for the available seats.

=== Eligible Voters ===
Current [http://www.owasp.org/index.php/Membership/members OWASP Individual Members] have (1) vote, [http://www.owasp.org/index.php/Template:OWASP_Members_Horizontal OWASP Supporters] have (1) vote via the primary point of contact.

=== 2011 OWASP Global Board of Directors Election Timeline ===

Milestone #1 Confirmed Candidate Official Announcement Jun 9th - [http://www.owasp.org/index.php/Category:OWASP_AppSec_Conference Global AppSec Europe] Dublin, Ireland. The individuals listed above should have a link to the campaign platform of who they plan to help OWASP Foundation and what they have done in OWASP to be considered a proven leader in the community. 

Milestone #2 Electronic Vote (eMail) managed by the [http://www.owasp.org/index.php/Global_Membership_Committee Global Membership Committee] will take place before the OWASP AppSec 2011 USA Event allowing candidates ample time to campaign the WHY ME. Announcement will be sent to all registered OWASP members email. This ballot will be sent to your online primary email address @OWASP.ORG or email that is on file with the [http://www.regonline.com/builder/site/Default.aspx?EventID=919827 regonline] membership system 

Below is a list of nominations to date .

=== HOW TO RUN ===
 If you want to be considered, login to the wiki, edit this page and link your profile in the same format . 
==== Candidates ====
* '''Christian Heinrich''' - [https://www.owasp.org/index.php/User:Cmlh BIO & why vote for me?] - New Candidate

* '''Michael Coates''' - [https://www.owasp.org/index.php/User:MichaelCoates BIO] - [https://www.owasp.org/index.php/User:MichaelCoates#OWASP_Board_Candidate_2011 Why Vote For Me?] - New Candidate

* '''Jim Manico''' - [http://www.owasp.org/index.php/User:jmanico BIO & why vote for me?] - New Candidate

* '''Dave Wichers''' - [http://www.owasp.org/index.php/User:Wichers BIO] - Current Board Member - Reelection

* '''Sebastien Deleersnyder''' - [http://www.owasp.org/index.php/User:Sdeleersnyder BIO & why vote for me?] - Current Board Member - Reelection

* <insert candidate name link to owasp wiki page, list bio and campaign platform views with why elect you?>

* <insert candidate name link to owasp wiki page, list bio and campaign platform views with why elect you?>

=== 2009 Election Results (History of how this process worked last time) ===
[http://www.owasp.org/index.php/Board_member Click Here for Election Information from 2009 / Results]

=== OWASP Foundation Bylaws ===
The [http://www.owasp.org/images/0/0d/OWASP_ByLaws.pdf current bylaws] that govern the OWASP Foundation and board member conduct are asked to adhere to and revise annually. *NOTE* - This is currently under revision as a result of the [http://www.owasp.org/index.php/Summit_2011_Outcomes OWASP 2011 Summit] revised bylaws are expected by the [http://www.owasp.org/index.php/OWASP_Board_Meetings June Board Meeting] - [https://docs.google.com/document/d/1r_hS2ioEBcNOKqmEjSJmlLUOdQEb5qPb_0GU_VU1Arw/edit?hl=en&authkey=CLe5nZwD Click here for DRAFT BYLAWS ]

=== Primary Responsibilities ===
1. Determine mission and purpose. It is the board's responsibility to create and review a statement of mission and purpose that articulates the organization's goals, means, and primary constituents served globally.

2. Select the Director(s) and employees. Boards must reach consensus on the Director responsibilities and undertake a careful search to find the most qualified individual for the position.

3. Support and evaluate all employees. The board should ensure that the employees have the moral and professional support he or she needs to further the goals of the organization.

4. Ensure effective planning. Boards must actively participate in an overall planning process and assist in implementing and monitoring the plan's goals.

5. Monitor, and strengthen programs and services. The board's responsibility is to determine which programs are consistent with the organization's mission and monitor their effectiveness.

6. Ensure adequate financial resources. One of the board's foremost responsibilities is to secure adequate resources for the organization to fulfill its mission.

7. Protect assets and provide proper financial oversight. The board must assist in developing the annual budget and ensuring that proper financial controls are in place.

8. Build a competent board. All boards have a responsibility to articulate prerequisites for candidates, orient new members, and periodically and comprehensively evaluate their own performance.

9. Ensure legal and ethical integrity. The board is ultimately responsible for adherence to legal standards and ethical norms.

10. Enhance the organization's public standing. The board should clearly articulate the organization's mission, accomplishments, and goals to the public and garner support from the community.

=== Questions ===
Please direct your questions to members of the [https://www.owasp.org/index.php/Global_Membership_Committee OWASP Global Membership Committee]

Membership/2011Election

2011-05-24T02:45:52Z

Cmlh: Added cmlh to list of candidates

User:Cmlh

2011-02-19T01:44:57Z

Cmlh: Added OWASP Netherlands

[mailto:christian.heinrich@owasp.org Christian Heinrich] has lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP [http://www.owasp.org/index.php/Category:OWASP_PCI_Project PCI], [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in:

*the Netherlands and;
*London, UK and;
*Sydney and Melbourne, Australia.

[mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org] and has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

User:Cmlh

2010-09-29T07:06:09Z

Cmlh: Added OpenSAMM contribution

[mailto:christian.heinrich@owasp.org Christian Heinrich] has lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP [http://www.owasp.org/index.php/Category:OWASP_PCI_Project PCI], [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten], [http://www.opensamm.org OpenSAMM] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in London, UK and Sydney and Melbourne, Australia.

[mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org] and has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

User:Cmlh

2010-09-28T11:53:50Z

Cmlh:

[mailto:christian.heinrich@owasp.org Christian Heinrich] has lead the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP [http://www.owasp.org/index.php/Category:OWASP_PCI_Project PCI], [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in London, UK and Sydney and Melbourne, Australia.

[mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org] and has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

User:Cmlh

2010-09-28T11:43:32Z

Cmlh:

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the former Project Leader of the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP [http://www.owasp.org/index.php/Category:OWASP_PCI_Project PCI], [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in London, UK and Sydney and Melbourne, Australia.

[mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org] and has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

OWASP Request for Proposals/New Project Leader/ASVS/Application 5

2010-09-06T06:53:04Z

Cmlh: Added Project Roadmap

{{Template:<includeonly>{{{1}}}</includeonly><noinclude>New Project Leader Applicants</noinclude>

| Applicant_Name = Christian Heinrich 
| Applicant_Email = christian.heinrich@owasp.org
| Applicant_Wiki_Username = cmlh
| Curriculum_Vitae_url = http://www.linkedin.com/in/ChristianHeinrich

| Proposed_Roadmap_url = http://cmlh.id.au/post/1073836740/owasp-asvs-project-leader

| Proposed_Roadmap_Text = Specified at http://cmlh.id.au/post/1073836740/owasp-asvs-project-leader


| Applicant_name_mask = Application_5
| Applicant_home_page = :OWASP_Request_for_Proposals/New_Project_Leader/ASVS/Application_5
| Applications_home_page = Seeking_New_Project_Leader_For/ASVS
}}

OWASP Request for Proposals/New Project Leader/ASVS/Application 5

2010-09-06T03:04:42Z

Cmlh: Commenced Application - saving prior to adding Proposed Roadmap

{{Template:<includeonly>{{{1}}}</includeonly><noinclude>New Project Leader Applicants</noinclude>

| Applicant_Name = Christian Heinrich 
| Applicant_Email = christian.heinrich@owasp.org
| Applicant_Wiki_Username = cmlh
| Curriculum_Vitae_url = http://www.linkedin.com/in/ChristianHeinrich

| Proposed_Roadmap_url = 

| Proposed_Roadmap_Text = 


| Applicant_name_mask = Application_5
| Applicant_home_page = :OWASP_Request_for_Proposals/New_Project_Leader/ASVS/Application_5
| Applications_home_page = Seeking_New_Project_Leader_For/ASVS
}}

User:Cmlh

2010-08-29T07:36:06Z

Cmlh: Added OWASP PCI Project

[mailto:christian.heinrich@owasp.org Christian Heinrich] is the Project Leader of the [http://www.owasp.org/index.php/Category:OWASP_Google_Hacking_Project OWASP "Google Hacking" Project] i.e. [http://code.google.com/p/dic "Download Indexed Cache"] and has contributed to the [http://www.owasp.org/index.php/Testing:_Spiders,_Robots,_and_Crawlers_(OWASP-IG-001) "Spiders/Robots/Crawlers"] and [http://www.owasp.org/index.php/Testing:_Search_engine_discovery/reconnaissance_(OWASP-IG-002) "Search Engine Reconnaissance"] sections of the OWASP Testing Guide v3 and more recently contributed to the development of the OWASP [http://www.owasp.org/index.php/Category:OWASP_PCI_Project PCI], [http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project Top Ten] and [http://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project Application Security Verification Standard (ASVS)] Projects.

[mailto:christian.heinrich@owasp.org Christian Heinrich] has presented at OWASP Conferences in USA, Australia and Europe and OWASP Chapters in London, UK and Sydney and Melbourne, Australia.

[mailto:christian.heinrich@owasp.org Christian Heinrich] can be reached at [mailto:christian.heinrich@owasp.org christian.heinrich@owasp.org] and has a Public Profile on LinkedIn at [http://www.linkedin.com/in/ChristianHeinrich http://www.linkedin.com/in/ChristianHeinrich]

Talk:OWASP Inquiries/Google Hacking Project

2010-07-19T01:27:14Z

Cmlh:

@Paulo Coimbra

The methodology, which I assume is generic for all inquiries, should be listed as a separate page but linked to all other inquiries.

The initial series of tabs should be:
6. "Record of Complaint(s)" - complaints made by anonymous and troll plaintiffs must be labeled as such and listed at the bottom of the wiki page with plaintiffs who are exist (including their contact information) and their complaints are listed at the top of the wiki page.
5. "Response from Project Leader"
4. "OWASP Board Responses [Rejected/Accepted]" i.e. Three parts i.e. 1. if an inquiry is required? and/or 2. other action items to avoid these type of complaints in the future 3. If these are accepted by the plaintiff (the attempted to contact them would be listed if they do not respond).

Obviously, if the inquiry is launched then the following could be recorded within one or multiple tabs
3. "Record of Legitimate Complaints"
2. "Acceptance by Project Leader"
1. "Concluding Statement by OWASP Board"

The number of 1-6 is based on the ordering as the public will read this left to right and hence more rumor, hearsay, etc may be created if the concluding tab isn't the left most tab (i.e. 4 or 1).

Talk:OWASP Inquiries/Google Hacking Project

2010-07-19T01:25:35Z

Cmlh: Suggested revised methodology

@Paulo Coimbra

The methodology, which I assume is generic for all inquiries, should be listed as a separate page but linked to the all inquiries.

The initial series of tabs should be:
6. "Record of Complaint(s)" - complaints made by anonymous and troll plaintiffs must be labeled and listed at the bottom of page with plaintiffs who are exist (including their contact information) complaints are listed at the top.
5. "Response from Project Leader"
4. "OWASP Board Responses [Rejected/Accepted]" i.e. Three parts i.e. 1. if an inquiry is required? and/or 2. other action items to avoid these type of complaints in the future 3. If these are accepted by the plaintiff (the attempted to contact them would be listed if they do not respond).

Obviously, if the inquiry is launched then the following could be recorded within one or multiple tabs
3. "Record of Legitimate Complaints"
2. "Acceptance by Project Leader"
1. "Concluding Statement by OWASP Board"

The number of 1-6 is based on the ordering as the public will read this left to right and hence more rumor, hearsay, etc may be created if the concluding tab isn't the left most tab (i.e. 4 or 1).

OWASP Google Hacking Project - PoC v0.2 - Post Google SOAP Search API Deprecation

2010-07-11T04:35:25Z

Cmlh: 1st DRAFT

Please note that executing "Download Indexed Cache" post September 2009 is in violation of Google's Terms of Service even with a valid Google SOAP Search API Key.

Information on how to download, build and install this release is available from http://code.google.com/p/dic/wiki/DownloadBuildInstall

GPC Project Details/OWASP Google Hacking Project

2010-07-11T04:33:03Z

Cmlh: Added RUXCON 2K8 Release as old_release

[[Category:OWASP Project|Google Hacking Project]]
[[Category:OWASP Tool]]
[[Category:OWASP Alpha Quality Tool]]

{{Template:<includeonly>{{{1}}}</includeonly><noinclude>OWASP Project Identification Tab</noinclude>
| project_name = OWASP Google Hacking Project
| project_description = "Download Indexed Cache" is a Proof of Concept (PoC) which implements the Google SOAP Search API to retrieve content indexed within the Google Cache and supports the "Search Engine Reconnaissance" section of the OWASP Testing Guide v3.
| project_license = [http://www.apache.org/licenses/LICENSE-2.0 Apache License 2.0]
| leader_name = Christian Heinrich
| leader_email = christian.heinrich@owasp.org
| leader_username = cmlh
| past_leaders_special_contributions =
| maintainer_name =
| maintainer_email =
| maintainer_username =
| contributor_name1 =
| contributor_email1 =
| contributor_username1 =
| contributor_name2 =
| contributor_email2 =
| contributor_username2 =
| contributor_name3 =
| contributor_email3 =
| contributor_username3 =
| contributor_name4 =
| contributor_email4 =
| contributor_username4 =
| contributor_name5 =
| contributor_email5 =
| contributor_username5 =
| contributor_name6 =
| contributor_email6 =
| contributor_username6 =
| contributor_name7 =
| contributor_email7 =
| contributor_username7 =
| contributor_name8 =
| contributor_email8 =
| contributor_username8 =
| contributor_name9 =
| contributor_email9 =
| contributor_username9 =
| contributor_name10 =
| contributor_email10 =
| contributor_username10 =
| pamphlet_link =
| presentation_link = http://www.slideshare.net/cmlh/download-indexed-cache
| mailing_list_name = owasp-google-hacking
| links_url1 = http://code.google.com/p/dic/
| links_name1 = Google Code
| links_url2 =
| links_name2 =
| links_url3 =
| links_name3 =
| links_url4 =
| links_name4 =
| links_url5 =
| links_name5 =
| links_url6 =
| links_name6 =
| links_url7 =
| links_name7 =
| links_url8 =
| links_name8 =
| links_url9 =
| links_name9 =
| links_url10 =
| links_name10 =
| project_road_map = Category:OWASP_Google_Hacking_Project_RoadMap
| project_health_status =
| current_release_name = PoC v0.2 - Post Google SOAP Search API Deprecation
| current_release_date = September 2009
| current_release_download_link = http://code.google.com/p/dic/downloads/list
| current_release_rating = -1
| current_release_leader_name = Christian Heinrich
| current_release_leader_email =
| current_release_leader_username = cmlh
| current_release_details = OWASP Google Hacking Project - PoC v0.2 - Post Google SOAP Search API Deprecation
| last_reviewed_release_name =
| last_reviewed_release_date =
| last_reviewed_release_download_link =
| last_reviewed_release_rating =
| last_reviewed_release_leader_name =
| last_reviewed_release_leader_email =
| last_reviewed_release_leader_username =
| old_release_name1 = RUXCON 2K8
| old_release_date1 = November 2008
| old_release_download_link1 = http://code.google.com/p/dic/source/browse/trunk/dic.pl?spec=svn6&r=2
| old_release_name2 =
| old_release_date2 =
| old_release_download_link2 =
| old_release_name3 =
| old_release_date3 =
| old_release_download_link3 =
| old_release_name4 =
| old_release_date4 =
| old_release_download_link4 =
| old_release_name5 =
| old_release_date5 =
| old_release_download_link5 =
| last_GPC_update = 06/July/2010
| GPC_Notes = This project has had its status changed (currently inactive) pending the outcome of an inquiry. 
| project_home_page = Category:OWASP_Google_Hacking_Project
| project_details_wiki_page = GPC_Project_Details/OWASP_Google_Hacking_Project
}}

GPC Project Details/OWASP Google Hacking Project

2010-07-11T04:29:21Z

Cmlh: Updated for PoC v0.2 - Post Google SOAP Search API Deprecation Release

[[Category:OWASP Project|Google Hacking Project]]
[[Category:OWASP Tool]]
[[Category:OWASP Alpha Quality Tool]]

{{Template:<includeonly>{{{1}}}</includeonly><noinclude>OWASP Project Identification Tab</noinclude>
| project_name = OWASP Google Hacking Project
| project_description = "Download Indexed Cache" is a Proof of Concept (PoC) which implements the Google SOAP Search API to retrieve content indexed within the Google Cache and supports the "Search Engine Reconnaissance" section of the OWASP Testing Guide v3.
| project_license = [http://www.apache.org/licenses/LICENSE-2.0 Apache License 2.0]
| leader_name = Christian Heinrich
| leader_email = christian.heinrich@owasp.org
| leader_username = cmlh
| past_leaders_special_contributions =
| maintainer_name =
| maintainer_email =
| maintainer_username =
| contributor_name1 =
| contributor_email1 =
| contributor_username1 =
| contributor_name2 =
| contributor_email2 =
| contributor_username2 =
| contributor_name3 =
| contributor_email3 =
| contributor_username3 =
| contributor_name4 =
| contributor_email4 =
| contributor_username4 =
| contributor_name5 =
| contributor_email5 =
| contributor_username5 =
| contributor_name6 =
| contributor_email6 =
| contributor_username6 =
| contributor_name7 =
| contributor_email7 =
| contributor_username7 =
| contributor_name8 =
| contributor_email8 =
| contributor_username8 =
| contributor_name9 =
| contributor_email9 =
| contributor_username9 =
| contributor_name10 =
| contributor_email10 =
| contributor_username10 =
| pamphlet_link =
| presentation_link = http://www.slideshare.net/cmlh/download-indexed-cache
| mailing_list_name = owasp-google-hacking
| links_url1 =
| links_name1 =
| links_url2 =
| links_name2 =
| links_url3 =
| links_name3 =
| links_url4 =
| links_name4 =
| links_url5 =
| links_name5 =
| links_url6 =
| links_name6 =
| links_url7 =
| links_name7 =
| links_url8 =
| links_name8 =
| links_url9 =
| links_name9 =
| links_url10 =
| links_name10 =
| project_road_map = Category:OWASP_Google_Hacking_Project_RoadMap
| project_health_status =
| current_release_name = PoC v0.2 - Post Google SOAP Search API Deprecation
| current_release_date = September 2009
| current_release_download_link = http://code.google.com/p/dic/downloads/list
| current_release_rating = -1
| current_release_leader_name = Christian Heinrich
| current_release_leader_email =
| current_release_leader_username = cmlh
| current_release_details = OWASP Google Hacking Project - PoC v0.2 - Post Google SOAP Search API Deprecation
| last_reviewed_release_name =
| last_reviewed_release_date =
| last_reviewed_release_download_link =
| last_reviewed_release_rating =
| last_reviewed_release_leader_name =
| last_reviewed_release_leader_email =
| last_reviewed_release_leader_username =
| old_release_name1 =
| old_release_date1 =
| old_release_download_link1 =
| old_release_name2 =
| old_release_date2 =
| old_release_download_link2 =
| old_release_name3 =
| old_release_date3 =
| old_release_download_link3 =
| old_release_name4 =
| old_release_date4 =
| old_release_download_link4 =
| old_release_name5 =
| old_release_date5 =
| old_release_download_link5 =
| last_GPC_update = 06/July/2010
| GPC_Notes = This project has had its status changed (currently inactive) pending the outcome of an inquiry. 
| project_home_page = Category:OWASP_Google_Hacking_Project
| project_details_wiki_page = GPC_Project_Details/OWASP_Google_Hacking_Project
}}

Category:OWASP Google Hacking Project RoadMap

2010-07-11T04:27:40Z

Cmlh: Updated for PoC v0.2 - Post Google SOAP Search API Deprecation Release

Sep 2008 - Oct 2008
PoC v0.1 demonstrated at OWASP USA Conference 2008, ToorCon X and SecTor 2008 
Nov 2008
PoC v0.1 released at RUXCON 2K8 
Sep 2009
PoC v0.2 released due to Google deprecating their SOAP Search API