Testing: Spidering and googling - Revision history

Mmeucci at 19:56, 21 May 2009

2009-05-21T19:56:50Z

2009-05-21T19:41:08Z

Undo revision 61163 by SitdrOnlar (Talk)

2009-05-21T19:33:10Z

2007-09-10T00:48:30Z

‎Spidering

2007-09-10T00:47:59Z

‎Spidering

2007-09-10T00:46:07Z

‎Spidering

2007-09-10T00:40:11Z

‎Spidering

2007-04-04T08:48:02Z

‎Googling

2007-04-04T03:53:53Z

‎Googling

2007-04-04T03:53:03Z

‎Googling

@@ Line 1: / Line 1: @@
-[[http://www.owasp.org/index.php/Web_Application_Penetration_Testing_AoC Up]] <br>
+{{Template:OWASP Testing Guide v3}}
 == Brief Summary ==
@@ Line 179: / Line 178: @@
 * Googlegath - http://www.nothink.org/perl/googlegath/<br>
 * Advanced Dork (Firefox Add-on) - https://addons.mozilla.org/firefox/2144/

@@ Line 1: / Line 1: @@
 [[http://www.owasp.org/index.php/Web_Application_Penetration_Testing_AoC Up]] <br>
 {{Template:OWASP Testing Guide v2}}
@@ Line 92: / Line 91: @@
 * To search for a phrase, supply the phrase surrounded by double quotes (" ").
 * A period (.) serves as a single-character wildcard.
-* An asterisk (*) represents any word â- not the completion of a word, as is traditionally used.
+* An asterisk (*) represents any word —- not the completion of a word, as is traditionally used.
 Google advanced operators help refine searches. Advanced operators use the following syntax: operator:search_term . Notice that there is no space between the operator, the colon, and the search term. A list of operators and search terms follows:
@@ Line 171: / Line 170: @@
 '''Tools'''<br>
-* Google â http://www.google.com<br>
+* Google – http://www.google.com<br>
 * Live Search - http://www.live.com
 * wget - http://www.gnu.org/software/wget/

@@ Line 1: / Line 1: @@
 [[http://www.owasp.org/index.php/Web_Application_Penetration_Testing_AoC Up]] <br>
 {{Template:OWASP Testing Guide v2}}
@@ Line 91: / Line 92: @@
 * To search for a phrase, supply the phrase surrounded by double quotes (" ").
 * A period (.) serves as a single-character wildcard.
-* An asterisk (*) represents any word —- not the completion of a word, as is traditionally used.
+* An asterisk (*) represents any word â- not the completion of a word, as is traditionally used.
 Google advanced operators help refine searches. Advanced operators use the following syntax: operator:search_term . Notice that there is no space between the operator, the colon, and the search term. A list of operators and search terms follows:
@@ Line 170: / Line 171: @@
 '''Tools'''<br>
-* Google – http://www.google.com<br>
+* Google â http://www.google.com<br>
 * Live Search - http://www.live.com
 * wget - http://www.gnu.org/software/wget/

@@ Line 37: / Line 37: @@
             => `index.html'
 Resolving www.<target>... 64.xxx.xxx.23, 64.xxx.xxx.24, 64.xxx.xxx.20, ...
-Connecting to www.cnn.com|64.xxx.xxx.23|:80... connected.
+Connecting to www.<target>|64.xxx.xxx.23|:80... connected.
 HTTP request sent, awaiting response...
    HTTP/1.1 200 OK

@@ Line 30: / Line 30: @@
 <pre>
 wget -S --spider <target>
 </pre>
 '''Result:'''
-<pre>http://www.cnn.com/
+<pre>http://www.<target>/
             => `index.html'
-Resolving www.cnn.com... 64.236.91.23, 64.236.91.24, 64.236.16.20, ...
+Resolving www.<target>... 64.xxx.xxx.23, 64.xxx.xxx.24, 64.xxx.xxx.20, ...
-Connecting to www.cnn.com|64.236.91.23|:80... connected.
+Connecting to www.cnn.com|64.xxx.xxx.23|:80... connected.
 HTTP request sent, awaiting response...
    HTTP/1.1 200 OK

@@ Line 26: / Line 26: @@
 '''Test:'''
-The -S option is used to collect the HTTP header of the web requests.
+The -S option is used to collect the HTTP header of the web requests. The --spider options is used to not download anything since we only want the HTTP header.
 <pre>
-wget -S <target>
+wget -S --spider <target>
 </pre>
-'''Result:'''
+Just look at this example using [CNN website|http://www.cnn.com/]
 <pre>
-HTTP/1.1 200 OK
+wget -S --spider www.cnn.com
 </pre>
 '''Test:'''

@@ Line 97: / Line 97: @@
 * The ''cache'' operator displays the version of a web page as it appeared when Google crawled the site. The URL of the site must be supplied after the colon.
 * The ''intitle'', ''allintitle'' operator instructs Google to search for a term within the title of a document.
-* The ''inurl'', 'allinurl'' operator instructs Google to search only within the URL (web address) of a document. The search term must follow the colon.
+* The ''inurl'', ''allinurl'' operator instructs Google to search only within the URL (web address) of a document. The search term must follow the colon.
 * The ''info'' operator instructs Google to search only within the summary information of a site
 * The ''phonebook'' operator instructs Google to search business or residential phone listing.

@@ Line 154: / Line 154: @@
 The only information that the legimitate user really needs is a message that says "Page Not found." Restricting the other information will prevent your page from turning up in an attacker's search for a specific flavor of server.
-Google periodically purges it's cache, but until then your sensitive files are still being offered to the public. If you realize that the search engine has cached files that you want to be unavailable to be viewed you can go to ( http://www.google.com/remove.html ) and follow the instructions on how to remove your page, or parts of your page, from their database.
+Google periodically purges it's cache, but until then your sensitive files are still being offered to the public. If you realize that the search engine has cached files that you want to be unavailable to be viewed you can go to  http://www.google.com/remove.html  and follow the instructions on how to remove your page, or parts of your page, from their database.
 '''Using a search engine to discover virtual hosts'''

@@ Line 147: / Line 147: @@
 Make sure you are comfortable with sharing everything in your public Web folder with the whole world, because Google will share it, whether you like it or not. Also, in order to prevent attackers from easily figuring out what server software you are running, change the default error messages and other identifiers. Often, when a "404 Not Found" error is detected, servers will return a page like that says something like:
-'''Not Found''' <br>
+<pre>
-'''The requested URL /cgi-bin/xxxxxx was not found on this server.'''<br>
+Not Found
-'''Apache/1.3.27 Server at your web site Port 80'''<br>
+The requested URL /cgi-bin/xxxxxx was not found on this server.
 The only information that the legimitate user really needs is a message that says "Page Not found." Restricting the other information will prevent your page from turning up in an attacker's search for a specific flavor of server.