Difference between revisions of "Review Webserver Metafiles for Information Leakage (OTG-INFO-003)"

Revision as of 02:33, 30 June 2008

OWASP Testing Guide v3 Table of Contents

This article is part of the OWASP Testing Guide v3. The entire OWASP Testing Guide v3 can be downloaded here.

OWASP at the moment is working at the OWASP Testing Guide v4: you can browse the Guide here

This is a draft of a section of the new Testing Guide v3

Brief Summary

This section describes how to traverse the content of the application under test using web spiders/robots/crawlers.

Description of the Issue

Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. While their accepted behavior is specified by the web server and its web pages they may accidentally or intentionally retrieve web content not intended to be stored or published.

Black Box testing and example

Testing for Topic X vulnerabilities:
...
Result Expected:
...

Gray Box testing and example

Testing for Topic X vulnerabilities:
...
Result Expected:
...

References

Whitepapers

• [1] "The Web Robots Pages" - http://www.robotstxt.org/
• [2] "How Google crawls my site" - http://www.google.com/support/webmasters/bin/topic.py?topic=8843
• [3] "Preventing content from appearing in Google search results " - http://www.google.com/support/webmasters/bin/topic.py?topic=8459

...
Tools

• wget - http://www.gnu.org/software/wget/
• Burp Spider - http://portswigger.net/spider/

...