This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit

Talk:Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

Revision as of 02:36, 24 August 2008 by Cmlh (talk | contribs) (Replied to Marco)

Jump to: navigation, search

It could be added that, from an attacker point of view, the robots.txt file can provide some useful information on the structure of the web server, e.g., directories that are supposed to be "private".

Marco 18:11, 17 August 2008 (EDT)

The intent of robots.txt is *not* to specify access control for directories. Hence to quote the wiki page "Web spiders/robots/crawlers can intentionally ignore the Disallow directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.".

If you believe this is not your communicated clearly or could be reworded then please amend the wiki page.

cmlh 12:34, 24 August 2008 (GMT +10)