Talk:Conduct search engine discovery/reconnaissance for information leakage (OTG-INFO-001)

v3 Review Comments

This section does not cover the items stated in the "brief summary". For v3, if the section is to remain completely google'centric I suggest we rename "Search engine discovery" to "Google searching your web application and accessing google's cache".

Reply to "v3 Review Comments" from @cmlh

The roadmap was to add Yahoo! and Bing to the next release of the OWASP Testing Guide (i.e. v3 -> v4) and to not appear to promote Google over Yahoo! and Bing. It should be noted that Yahoo! and Bing might refer to the same "entity" as further research is undertaken i.e. the "Yahoo! and Microsoft Search Alliance"/"Yahoo! Bing Network".

Furthermore, the intent is *not* to promote the inferior http://www.hackersforcharity.org/ghdb/, rather a more scientific and innovative approach.

RM - Hi cmlh, thanks for the follow-up. That comment was really old and seems to have been migrated for the v3 > v4 draft. I think the new heading/title is more appropriate than previously, however, the content still seems awfully google'centric.

Should we also be including some Shodan stuff? (http://www.shodanhq.com/) Rick.mitchell (talk)

RM - Actually now that I'm looking at this. I'm not sure how the heading has changed since v3 was a draft (when the comment was originally made). However, again looking at this now there are a number of goals, etc stated in the summary that don't seem to be covered by the content. Also the summary seems to be written from the perspective of a app/system owner not a tester.

I also wonder if we should be including examples such as xssed.com and their ilk, web.archive.org, etc Rick.mitchell (talk)

Adding web services, such as xxsed.com or web.archive.org, would depend on if they an API available to the public (I believe archive.org has and API) and if there is a product available (possibly released under FOSS licenses) to provide an example.

RM - IMHO that only applies from a purely automated point of view. There is no reason we shouldn't be referencing such as a manual step (or steps).

CMLH - I am aware of archive.org, I am not sure about the xxsed example that you are referring too?

RM - xssed.com is a community site that catalogs (by submission) vulnerabilities found on public sites. If I were doing a Web App test for a client I'd look to see if anyone else had reported an issue for their site. Having vulnerabilities in your web app publicly outted seems like a serious information leak to me.

CMLH - I know what xssed is already :) Instead I meant did they offer a Public API (which is not the case as far as I am aware). That stated, a loud minority of OWASP members may complain that we are promoting an unethical web site but I have no issue with including xssed (perhaps in another related section of the testing guide).

RM - I'm missing the significance of having a public API. Goto the site (it's public), look-up your targets. Not everything has to be automated. Rick.mitchell (talk) 08:47, 23 August 2013 (CDT)

RM - I'm not sure how the majority of the industry ends up getting involved in Web App VA but from my perspective and experience there is usually limited targets so doing a few manual lookups isn't a major stumbling block.

CMLH - I believe some of these other services might be out of scope of OTG-INFO-001.

RM - Shodan is a search engine specifically designed to catalog systems and version/configuration information. Finding listings for your target client/app with Shodan can provide further information about your target.

CMLH - I believe some of these other services might still be out of scope of OTG-INFO-001 in light of the clarification above but could be addressed in a related section of v4.

RM - Further the summary for 001 talks about " Indirect methods relate to gleaning sensitive design and configuration information by searching forums, newsgroups and tendering websites." none of which is actually covered. There is a huge open source intelligence gathering activity which should be covered based on that statement. (Finding related employees on linkedin, searching google groups for SW and HW questions, etc) Rick.mitchell (talk) 07:23, 19 August 2013 (CDT)

CMLH - That section was added *after* I had contributed to v3 i.e. http://lists.owasp.org/pipermail/owasp-testing/2013-August/002160.html

RM - Understood. So are we keeping those ideas for v4 and covering some suggestions on what can be looked at, what can be found, and what might matter? Rick.mitchell (talk) 08:47, 23 August 2013 (CDT)

RM - Oh and if it matters somehow Shodan does have an API. Though I stick by the statement that not everything has to be automated to automate'able :) Here's one small intro for it: http://raidersec.blogspot.ca/2012/02/searching-for-devices-using-shodan.html Rick.mitchell (talk) 07:27, 19 August 2013 (CDT)

CMLH - Yes, I am aware of their API i.e. http://cmlh.id.au/tagged/shodan

v4 Reviewer feedback

"Direct methods relate to searching the Google Index and remove the associated web content from the Google Cache"

What does direct detection/use have to do with removal of content from google's cache?

"Therefore, it must be removed from the Google Cache."

What? Should this be moved to the remediation section?

General: content remains completely google'centric. Also while the examples provided do result in the expected output this is all based on expected behavior. (Search engines index things and let you search them.) The examples do not seem to cover any "information leakage" (un-intended files being indexed, version information leaks, details of infrastructure components, etc). Rick.mitchell (talk) 11:03, 1 October 2013 (CDT)