This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit https://owasp.org

Difference between revisions of "OWASP Web Mapper Project"

From OWASP
Jump to: navigation, search
(FAQs)
(OWASP Web Mapper Project)
 
(36 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
=Main=
 
=Main=
  
OWASP Web Mapper is a pure Ruby library for the Internet web application discovery and tracking. Project is currently hosted under GitHub at:
 
 
*  https://github.com/yangsec888/wmap/
 
 
<!-- DO NOT ALTER OR REMOVE THE TEXT ON NEXT LINE -->
 
<!-- DO NOT ALTER OR REMOVE THE TEXT ON NEXT LINE -->
 
<div style="width:100%;height:160px;border:0,margin:0;overflow: hidden;">[[File:OWASP_Project_Header.jpg|link=]]</div>
 
<div style="width:100%;height:160px;border:0,margin:0;overflow: hidden;">[[File:OWASP_Project_Header.jpg|link=]]</div>
Line 16: Line 13:
  
  
This project is designed for the web application asset discovery and tracking. It was originally developed to to cover the gaps of Veracode's Discovery service. Over the time it grows to be a more capable and complete replacement (IMHO).
+
What if there are quite a few web applications under your organization, but nobody seems to know all of them (before the talk of application risk assessment)? Then this project may be a right fit for you.
 +
 
 +
This project is designed to perform the web application asset discovery and auto tracking with scale.
  
 
==Description==
 
==Description==
  
A pure Ruby library for the web application asset discovery and tracking. The tool is useful when you're handling a larger size organization with multiple Internet domain and network registered under the name, where manual discovery and tracking prove to be both time-consuming and problematic, if not impossible. The program will help you quickly identify the 'unknown' web application asset, and keep track of them automatically for you as a repeatable process.  If you are serious about your organization's web exposure, this might be the perfect footprinting tool you're looking for.  
+
A pure Ruby library for the web application asset discovery and tracking. The tool is useful when you're handling a larger size organization with multiple Internet domains and networks registered under the name. Where both legacy and new web applications are omni-present but nobody seems to be able to provide a complete list of application URLs to you. Yes you can always do it the old way by using tool sets such as NMAP, OWASP Zap web crawler, along with others. But such tool sets could quickly become too much manual-driven and inaccurate, if not impossible. In the contrary, once setup, this project will help you quickly identify all the 'unknown' web application asset, and keep track of them automatically.  If you are serious about your organization's Internet web application exposure, this might be the perfect all-in-one footprinting tool you're looking for.  
  
Built as an open source project, the source code is both free and robust. You're welcome to keep building on top of the current code base, or include it as part of your larger project distribution.
+
Built as an open source project, the source code is both free and scalable. You're welcome to keep building on top of the current code base, or include it as part of your larger project distribution.
  
 
==Licensing==
 
==Licensing==
Line 39: Line 38:
 
Ideally we could document various reverse-engineering techniques using by this project and publish it through the OWASP Press. Of course, it will always remain freely available, and any money collected will go directly into the project and to the OWASP Foundation.
 
Ideally we could document various reverse-engineering techniques using by this project and publish it through the OWASP Press. Of course, it will always remain freely available, and any money collected will go directly into the project and to the OWASP Foundation.
  
== Presentation ==
+
== Demo ==
 +
 
 +
*[http://wmap.io/ Demo] A demo web app exploring WMAP library power.
  
TBD
 
  
== Project Leader ==
+
== Project Code and Documents ==
  
 +
* [https://github.com/yangsec888/wmap Latest WMAP source tree.]
 +
* [https://github.com/yangsec888/www_wmap WMAP web app demo built in Ruby on Rails 5+]
  
* [https://www.owasp.org/index.php/User:Yang_Li Yang Li]
+
<!-- DO NOT ALTER OR REMOVE THE TEXT ON NEXT LINE -->
 +
| valign="top"  style="padding-left:25px;width:200px;" |
  
 
== Related Projects ==
 
== Related Projects ==
Line 53: Line 56:
 
* [[OWASP_Zed_Attack_Proxy_Project]]
 
* [[OWASP_Zed_Attack_Proxy_Project]]
  
== Openhub ==
+
== News and Events ==
 
 
* [https://www.openhub.net/orgs/OWASP OWASP Project Openhub]
 
 
 
<!-- DO NOT ALTER OR REMOVE THE TEXT ON NEXT LINE -->
 
| valign="top"  style="padding-left:25px;width:200px;" |
 
 
 
== Quick Download ==
 
  
 
<!-- Instructions are in RED and should be removed from your document by deleting the text with the span tags.-->
 
<!-- Instructions are in RED and should be removed from your document by deleting the text with the span tags.-->
  
  
The home of the OWASP Web Mapper is on [https://github.com/yangsec888/wmap GitHub.] You are encouraged to fork, edit and push your changes back to the project through git or edit the project directly on GitHub.
+
* [January 1 2018] OWASP Web Mapper demo web application released.
 +
* [August 1 2015] OWASP Web Mapper Project created.
  
However, if you like you may also download the master repository from the following links:
 
* [https://raw.githubusercontent.com/yangsec888/wmap/master/wmap-2.4.2.gem .gem file.]
 
* [https://github.com/yangsec888/wmap/zipball/master .zip file.]
 
* [https://github.com/yangsec888/wmap/tarball/master .tgz file.]
 
  
Project Code and Documents:https://github.com/yangsec888/wmap
+
== Project Leader ==
  
== News and Events ==
+
* [https://www.owasp.org/index.php/User:Yang_Li Sam (Yang) Li]
  
<!-- Instructions are in RED and should be removed from your document by deleting the text with the span tags.-->
 
 
 
* [August 1 2015] OWASP Web Mapper Project created.
 
  
 
== In Print ==
 
== In Print ==
  
 
+
I'm working on project document. But it's far from become a book at this moment. Instead, please refer to the project hosting site for more project document.
I'm working on project document. But it's far from become a book at this moment. Instead, please refer to the project site for project document.
 
  
 
==Classifications==
 
==Classifications==
Line 151: Line 139:
 
   $ wmap <seed file | target host | target url | target IP or network cidr>
 
   $ wmap <seed file | target host | target url | target IP or network cidr>
  
The above executable is intelligent enough to take argument as either a seed file, or a string such as a host, an IP, a network block, or a URL. The new discoveries will be automatically tracked in the data file 'lib/wmap/data/target_sites'.
+
The above utility is intelligent enough to take argument as either a seed file, or a string such as a host, an IP, a network block, or a URL. The new discoveries will be automatically tracked in the data file 'lib/wmap/data/target_sites'.
 
   Note: seed file - mix of url, cidr and domain seed, one entry per line.
 
   Note: seed file - mix of url, cidr and domain seed, one entry per line.
 
url seed - known URL(s) for further discovery via the web crawler.
 
url seed - known URL(s) for further discovery via the web crawler.
Line 161: Line 149:
 
   $ wdump [output file name from you]
 
   $ wdump [output file name from you]
  
The above executable will dump out the discovery database into a single file as program output. Currently, the supported file format is Comma-separated Value (.csv) and Extensible Markup Language (.xml)
+
The above utility will dump out the discovery database into a single file as program output. Currently, the supported file format is Comma-separated Value (.csv) and Extensible Markup Language (.xml)
 +
 
 +
== Other Build-in Utilities ==
 +
You may need to update your sset repository from time to time. For this purpose, ‘wmap’ provide three utilities ‘wcheck’, ‘wadd’ and ‘wdel’ to perform the tasks for you. For example, you can check out the status of a site, as shown below:
 +
  $ wcheck https://www.owasp.org/
 +
  <nowiki>{"ip"=>"104.130.192.89", "port"=>443, "url"=>"https://www.owasp.org/", "code"=>301, "redirection"=>"https://www.owasp.org/index.php/Main_Page", "md5"=>"d41d8cd98f00b204e9800998ecf8427e", "server"=>"Apache", "timestamp"=>2015-09-02 13:32:31 -0400, "status"=>"ext_hosted"}</nowiki>
 +
 
 +
You may want to use 'wadd' utility to add it, as shown below:
 +
  $ wadd https://www.owasp.org/
 +
 
 +
You may want to use 'wdel' utility to delete it, as shown below:
 +
  $ wdel https://www.owasp.org/
  
== More Usage Cases ==
+
== More Usage Examples ==
There are more examples under the 'demos' folder of this package. The examples show how to use the 'wmap' API to get your job done easily. Please check out the code - they should be easy and straightforward to be understood.
+
There are also some code examples under the 'demos' folder of this package. The examples show how to use the 'wmap' API to get your   job done easily. Please check out the code - they should be easy and straightforward to be understood.
  
 
== More document(s) ==
 
== More document(s) ==
The software comes with the Ruby doc during your installation as shown above. For your convenience, the Ruby doc is also distributed with this software. You can navigate to the 'doc' folder, and click the 'index.html' to open the start page in your favorite browser.
+
The software comes with the Ruby doc during your installation as shown above. For your convenience, the Ruby doc is also distributed with the software. You can navigate to the 'doc' folder of your local installation, and click the 'index.html' to open the start page in your favorite browser. You can also download the wmap-x.x.x.rdoc.zip documentation package alone from the hosting site, unzip and open the doc/index.html in your browser.
  
If you need additional documentation / information other than this README file and the Ruby document package, please be patient - as I'm still working on it :)
+
If you need additional documentation / information other than this README file and the Ruby document package, please be patient - as I'm still working on it.
  
 
==How do I report the bugs, or maybe require some new features?==
 
==How do I report the bugs, or maybe require some new features?==
Line 191: Line 190:
 
* [Your name here]
 
* [Your name here]
  
= Road Map and Getting Involved =
+
= Road Map =  
  
 
<!-- Instructions are in RED and should be removed from your document by deleting the text with the span tags.-->
 
<!-- Instructions are in RED and should be removed from your document by deleting the text with the span tags.-->
 
<span>
 
<span>
Details TBD. Please refer to TODO file at code repository for additional information:
+
Please refer to TODO file at code repository for additional information:
 
*https://github.com/yangsec888/wmap/blob/master/TODO
 
*https://github.com/yangsec888/wmap/blob/master/TODO
 
</span> 
 
</span> 
  
 +
= Getting Involved =
  
 
+
Involvement in the development and promotion of the OWASP Web Mapper Project is encouraged!
Involvement in the development and promotion of the OWASP Web Mapper Project is actively encouraged!
 
 
You do not have to be a developer in order to contribute.
 
You do not have to be a developer in order to contribute.
Some of the ways you can help:
+
Some of the ways you can get involved and contribute:
 +
* Join the project [https://lists.owasp.org/mailman/listinfo/owasp-web-mapper-project Mailing List].
 
* Help QA test the project.
 
* Help QA test the project.
 
* Help build a user-friendly front-end.
 
* Help build a user-friendly front-end.
Line 210: Line 210:
 
* Writing support for the book.  
 
* Writing support for the book.  
  
 
+
If you can help, please sign up a [https://github.com/ GITHUB] account, then [mailto:[email protected] notify me]. 
  
  

Latest revision as of 14:28, 15 September 2019

OWASP Project Header.jpg


OWASP Web Mapper Project

What if there are quite a few web applications under your organization, but nobody seems to know all of them (before the talk of application risk assessment)? Then this project may be a right fit for you.

This project is designed to perform the web application asset discovery and auto tracking with scale.

Description

A pure Ruby library for the web application asset discovery and tracking. The tool is useful when you're handling a larger size organization with multiple Internet domains and networks registered under the name. Where both legacy and new web applications are omni-present but nobody seems to be able to provide a complete list of application URLs to you. Yes you can always do it the old way by using tool sets such as NMAP, OWASP Zap web crawler, along with others. But such tool sets could quickly become too much manual-driven and inaccurate, if not impossible. In the contrary, once setup, this project will help you quickly identify all the 'unknown' web application asset, and keep track of them automatically. If you are serious about your organization's Internet web application exposure, this might be the perfect all-in-one footprinting tool you're looking for.

Built as an open source project, the source code is both free and scalable. You're welcome to keep building on top of the current code base, or include it as part of your larger project distribution.

Licensing

The OWASP Web Mapper Project is free to use. In fact it is encouraged!

The OWASP Security Principles are licensed under the Apache 2.0 license, so you can copy, distribute and transmit the work, and you can adapt it, and use it commercially, but all provided that you attribute the work and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

What is OWASP Web Mapper Project?

The goal is to help you better identify, and keep track of the web application asset under your watch. Ideally we could document various reverse-engineering techniques using by this project and publish it through the OWASP Press. Of course, it will always remain freely available, and any money collected will go directly into the project and to the OWASP Foundation.

Demo

  • Demo A demo web app exploring WMAP library power.


Project Code and Documents

Related Projects

News and Events

  • [January 1 2018] OWASP Web Mapper demo web application released.
  • [August 1 2015] OWASP Web Mapper Project created.


Project Leader


In Print

I'm working on project document. But it's far from become a book at this moment. Instead, please refer to the project hosting site for more project document.

Classifications

New projects.png Owasp-builders-small.png
Owasp-defenders-small.png
Cc-button-y-sa-small.png
Project Type Files DOC.jpg

Installation

  1. The easiest way to install WebMapper is by using Ruby Gems. Download the latest gem 'wmap-x.x.x.gem' into the local file system. Then install it from your shell environment:
$ gem install wmap-x.x.x.gem --no-rdoc

Dependency

You need the Ruby 1.9.2 or above in order to use this program. In addition, I developed and tested the code in a Ubantu Linux VM with Ruby 1.9.3.

  • You need to setup Ruby 1.9.x environment. In my test environment, I was able to set it up with RVM. Please refer to this page for more installation information: http://www.ruby-lang.org/en/downloads/
  • In addition, the following Ruby GEM dependency are needed by different components of this software:
  1. require "uri"
  2. require "open-uri"
  3. require "open_uri_redirections"
  4. require "nokogiri"
  5. require "net/http"
  6. require "httpclient"
  7. require "net/ping"
  8. require "netaddr"
  9. require "socket"
  10. require "openssl"
  11. require "whois"
  12. require "resolv"
  13. require "geoip"
  14. require "parallel"
  15. require "dnsruby"
  • Install dependency gem "uri" example
 $ gem install uri
  • Ruby-whois gem patches

This software also depends on a patched version of Ruby gem ruby-whois-2.7.0 (http://www.ruby-whois.org/) for the domain whois lookup feature. For better result, you could manually add the patches into your local whois gem installation directory as shown below:

 $ cp whois_patches/* [Your_ruby_whois_gem_path]/whois/lib/whois/record/parser/

Or you can directly download the branched whois gem from this repository - https://github.com/yangsec888/whois

Before Using This Program

You need to define a scope for the program to run successful. The scope includes both your legitimate Internet domain, and your public network block in the CIDR format.

To add your Internet domain into the scope, use the build-in shell command below:

 $ trust XYZ.COM

To add your public network block into the scope:

 $ trust x.x.x.x/x

Automatic Discovery and Tracking

 $ wmap <seed file | target host | target url | target IP or network cidr>

The above utility is intelligent enough to take argument as either a seed file, or a string such as a host, an IP, a network block, or a URL. The new discoveries will be automatically tracked in the data file 'lib/wmap/data/target_sites'.

 Note: seed file - mix of url, cidr and domain seed, one entry per line.

url seed - known URL(s) for further discovery via the web crawler. cidr seed - list of known network blocks, for discovering web service via port scanning; it is also used to validate if the web service has a known IP (internal hosted). domain seed - validated internet domain to be used for DNS record brute-forcing; it is also used to validate the ownership of found web service.

Dump Out Discovery Database

You can dump out the program output by using the build-in utility 'wdump' as shown below:

 $ wdump [output file name from you]

The above utility will dump out the discovery database into a single file as program output. Currently, the supported file format is Comma-separated Value (.csv) and Extensible Markup Language (.xml)

Other Build-in Utilities

You may need to update your sset repository from time to time. For this purpose, ‘wmap’ provide three utilities ‘wcheck’, ‘wadd’ and ‘wdel’ to perform the tasks for you. For example, you can check out the status of a site, as shown below:

 $ wcheck https://www.owasp.org/
 {"ip"=>"104.130.192.89", "port"=>443, "url"=>"https://www.owasp.org/", "code"=>301, "redirection"=>"https://www.owasp.org/index.php/Main_Page", "md5"=>"d41d8cd98f00b204e9800998ecf8427e", "server"=>"Apache", "timestamp"=>2015-09-02 13:32:31 -0400, "status"=>"ext_hosted"}

You may want to use 'wadd' utility to add it, as shown below:

 $ wadd https://www.owasp.org/

You may want to use 'wdel' utility to delete it, as shown below:

 $ wdel https://www.owasp.org/

More Usage Examples

There are also some code examples under the 'demos' folder of this package. The examples show how to use the 'wmap' API to get your job done easily. Please check out the code - they should be easy and straightforward to be understood.

More document(s)

The software comes with the Ruby doc during your installation as shown above. For your convenience, the Ruby doc is also distributed with the software. You can navigate to the 'doc' folder of your local installation, and click the 'index.html' to open the start page in your favorite browser. You can also download the wmap-x.x.x.rdoc.zip documentation package alone from the hosting site, unzip and open the doc/index.html in your browser.

If you need additional documentation / information other than this README file and the Ruby document package, please be patient - as I'm still working on it.

How do I report the bugs, or maybe require some new features?

Contact the developer Yang Li directly at his email '[email protected]'.

How can I participate in your project?

All you have to do is make the Project Leader's aware of your available time to contribute to the project. It is also important to let the Leader's know how you would like to contribute and pitch in to help the project meet it's goals and milestones. There are many different ways you can contribute to an OWASP Project, but communication with the leads is key.

If I am not a programmer can I participate in your project?

Yes, you can certainly participate in the project if you are not a programmer or technical. The project needs different skills and expertise and different times during its development. Currently, we are looking for QA tester, front-end builder, document writers, graphic designers, and a project administrator.

Contributors

The OWASP Web Mapper Project is originally developed by Yang Li. A live update of project contributors can be found here.

The first contributors to the project are:

Please refer to TODO file at code repository for additional information:

 

Involvement in the development and promotion of the OWASP Web Mapper Project is encouraged! You do not have to be a developer in order to contribute. Some of the ways you can get involved and contribute:

  • Join the project Mailing List.
  • Help QA test the project.
  • Help build a user-friendly front-end.
  • Project administration support.
  • Wiki editing support.
  • Writing support for the book.

If you can help, please sign up a GITHUB account, then notify me.