This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit https://owasp.org

Difference between revisions of "OWASP Hatkit Datafiddler Project"

From OWASP
Jump to: navigation, search
Line 32: Line 32:
 
[[Image:hatkit-datafiddler-tabledata-settings-2.png|thumb|The tabledata settings also has filters]]
 
[[Image:hatkit-datafiddler-tabledata-settings-2.png|thumb|The tabledata settings also has filters]]
 
[[Image:hatkit-datafiddler-tabledata-settings-3.png|thumb|The filters can be arbitrary javascript expressions]]
 
[[Image:hatkit-datafiddler-tabledata-settings-3.png|thumb|The filters can be arbitrary javascript expressions]]
 +
 +
If you select Settings, you will be met by the settings-window. On the left side, there are '''variables'''. For each object which is fetched from the database, these expressions determines exactly what parts are fetched and places these parts, into python variables with the names '''v0''' and onward.
 +
 +
'''Tip:''' The section below is pretty technical. You don't '''have''' to know python or javascript to use this tool, Datafiddler comes with predefined expressions and views that you can use. When you learn the ropes a bit, you can just make modifications to these and you should be fine.  More predefined views are coming.
 +
 +
For example, a database object stored by Hatkit Proxy always contain these fields:
 +
<pre>
 +
request
 +
response
 +
</pre>
 +
(For more details about storage format, see [[OWASP_Hatkit_Proxy_Project#tab=Storage Storage]]
 +
If the '''request''' part of a database object is loaded into '''v0''', it means that '''v0''' will contain python dictionary containing everything that concerns the request. E.g. The python expression <code>v0['method']</code> will be the request verb (GET/POST/FOO),while the expression <code>v0['headers']</code> will be another python dictionary containing the request headers.
 +
 +
This means that this object introspection can be performed '''either''' inside the database - which is using javascript, '''or''' in the application itself, using python. Example:
 +
<pre>
 +
v0 = request.headers.Host === (v0 = request)['headers']['Host]
 +
</pre>
 +
 +
Worth mentioning though, is that accessing a non-existant attribute (or member) in javascripts returns undefined:
 +
<pre>
 +
var x = {};
 +
alert(x.foo); // alerts "undefined"
 +
alert(x.foo.bar); // yields exception
 +
</pre>
 +
While in python, a similar operation yields exception sooner:
 +
<pre>
 +
>>> x={}
 +
>>> x["foo"]
 +
Traceback (most recent call last):
 +
  File "<stdin>", line 1, in <module>
 +
KeyError: 'foo'
 +
</pre>
 +
Also, it makes sense to fetch only what is required for the kind of view that you are interested in. If you are analysing session tokens, it is less resource intensive on your machine not to fetch the html content of each response.
  
 
=== What to show: Database Filtering ===
 
=== What to show: Database Filtering ===

Revision as of 19:24, 10 April 2011

Main

Hatkit-datafiddler-logo.png

The Hatkit Datafiddler is a tool for performing analysis of captured http traffic. It currently consists of two main views, one table-based and one tree-based. These views allow the user to study different aspects of the http traffic, with very high degree of configurability. The tool is also meant to be a framework which can utilize existing tools analyze traffic.

It is written in Python with a Qt-based UI and uses a MongoDB database. It has a sister-project, which is the Hatkit Proxy


Getting started

First of all, visit BitBucket download page to check which is the latest release. Then get it:

$ wget https://bitbucket.org/holiman/hatkit-datafiddler/downloads/hatkit_datafiddler-0.5.0.zip
$ unzip hatkit_datafiddler-0.5.0.zip 
$ cd hatkit_datafiddler-0.5.0/
$ python datafiddler.py

Datafiddler will tell you about any missing dependencies with something like this:

Unfortunately, you have the following missing dependencies:
 * python-qt4 : Python bindings for Qt4
 * pymongo : Python drivers for MongoDB

Fetch them via your favourite package manager (on *nix systems. Windows is currently not endorsed). Naturally, you need a MongoDB also. MongoDB is available either from the package repositories or from MongoDB download section.

If all goes well, you should be met by this screen, where you can choose which session to use. Sessions are really just databases, but Datafiddler only lists the databases in your MongoDB which contain a collection called conversations.

Hatkit-datafiddler-startup.png

Table view

An example of how the table view window can look
Via Settings, you can access the table definitions
Another example, using a raw python expression in the definition for column 3 and "Python" as column title
The tabledata settings also has filters
The filters can be arbitrary javascript expressions

If you select Settings, you will be met by the settings-window. On the left side, there are variables. For each object which is fetched from the database, these expressions determines exactly what parts are fetched and places these parts, into python variables with the names v0 and onward.

Tip: The section below is pretty technical. You don't have to know python or javascript to use this tool, Datafiddler comes with predefined expressions and views that you can use. When you learn the ropes a bit, you can just make modifications to these and you should be fine. More predefined views are coming.

For example, a database object stored by Hatkit Proxy always contain these fields:

request
response

(For more details about storage format, see OWASP_Hatkit_Proxy_Project#tab=Storage Storage If the request part of a database object is loaded into v0, it means that v0 will contain python dictionary containing everything that concerns the request. E.g. The python expression v0['method'] will be the request verb (GET/POST/FOO),while the expression v0['headers'] will be another python dictionary containing the request headers.

This means that this object introspection can be performed either inside the database - which is using javascript, or in the application itself, using python. Example:

v0 = request.headers.Host === (v0 = request)['headers']['Host]

Worth mentioning though, is that accessing a non-existant attribute (or member) in javascripts returns undefined:

var x = {};
alert(x.foo); // alerts "undefined"
alert(x.foo.bar); // yields exception

While in python, a similar operation yields exception sooner:

>>> x={}
>>> x["foo"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'foo'

Also, it makes sense to fetch only what is required for the kind of view that you are interested in. If you are analysing session tokens, it is less resource intensive on your machine not to fetch the html content of each response.

What to show: Database Filtering

Todo

How to show it

Todo

Transformers

Todo

Aggregation

Todo

Database Filtering

Todo

Development

Todo

Getting the source code

Todo

Project About

PROJECT INFO
What does this OWASP project offer you?
RELEASE(S) INFO
What releases are available for this project?
what is this project?
Name: OWASP Hatkit Datafiddler Project (home page)
Purpose:
  • The Datafiddler is a tool for performing advanced analysis of http traffic. It currently consists of two main views, one table-based and one tree-based. These views allow the user to study different aspects of the http traffic, with very high degree of configurability. The tool is also meant to be a framework which can utilize existing tools to analyze traffic post mortem (or real-time).
  • Built in Python/Qt + MongoDB.
License: GNU General Public License v3
who is working on this project?
Project Leader(s):
how can you learn more?
Project Pamphlet: Not Yet Created
Project Presentation: View
Mailing list: Mailing List Archives
Project Roadmap: View
Main links:
Key Contacts
current release
Hatkit Fiddler v 0.5.0 - April 9 2011 - (download)
Release description:
  • This tool allows you to analyze data which has been catpured into a MongoDB. You can view the data in table-mode (with dynamically evaluated column-definitions which can reach into html, transform values etc), aggregate-mode (performs advanced data aggregation, showing different characteristics on the data depeneding on how you define (or use the pre-defined) javascript aggregators, which are sent right into MongoDB). It also has a third-party-app replay functionality, to let w3af or ratproxy do their thing and analyze the data for you.
  • Requirements: Python, Python-qt bindings, Qt4, MongoDB, Python Mongodb drivers.
Rating: Yellow button.JPG Not Reviewed - Assessment Details
last reviewed release
Not Yet Reviewed


other releases