This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit https://owasp.org

Category:OWASP ModSecurity Core Rule Set Project

From OWASP
Revision as of 15:17, 8 March 2010 by Paulo Coimbra (talk | contribs)

Jump to: navigation, search


About

ModSecuriity.JPG

Overview

ModSecurity is an Apache web server module that provides a web application firewall engine. The ModSecurity Rules Language engine is extrememly flexible and robust and has been referred to as the "Swiss Army Knife of web application firewalls." While this is certainly true, it doesn't do much implicitly on its own and requires rules to tell it what to do. In order to enable users to take full advantage of ModSecurity out of the box, we have developed the Core Rule Set (CRS) which provides critical protections against attacks across most every web architecture.

Unlike intrusion detection and prevention systems, which rely on signatures specific to known vulnerabilities, the CRS is based on generic rules which focus on attack payload identification in order to provide protection from zero day and unknown vulnerabilities often found in web applications, which are in most cases custom coded.

Why The Core Rule Set?

The focus of the core rule set is to be a "rule set" rather than a set of rules. What makes a rule set different than a set of rules?

  • Performance - The Core Rule Set is optimized for performance. The amount and content of the rules used predominantly determines the performance impact of ModSecurity, so the performance optimization of the rule set is very important.
  • Quality - While there will always be false positives, a lot of effort is put into trying to make the Core Rule Set better. Some of the things done are:
    • Regression tests - a regression test is used to ensure that every new version shipped does not break anything. Actually every report of a false positive, once solved, gets into the regression test.
    • Real traffic testing – A large amount of real world capture files have been converted to tests and sent through ModSecurity to detect potential false positives.
  • Generic Detection - The core rule set is tuned to detect generic attacks and does not include specific rules for known vulnerabilities. Due to this feature the core rule set has better performance, is more "plug and play" and requires less updates.
  • Event Information - Each rule in the Core Rule Set has a unique ID and a textual message. In the future rules are also going to be classified using a new tag action in ModSecurity, as well as longer information regarding each rule using comments in the files themselves.
  • Plug and Play – The Core Rule Set is designed to be as plug and play as possible. Since its performance is good and it employs generic detection, and since the number of false positives is getting lower all the time, the Core Rule Set can be installed as is with little twisting and tweaking.


Content

In order to provide generic web applications protection, the Core Rules use the following techniques:

  • Protocol compliance:
    • HTTP request validation - This first line of protection ensures that all abnormal HTTP requests are detected. This line of defense eliminates a large number of automated and non targeted attacks as well as protects the web server itself.
    • HTTP protocol anomalies - Common HTTP usage patterns are indicative of attacks.
    • Global constraints - Limiting the size and length of different HTTP protocol attributes, such as the number and length of parameters and the overall length of the request. Ensuring that these attributed are constrained can prevent many attacks including buffer overflow and parameter manipulation.
    • HTTP Usage policy – validate requests against a predefined policy, setting limitations request properties such as methods, content types and extensions.
  • Attack Detection:
    • Malicious client software detection - Detect requests by malicious automated programs such as robots, crawlers and security scanners. Malicious automated programs collect information from a web site, consume bandwidth and might also search for vulnerabilities on the web site. Detecting malicious crawlers is especially useful against comments spam.
    • Generic Attack Detection - Detect application level attacks such as described in the OWASP top 10. These rules employ context based patterns match over normalized fields. Detected attacks include:
      • SQL injection and Blind SQL injection.
      • Cross Site Scripting (XSS).
      • OS Command Injection and remote command access.
      • File name injection.
      • ColdFusion, PHP and ASP injection.
      • E-Mail Injection
      • HTTP Response Splitting.
      • Universal PDF XSS.
    • Trojans & Backdoors Detection - Detection of attempts to access Trojans & backdoors already installed on the system. This feature is very important in a hosting environment when some of these backdoors may be uploaded in a legitimate way and used maliciously.
  • Other:
    • Error Detection - Prevent application error messages and code snippets from being sent to the user. This makes attacking the server much harder and is also a last line of defense if an attack passes through.
    • XML Protection – The Core Rule Set can be set to examine XML payload for most signatures.
    • Search Engine Monitoring - Log access by search engines crawlers to the web site.

Download

Current Stable Release is always available here:

https://sourceforge.net/projects/mod-security/files/modsecurity-crs/0-CURRENT/

Bug Tracker

JIRA Ticket System:

https://www.modsecurity.org/tracker/browse/CORERULES

Installation

Quick Start

Core Rule Set Quick Setup

=============

To activate the rules for your web server installation: 

 1) The modsecurity_crs_10_config.conf includes management rules and directives
    that can control important CRS functions. Pay attention to
    the SecRuleEngine setting (On by default) and that the SecDefaultAction
    directive is set to "pass".  The 49 inbound blocking and 59 outbound blocking
    rules files use the "block" action which
    inherits this setting.  The effectively means that you can toggle the
    SecDefaultAction setting to decide if you would like to deny on an
    anomaly scoring/correlation match.

    Update the PARANOID_MODE variable setting if you want to become more 
    aggressive in your detection.  Caution - this will cause more false positives.

    Should also update the appropriate anomaly scoring levels that will be propagated
    to the inbound/outbound blocking files.

    Update the TX policy settings for allowed Request Methods, File Extensions, etc...

 2) Add the following line to your httpd.conf (assuming
    you've placed the rule files into conf/modsecurity_crs/):

       <IfModule security2_module>
               Include conf/modsecurity_crs/*.conf
               Include conf/modsecurity_crs/base_rules/*.conf
       </IfModule>

 3) Restart web server.

 4) Make sure your web sites are still running fine.

 5) Simulate an attack against the web server. Then check
    the attack was correctly logged in the Apache error log,
    ModSecurity debug log (if you enabled it) and ModSecurity
    audit log (if you enabled it).


Documentation

ModSecurity Core Rule Set (CRS)

(c) 2006-2010 Breach Secuiry Inc.

The ModSecurity Core Rule Set is provided to you under the terms and conditions of GPL version 2

This directory contains the files for Core ModSecurity Rule Set The rules are compatible with ModSecurity 2.5 (as of version 1.4.3)

Overview

Using ModSecurity requires rules. In order to enable users to take full advantage of ModSecurity immediately, Breach Security Inc. is providing a free Core rule set. Unlike intrusion detection and prevention systems which rely on signature specific to known vulnerabilities, the Core Rule Set provides generic protection from unknown vulnerabilities often found in web application that are in most cases custom coded. This is what we call "Attack Payload Detection."

Keep in mind that a predefined rule set is only part of the work required to protect your web site. We strongly urge you to consult Ivan Ristic's book, "ModSecurity Handbook" http://store.feistyduck.com/products/modsecurity-handbook and the ModSecurity Reference Manual - http://www.modsecurity.org/documentation/. The CRS is heavily commented to allow it to be used as a step-by-step deployment guide for ModSecurity.

For more information refer to the OWASP Core Rule Set Project page at http://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project

Core Rules Mail-list - Suscribe here: https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set Archive: https://lists.owasp.org/pipermail/owasp-modsecurity-core-rule-set/


CRS 2.0 Design Concepts

CRS < 2.0 - Self-Contained Rules

Older (<2.0) CRS used individual, “self-contained” actions in rules 

- If a rule triggered, it would either deny or pass and log
- No intelligence was shared between rules

Not optimal from a rules management perspective (handling false positives/exceptions) 

- Editing the regex could blow it up
- Typical method was to copy/paste rules into custom rules files and then edit rule logic
  and disable core rule ID.
- Heavily customized rules were less likely to be updated by the user

Not optimal from a security perspective 

- Not every site had the same risk tolerance
- Lower severity alerts were largely ignored
- Individual low severity alerts are not important but several low severity events
  in the same transaction are.

CRS 2.0 - Collaborative Detection

Rules logic has changed by decoupling the inspection/detection from the blocking functionality 

- Rules log.pass and set transactional variables (tx) to track anomaly scores and to 
  store meta-data about the rule match
- This TX rule match data can be used by other 3rd party rules (converter Emerging Threats
  Snort web attack rules) to more accurately correlate identified attacks with their
  attack vector locations.
- TX data of previous strong rule matches can also be used to conditionally apply weaker signatures
  that normally would have a high fasle positive rate.
- Rules also increase anomaly scores for both the attack category and global score which allows
  users to set a threshold that is appropriate for them.
- This also allows several low severity events to trigger alerts while individual ones are suppressed.
- Exceptions may be handled by either increasing the overall anomaly score threshold, or
  by adding rules to a local custom exceptions file where TX data of previous rule matches
  may be inspected and anomaly scores re-adjusted based on the false positive criteria.

User can now globally update which variables to inspect and the anomaly score settings in the modsecurity_crs_10_config.conf file. 

- PARANOID_MODE setting which will apply rules to locations that have a higher false positive rate
- INBOUND_ANOMALY_SCORE setting will be populated in the inbound blocking file and if a transaction
  score at the end of phase:2 is equal to or greater than this number, it will be denied.
- OUTBOUND_ANOMALY_SCORE setting will be populated in the outbound blocking file and it a transaction
  score at the end of phase:4 is equal to or greater than this number, it will be denied.

The CRS rules themselves are configured with the pass action, which allows all the rules to be processed and for the proposed anomaly scoring/collaborative detection concept to work. The inbound/outbound anomaly score levels may be set in the modsecurity_crs_10_config.conf file. These scores will be evaluated in the modsecurity_crs_49_inbound_blocking.conf and modsecurity_crs_59_outbound_blocking.conf files.

One of the top feedback items we have heard is that the CRS events in the Apache error_log file were very chatty. This was due to each rule triggering its own error_log entry. What most people wanted was for 1 correlated event to be generated that would give the user a higher level determination as to what the event category was.

To that end- each CRS rule will generate an audit log event Message entry but they will not log to the error_log on their own. These rules are now considered basic or reference events and may be reviewed in the audit log if the user wants to see what individual events contributed to the overall anomaly score and event designation.

After the transaction has completed (in the logging phase), the rules in the base_rules/modsecurity_crs_60_correlation.conf file will conduct further post-processing by analyzing any inbound events with any outbound events in order to provide a more intelligent/priority correlated event. 

- Was there an inbound attack?
- Was there an HTTP Status Code Error (4xx/5xx level)?
- Was there an application information leak?

If an inbound attack was detected and either an outbound application status code error or infolead was detected, then the overall event severity is raised - 

- 0: Emergency - is generated from correlation where there is an inbound attack and
  an outbound leakage.
- 1: Alert - is generated from correlation where there is an inbound attack and an
  outbound application level error.

Core Rule Set Content

In order to provide generic web applications protection, the Core Rule Set uses the following techniques:

HTTP Protocol Validation and Protection

Detecting violations of the HTTP protocol and a locally defined usage policy. This first line of protection ensures that all abnormal HTTP requests are detected. This line of defense eliminates a large number of automated and non targeted attacks as well as protects the web server itself.

Protocol vulnerabilities such as Response Splitting, Request Smuggling, Premature URL ending 

- Content length only for non GET/HEAD methods 
- Non ASCII characters or encoding in headers
- Valid use of headers (for example, content length is numerical)
- Proxy Access

Attack requests are different due to automation 

- Missing headers such as Host, Accept, User-Agent
- Host is an IP address (common worm propagation method)

Policy is usually application specific 

- Some restrictions can usually be applied generically
- White lists can be build for specific environments
  Limitations on Sizes
- Request size, Upload size
- # of parameters, length of parameter

Items that can be allowed or restricted 

- Methods - Allow or restrict WebDAV, block abused methods such as CONNECT, TRACE or DEBUG 
- File extensions – backup files, database files, ini files
- Content-Types (and to some extent other headers)

Automation Detection

Automated clients are both a security risk and a commercial risk. Automated crawlers collect information from your site, consume bandwidth and might also search for vulnerabilities on the web site. Automation detection is especially useful for generic detection of comments spam.

Detecting bots, crawlers, scanners and other surface malicious activity. Not aimed against targeted attacks, but against general malicious internet activity 

- Offloads a lot of cyberspace junk & noise
- Effective against comment spam
- Reduce event count

Detection of Malicious Robots 

- Unique request attributes: User-Agent header, URL, Headers
- RBL Check of IP addresses
- Detection of security scanners
- Blocking can confuse security testing software (WAFW00f)

This rules file is only relevant if you are concerned about comment SPAM attacks. The rules file will run an RBL check against the source IP address at SPAMHAUS and will cache the response for 1 day. If the client sends subsequent requests, it will be denied without having to re-run an RBL check.

This file will also look for comment SPAM posting attacks which submit URL links.

Common Web Attacks Protection

Common Web Attacks Protection Rules on the second level address the common web application security attack methods. These are the issues that can appear in any web application. Some of the issues addressed are: 

- SQL Injection
- Cross-Site Scripting (XSS)
- OS Command execution
- Remote code inclusion
- LDAP Injection
- SSI Injection
- Information leak
- Buffer overflows
- File disclosure

- OS command injection and remote command access
- Remote file inclusion
- Session Fixation

The rules in this file are considered BETA quality as they have not been rigorously tested. They attempt to address advanced attacks such as HTTP Parameter Pollution or use new rule features or techniques.

This rules file attempts to identify all directory traversal variations. It is prone to a high level of false positives so set PARANOID_MODE if you want to run these rules. 

- SQL injection and blind SQL injection

- Cross site scripting (XSS)

Breach Security Labs received authorization from PHPIDS (http://phpids.net/) to convert their rules and include them in the CRS 

- Thanks to Mario Heiderich

Converted version of PHPIDS Converter.php functionality. https://svn.php-ids.org/svn/trunk/lib/IDS/Converter.php These rules look for common evasion tactics.

Converted version of PHPIDS default_filters.xml data. https://svn.php-ids.org/svn/trunk/lib/IDS/default_filter.xml 

- Filters are heavily tested by the community and updated frequently
- ~70 regular expression rules to detect common attack payloads
- XSS
- SQL Injection
- RFI

Due to the high number of rules and the possible impact on performance, these rules have been placed in the optional_rules directory.

Breach Security Labs received authorization from ET to convert their Snort rules and include them in the CRS http://www.emergingthreats.net/

Converted the following ET Snort rule files 

- emerging-web_server.rules
- emerging-web_specific_apps.rules

Identifying attacks against known web vulnerabilities does have value 

- Raised threat level
- If done correctly, lessens false positives

The issue to overcome is that the PCRE RegExs used in the rules are pretty poor. What we want to do is to combine the *what* of our generic attack payload detection (attack payloads) with the *where* (attack vector - URL + Parameter Name) of the ET known vuln data. The approach we took was to have most of the ET rules look for the attack vector data and then simply check all saved TX data for a corresponding attack vector match.

Trojan Protection

ModSecurity Core Rule Set detects access to back doors installed on a web server. This feature is very important in a hosting environment when some of this backdoors may be uploaded in a legitimate way and used maliciously. In addition the Core Rule Set includes a hook for adding an Anti-Virus program such as ClamAV for checking file uploads. 

- Check uploading of http backdoor page
- Access detection
- Known signatures (x_key header)
- Generic file management output (gid, uid, drwx, c:\)

InfoLeakages

If all fails, the Core Rule Set will detect errors sent by the web server. Detecting and blocking errors prevents attackers from collecting reconnaissance information about the web application and also server as a last line of defense in case an attack was not detected eariler. 

- HTTP Error Response Status Codes
- SQL Information Leakage
- Stack Dumps
- Source Code Leakage

Request Header Tagging

This concept is similar to anti-SPAM SMTP apps that will add additional mime headers to emails providing the SPAM detection analysis information. The CRS is attempting to mimic this concept at the HTTP layer by adding additional request headers that provide insight into any ModSecurity events that may have triggered during processing. The advantage of this approach is that it allows a WAF to be in a detection-only mode while still providing attack data to the destination application server. The recieving app server may then inspect the WAF request headers and make a determination whether or not to process the transaction. This concept is valuable in distributed web environments and hosting architectures where a determination to block may only be appropriate at the destination app server.

This rules file will take all of the TX attack variable data and populate Apache ENV variables that Apache can then use to add X-WAF-Event request header data to the request.

Example showing the consolidated X-WAF-Events and X-WAF-Score data - 

GET /path/to/foo.php?test=1%27%20or%20%272%27=%272%27;-- HTTP/1.1
Host: www.example.com 
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-WAF-Events: TX: / 999935-Detects common comment types-WEB_ATTACK/INJECTION-ARGS:test, TX:999923-Detects JavaScript location/document property access and window access obfuscation-WEB_ATTACK/INJECTION-REQUEST_URI_RAW, TX:950001- WEB_ATTACK/SQL_INJECTION-ARGS:test
X-WAF-Score: Total=48; sqli=2; xss=
Connection: Keep-Alive

Presentations and Whitepapers

Current CRS v2 File:OWASP ModSecurity Core Rule Set.ppt presented at AppSec DC 2009.

Ofer Shezaf's presentation and whitepaper on the Core Rule Set v1 presented at 6th OWASP AppSec conference in Milan, Italy, in May 2007

Related Projects

ModSecurity-Open Source Web Application Firewall
OWASP Securing WebGoat using ModSecurity
OWASP AppSensor Project


Latest News and Mail List

Current Stable Version CRS 2.0.5

Version 2.0.5 - 02/01/2100

Improvements: 

- Removed previous 10 config files as they may conflict with local customized Mod configs.
- Added a new 10 config file that allows the user to globally set TX variables to turn on/off
  PARANOID_MODE inspection, set anomaly score levels and http policies.
  Must have ModSecurity 2.5.12 to use the macro expansion in numeric operators.
- Added Rule Logic and Reference links to rules descriptions.
- Added Rule IDs to all rules.
- Added tag data mapping to new OWASP Top 10 and AppSensor Projects, WASC Threat Classification
- Removed Apache limit directives from the 23 file
- Added macro expansion to 23 file checks.
- Added @pmFromFile check to 35 bad robots file
- Added malicious UA strings to 35 bad robots check
- Created an experimental rules file
- Updated HTTP Parameter Pollution (HPP) rule logic to concat data into a TX variable for inspection
- Removed TX inspections for generic attacks and reverted to standard ARGS inspection
  https://www.modsecurity.org/tracker/browse/MODSEC-120
- Updated the variable list for standard inspections (ARGS|ARGS_NAMES|XML:/*) and moved the other
  variables to the PARANOID list (REQUEST_URI|REQUEST_BODY|REQUEST_HEADERS|TX:HPP_DATA)
- Moved converted ET Snort rules to the /optional_rules directory
- Created a new Header Tagging ruleset (optional_rules) that will add matched rule data to the
  request headers.
- Updated Inbound blocking conf file to use macro expansion from the 10 config file settings
- Added separate anomaly scores for inbound, outbound and total to be evaluated for blocking.
- Updated the regex logic in the (1=1) rule to factor in quotes and other logical operators.
- Updated the SPAMMER RBL check rules logic to only check once per IP/Day.
- Added new outbound malware link detection rules.

Bug Fixes: 

- Removed Non-numeric Rule IDs
  https://www.modsecurity.org/tracker/browse/CORERULES-28
- Updated the variable list on SQLi rules.
- Fixed outbound @pmFromFile action from allow to skipAfter to allow for outbound anomaly scoring
  and blocking

Version 2.0.4 - 11/30/2009

Improvements: 

- Updated converted PHPIDS signatures (https://svn.php-ids.org/svn/trunk/lib/IDS/default_filter.xml)
- Updated PHPIDS rules logic to first search for payloads in ARGS and then if there is no match found
  then search more generically in request_body|request_uri_raw
- Updated PHPIDS rules logic to only set TX variables and to not log.  This allows for more clean
  exceptions in the 48 file which can then expire/delete false positive TX matches and adjust the
  anomaly scores.  These rules will then inspect for any TX variables in phase:5 and create appropriate
  alerts for any variable matches that exist.

Bug Fixes: 

- Added Anomaly Score check to the 60 correlation file to recheck the anomaly score at the end of
  phase:4 which would allow for blocking based on information leakage issues.

Project Mail List
Subscribe here
Use here

Contributors, Users and Adopters

Project Leader

Ryan Barnett

Project Contributors

Brian Rectanus

The Core Rule Set (CRS) project is sponsored by:
BreachSecurityLabs.jpg

Project Users

WASC Distributed Open Proxy Honeypot Project uses the Core Rule Set - http://projects.webappsec.org/Distributed-Open-Proxy-Honeypots

Akamai's WAF Service is based on a previous version of the Core Rule Set - http://wwwfp.akamai.com/html/about/press/releases/2009/press_121409.html


Project Details

PROJECT INFO
What does this OWASP project offer you? 		RELEASE(S) INFO
What does this OWASP project release offer you?
what is this project?
OWASP ModSecurity Core Rule Set Project

Purpose: ModSecurity is an Apache web server module that provides a web application firewall engine. The ModSecurity Rules Language engine is extrememly flexible and robust and has been referred to as the "Swiss Army Knife of web application firewalls." While this is certainly true, it doesn't do much implicitly on its own and requires rules to tell it what to do. In order to enable users to take full advantage of ModSecurity out of the box, we have developed the Core Rule Set (CRS) which provides critical protections against attacks across most every web architecture.

Unlike intrusion detection and prevention systems, which rely on signatures specific to known vulnerabilities, the CRS is based on generic rules which focus on attack payload identification in order to provide protection from zero day and unknown vulnerabilities often found in web applications, which are in most cases custom coded.

License: Apache Software License - Version 2.0

who is working on this project?
Project Leader: Ryan Barnett @

Project Maintainer: Ryan Barnett @

Project Contributor(s):

how can you learn more?
Project Pamphlet: N/A

3x slide Project Presentation: View

Mailing list: Subscribe or read the archives

Project Roadmap: To view, click here

Main links:

Project Health: Yellow button.JPG Not Reviewed (Provisional)
To be reviewed under Assessment Criteria v2.0

Key Contacts
  • Contact Ryan Barnett @ to contribute, review or sponsor this project
  • Contact the GPC to report a problem or concern about this project or to update information.
current release
ModSecurity 2.2.5 - 2012-06-14 - (download)

Release Leader: Ryan Barnett @

Release details: Main links, release roadmap and assessment

Rating: Yellow button.JPG Not Reviewed
To be reviewed under Assessment Criteria v2.0



The CRS is an open source rule set licensed under GPLv2. ModSecurity Core Rule Set works with ModSecurity 2.5 and above.

Retrieved from "https://wiki.owasp.org/index.php?title=Category:OWASP_ModSecurity_Core_Rule_Set_Project&oldid=79308"