This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit https://owasp.org

Difference between revisions of "OWASP Security Logging Project"

From OWASP
Jump to: navigation, search
(Undo revision 188134 by Miltonsmith (talk))
(2nd attempt initial post, fixed up headings bug)
Line 86: Line 86:
  
  
−
= Road Map and Getting Involved =
+
Today many logging technologies are available providing powerful application logging capabilities.  But while powerful, these technologies are not designed for specific use-cases like security and auditing.  The generalized approach to logging platforms makes these platforms more useful to the widest possible audience but it also places more responsibility on designers.  In short, we don't consider our desire for additional improvement for security and audit logs is no oversight on the part of logging platform designers.
  
−
<!-- Instructions are in RED and should be removed from your document by deleting the text with the span tags.-->
+
It's the OWASP Security Logging Project desire to leverage existing technologies and apply them to improve security, audit, in addition to diagnostic logging. We understand logging is mostly an afterthought on many project schedules, if it's included at all.   We believe a logging solution embracing this project will help the community produce better logs, a better understanding of our information systems, and higher quality software.
−
<span style="color:#ff0000">
 
−
A project roadmap is the envisioned plan for the project. The purpose of the roadmap is to help others understand where the project is going as well as areas that volunteers may contribute. It gives the community a chance to understand the context and the vision for the goal of the project. Additionally, if a project becomes inactive, or if the project is abandoned, a roadmap can help ensure a project can be adopted and continued under new leadership.
 
−
Roadmaps vary in detail from a broad outline to a fully detailed project charter. Generally speaking, projects with detailed roadmaps have tended to develop into successful projects. Some details that leaders may consider placing in the roadmap include: envisioned milestones, planned feature enhancements, essential conditions, project assumptions, development timelines, etc. You are required to have at least 4 milestones for every year the project is active.  
 
−
</span>
 
−
==Roadmap==
 
  
 +
==Getting involved==
 +
Are you passionate about logging?  Are you motivated share your time and knowledge with the community?  Send the project leads an email, listed on project home page, and explain your ideas and how you can help.  Don't be discouraged if we don't immediately respond.  We occasionally get distracted with life but rest assured we will respond.
  
 +
==What is the OWASP Security Logging Project?==
 +
OWASP Security Logging Project purpose is to deliver a suitable logging solution for general-purpose security, audit, and diagnostics log messaging.  Beyond code and technology, the project provides architectural and implementation considerations you may find useful in your own projects, or technologies you may not have previously considered.
  
−
==Getting Involved==
+
==Project goals==
 +
* Develop a set of logging requirements for key domains like security, auditing, and diagnostics
 +
* Develop interface specifications that support the projects requirements
 +
* Develop a base implementation supporting project interface specifications
 +
* Develop documentation artifacts (described later)
  
 +
==Considerations and restraints==
 +
* Ease of use
 +
* Compelling value on initial deployment (without any refactoring).  Increased value for refactoring
 +
* Compatibility with existing industry standard logging technologies (e.g., log4*, logback, FluentD, etc)
 +
* Typical scenarios considered, 1) stand-alone applications on mobile or desktop, 2) enterprise applications, and 3) cloud-based applications.
  
−
===Coding===
+
==Anticipated support==
 +
* Java 1.7 and Java 1.8
 +
* .NET (tbd)
 +
''We have considered other platforms for the future but everything depends upon community interest and support.''
  
−
===Localization===
+
==Proposed features==
−
===Testing===
+
Following is a list of numbered features. 
  
−
===Feedback===
+
:1. MDC metadata improvements
 +
:: a. process id (TBD)
 +
:: b. application id and application instance id
 +
:: c. server time\date in UTC
 +
:: d. client time\date in UTC
 +
:: e. client IP address
 +
:: f. username or ID
 +
:: g. global client session ID
 +
:: h. security policy identifier
 +
:: i. transaction id
 +
:2. Log system properties on startup
 +
:3. Log command line options on startup
 +
:4. Log application server properties on startup
 +
:5. Log HTTP request parameters
 +
:6. Log HTTP session attributes
 +
:7. Internationalization considerations
 +
:8. Redirect system streams like system.out and system.err security logging framework
 +
:9. Asynchronous message logging, store and forward
 +
:10. Message correlation
 +
:11. Performance options for transport compression
 +
:12. Authenticated client logging
 +
:13. Secure log message transport
 +
:14. Signed log messages
 +
:15. Guaranteed log message delivery
  
 +
==Delivery phases==
 +
'''Alpha 1''', some features code complete.<br/>
 +
'''Alpha 2''', more features code complete.<br/>
 +
'''Beta''', release code complete.  Public encouraged to test and respond with comments.<br/>
 +
'''Early Availability(EA)''', includes improvements to beta based upon public and team recommendations.<br/>
 +
 +
==Use-case applicability & delivery schedule==
 +
The following table shows a proposed applicability of each feature to the projects areas of concern, diagnostics, security, and audit logging along with a suggested delivery phase.
 +
 +
{| class="wikitable" style="color:#555555; background-color:#ffffcc;" cellpadding="10"
 +
!&nbsp;
 +
!Diagnostics
 +
!Security
 +
!Audit
 +
!Delivery
 +
|-
 +
|Feature 1a, process id
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1b, application id and application instance id
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1c, server time\date in UTC
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1d, client time\date in UTL
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1e, client IP address
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1f, username or ID
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1g, global client session ID
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1h, security policy identifier
 +
|&nbsp;
 +
|'''M'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 1i, transaction id
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 2, Log system properties on startup
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 3, Log command line properties on startup
 +
|'''X'''
 +
|&nbsp;
 +
|&nbsp;
 +
|Alpha 1
 +
|-
 +
|Feature 4, Log application server properties on startup
 +
|'''X'''
 +
|&nbsp;
 +
|&nbsp;
 +
|Alpha 2
 +
|-
 +
|Feature 5, Log HTTP request parameters
 +
|'''X'''
 +
|'''X'''
 +
|&nbsp;
 +
|Alpha 2
 +
|-
 +
|Feature 6, Log HTTP session attributes
 +
|'''X'''
 +
|'''X'''
 +
|'''?'''
 +
|Alpha 2
 +
|-
 +
|Feature 7, Internationalization considerations
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 8, Redirect system streams like System.out and System.err to logging framework
 +
|'''X'''
 +
|&nbsp;
 +
|&nbsp;
 +
|Alpha 2
 +
|-
 +
|Feature 9, Asynchronous message logging, store and forward
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 2
 +
|-
 +
|Feature 10, Message correlation
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 2
 +
|-
 +
|Feature 11, Performance options for transport compression
 +
|'''X'''
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 2
 +
|-
 +
|Feature 12, Authenticated client logging
 +
|&nbsp;
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 2
 +
|-
 +
|Feature 13, Secure log message transport
 +
|&nbsp;
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 2
 +
|-
 +
|Feature 14, Signed log messages
 +
|&nbsp;
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 1
 +
|-
 +
|Feature 15, Guaranteed log message delivery
 +
|&nbsp;
 +
|'''X'''
 +
|'''X'''
 +
|Alpha 2
 +
|}
 +
'''''Legend, X=applicable use-case, M=maybe useful, ?=tbd'''''
 +
 +
==Project delivery artifacts==
 +
:'''Logging primer''', architectural considerations for security, audit, and diagnostics for community projects.  Provide information how logging project can be leverage to address concerns provided by each use case, general logging best practices, template for using message levels (e.g., INFO, WARN, etc).
 +
:'''Logging design''', specific technical details to apply project logging to community logging projects.
 +
:'''Code''', software program code that implements project feature goals.
 +
 +
==Code areas==
 +
:'''Logging layouts''', at the moment this is Common Event Format(CEF) and Common Log File System(CLFS).
 +
:'''MDC filter''', include system information handy for most deployments into logbacks Mapped Diagnostics Context(MDC).
 +
:'''MDC marker''',
 +
:'''Unit testing''', various software code we use (and you can also use) to test project code.
 +
 +
==Detailed use-case descriptions==
 +
Following are detailed use-case descriptions for each feature.  The purpose of this section is to help readers to understand more about each feature and it's potential benefits.
 +
 +
==Feature 1, MDC metadata improvements==
 +
This feature adds certain metadata useful for security purposes to logback’s Mapped Diagnostics Content.  The following metadata will be mapped where available.
 +
 +
===process id (feature 1a)===
 +
This is the process id of the application as assigned by the operating system at execution.  On *nix and Windows environments this the PID.  Depending upon the language platform process id may not be readily available.  As an alternative, server hostname or IP may be used.
 +
 +
===application id and application instance id (feature 1b)===
 +
This an identifier set by the application designer to identify a unique application instance.  This identifier is useful to identify applications uniquely where many instances of the same program (e.g., web application) are hosted on 1 or more physical servers.  The application id is useful visual indicator of the type of application component.  The instance id is useful to identify the application instance.  The instance is particularly useful where the same process may host 2 or more application instances.  An instance id may be a generated hash (e.g., VMID) or unique index where size is a concern.  Once the id is used it should persist between process restarts.  A suggested format:  {APP ID}:{APP INSTANCE ID}.  An sample POS:ace22c02aa858f670e3c227fbab141e2d8d6bea6 or POS:14563.
 +
 +
===server time\date in UTC (feature 1c)===
 +
Time, date, and day, on the server with timezone offset at the time the message was logged.  A suggested format[1], {yyyy-MM-dd'T'HH:mm:ss.SSSZ}.  An example, 2001-07-04T12:08:56.235-0700
 +
 +
===client time\date in UTC (feature 1d)===
 +
Time, date, and day, on the client with timezone offset at the time the message was logged.  A suggested format[1], {yyyy-MM-dd'T'HH:mm:ss.SSSZ}.  An example, 2001-07-04T12:08:56.235-0700
 +
 +
===client ip address (feature 1e)===
 +
MDC property for the IP address of the client host where the log message originated.  An example, 192.168.1.30
 +
 +
===user name or ID (feature 1f)===
 +
This MDC property to property is an application account name associated with a human (if available) this is associated with this log message.  This property may not be available if the log message is not specifically related to an individual's activity.  An example, milton.smith
 +
 +
===global client session id (feature 1g)===
 +
This MDC property is a session id assigned by an application designer that is shared across multiple application instances.  Usually this is a secure hash to avoid reverse engineering.  An example, ace22c02aa858f670e3c227fbab141e2d8d6bea6
 +
 +
===security policy identifier (feature 1h)===
 +
MDC property that identifies activities associated with a sites security policy.  The value is site defined and can be useful when producing information for audits.  An example, Violation:SEC.5.2a
 +
 +
===transaction id (feature 1i)===
 +
MDC property to identify activities associated with a single user action.  For example, execution of a single application user feature may require many activities from the main application program along with components like LDAP servers and databases.  The transaction id is useful to correlate all the related system activities that support a specific user request.  Each subsequent user request receives a new transaction id.  An example, TRX:1005862
 +
 +
==Feature 2, Log system properties on startup==
 +
The requirement is to log all system properties on application startup.  Often it’s difficult to perform an investigation without understanding the initial state of the system.  An example how properties may appear in logs (without MDC information).
 +
 +
<pre>
 +
*******************************************
 +
JAVA PROPERTY SETTINGS
 +
*******************************************
 +
Setting, java.runtime.name=Java(TM) SE Runtime Environment
 +
Setting, sun.boot.library.path=C:\Program Files\Java\jre6\bin
 +
Setting, java.vm.version=14.0-b16
 +
Setting, java.vm.vendor=Sun Microsystems Inc.
 +
Setting, java.vendor.url=http://java.sun.com/
 +
Setting, path.separator=;
 +
Setting, java.vm.name=Java HotSpot(TM) Client VM
 +
Setting, file.encoding.pkg=sun.io
 +
Setting, sun.java.launcher=SUN_STANDARD
 +
Setting, user.country=US
 +
Setting, sun.os.patch.level=
 +
Setting, java.vm.specification.name=Java Virtual Machine Specification
 +
Setting, user.dir=C:\Users\Milton\workspace\MyProject
 +
Setting, java.runtime.version=1.6.0_14-b08
 +
Setting, java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
 +
Setting, java.endorsed.dirs=C:\Program Files\Java\jre6\lib\endorsed
 +
Setting, os.arch=x86
 +
Setting, java.io.tmpdir=C:\Users\Milton\AppData\Local\Temp\
 +
Setting, line.separator=
 +
   
 +
Setting, java.vm.specification.vendor=Sun Microsystems Inc.
 +
Setting, user.variant=
 +
Setting, os.name=Windows 7
 +
Setting, sun.jnu.encoding=Cp1252
 +
Setting, java.library.path=C:\Program Files\Java\jre6\bin;.;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:/Program Files/Java/jre6/bin/client;C:/Program Files/Java/jre6/bin;C:\Program Files\JavaFX\javafx-sdk1.2\bin;C:\Program Files\JavaFX\javafx-sdk1.2\emulator\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;c:\usershellcommands;C:\Program Files\QuickTime\QTSystem\
 +
Setting, java.specification.name=Java Platform API Specification
 +
Setting, java.class.version=50.0
 +
Setting, sun.management.compiler=HotSpot Client Compiler
 +
Setting, os.version=6.1
 +
Setting, user.home=C:\Users\Milton
 +
Setting, user.timezone=
 +
Setting, java.awt.printerjob=sun.awt.windows.WPrinterJob
 +
Setting, file.encoding=Cp1252
 +
Setting, java.specification.version=1.6
 +
Setting, java.class.path=C:\Users\Milton\workspace\SDA\bin;C:\Java-Libs\jmx-1_2_1-bin\lib\jmxri.jar;C:\Java-Libs\apache-log4j-1.2.15\log4j-1.2.15.jar
 +
Setting, user.name=Milton
 +
</pre>
 +
 +
==Feature 3, Log command line options on startup==
 +
The requirement is to log all command line arguments on application startup.  All command line arguments must be logged.  In Java, the entire arg array passed into the main(String args[]) method should be logged.  Any whitespace or special characters should be filtered before logged.  For example a small program that echos the input to the command line may produce an output that looks like the following.
 +
 +
<pre>
 +
*******************************************
 +
    COMMAND LINE ARGS
 +
*******************************************
 +
java testapp “Hello World!”
 +
Hello World!
 +
</pre>
 +
 +
==Feature 4, Log application server properties on startup==
 +
The requirement is to log all key\value pairs that influence application behavior upon execution.  In Java, there parameters are defined by HttpServlet.getInitParameterNames()  An example of logged J2EE properties may look like the following.
 +
 +
<pre>
 +
*******************************************
 +
    J2EE PROPERTIES
 +
*******************************************
 +
Setting, thread.pool.size=1000
 +
Setting, request.ttlms=30000
 +
</pre>
 +
 +
==Feature 5, Log HTTP request parameters==
 +
The requirement is to log all key\value pairs associated with all application HTTP requests.  Raw HTTP requests parameters across the cloud may generate significantly increase log volume.  The goal is to define a request log that overwrites itself (e.g., a ring buffer) at a small designer specified interval or a default of 15 mins.  This allows highly granular diagnostic messages over a short duration.
 +
 +
An ancillary requirement is that sensitive key\value pairs will be masked.  A default set of masking rules will be included with the project with an option for designers to assign their own masking rules specific for their applications.
 +
 +
(TODOMS: need to insert some raw http requests from zap in a suitable log format)
 +
 +
==Feature 6, Log HTTP session attributes==
 +
 +
The requirement is to log all key\value pairs associated with a users HttpSession instance.  These properties should be logged once upon user session initialization.  In Java, key\value pairs from HttpSession.getAttributeName() should be logged when the HttpSession is created.
 +
 +
(TODOMS: need to insert some sample HTTP session attributes)
 +
 +
==Feature 7, Internationalization considerations==
 +
The action is to use string resources so that logs are compatible across languages.  The project will initially define US English.  Designers are encouraged to translate resources to different languages.  If the translations are made available to us we may include them.
 +
 +
==Feature 8, Redirect system streams like System.out and System.err to security logging framework==
 +
This requirement captures any legacy messaging from older code without refactoring.  The approach redirects any messages to system defined streams into the logging framework.  Log messages will not be a content rich since since the caller, old code in this case, does not calling the Security API directly.  The advantage is instant out of the box compatibility with no refactoring.  In Java, the action is to capture calls like System.out.println(“My wife loves security.”) and System.err() reroute them to the logging framework without modification to legacy programs.
 +
 +
An ancillary requirement is that sensitive key\value pairs will be masked.  A default set of masking rules will be included with the project with an option for designers to assign their own masking rules specific for their applications.
 +
 +
==Feature 9, Asynchronous message logging, store and forward==
 +
The requirement for this feature one of performance.  Log messages sent to a remote location (e.g., central log server) can take some time to send over networks.  It may be desirable in some deployments for the caller not to block when logging these messages.  The goal is to log the message locally, freeing the caller, then send the message in a background thread to the remote server.  See Feature 15 also.
 +
 +
==Feature 10, Message correlation==
 +
A problem with logs today is that it’s often difficult to reconstruct a series of activities leading to an event of interest.  System logs are often out of order with messages originating from different threads and hosts.  The goal of message correlation is to provide identifier(s) so that all log messages can be sequenced into a narrative of system activities leading to an event of interest.  For example, with correlation it will be possible to separate log entries to see the activities involved in a single administrative user operation like Add User.  Log entries to add a user may begin with HTTP posts from the clients browser, system permission checks, next a log message describing the insert of the new user into the user table, a log message of positive confirmation a SMTP message was sent to indicate the users new account is ready for initial signon.
 +
 +
==Feature 11, Performance options for transport compression==
 +
Where log message will transit networks facilities will be provided to compress traffic to remote hosts.
 +
 +
==Feature 12, Authenticated client logging==
 +
This feature is useful to ensure each message logged is attributable to a known source and trusted source.  Messages from anonymous sources may still be allowed, depending upon system preferences, but authenticated messages will clearly indicate the identity of the source.
 +
 +
==Feature 13, Secure transport==
 +
To facilitate secure transport a TLS 1.2 compliant connection be negotiated.  Options must be provided to allow designers to control ciphersuite negotiation.  Negotiation options must include provision for, a) the name of each ciphersuite permitted, b) order of negotiation which is ideally strongest suites first as a default but can be changed by the designer.  The trust roots will be those supplied by the supporting language platform (e.g., Java, .NET, etc).
 +
 +
==Feature 14, Signed log messages==
 +
To facilitate tamper resistant log messages log messages will be signed by the client.  Each field of the log message will be included in the signing process.  The signature will be included with the log message entry along with strongest fingerprint included within signing certificate.  The fingerprint of the signing certificate is an aid to identify the signing certificate and may be important for enterprise or cloud environments where many clients are logging.  Signed logs may or may not be encrypted.
 +
 +
==Feature 15, Guaranteed log message delivery==
 +
This feature builds upon the Feature 9, Asynchronous message logging, store and forward to include guaranteed delivery.  The goal is that no messages are lost.  Messages received from the caller will be queued for delivery.  Clients logging messages must block until their log message is committed to a queue.  For simplicity, the queue will exist on the client computer.  The function is somewhat analogous to a local print spooler.  If committing to a queue is not possible an instance of a RuntimeException must be thrown to the caller.  Once committed to a queue, worker threads will send the message in the background to the remote server.  On the client, worker threads will not remove the log message from the queue until the server has acknowledged receipt.
 +
 +
From the server side, the server must maintain the client connection until the message is logged.  If the message cannot be logged an instance of an Exception must be thrown.  Using this system no message will ever be lost.  A message will exist in only 3 states, 1) with the blocked client, 2) within the client’s log queue, 3) logged on the server.  For a completely reliable solution, HA hardware and RAID media are required which is a consideration for system designers.
 +
 +
==Feedback==
 +
Please report any concerns, correction, or other feedback to any of the project leads listed on the main project page.
  
 
=Minimum Viable Product=
 
=Minimum Viable Product=

Revision as of 00:36, 15 January 2015

OWASP Project Header.jpg

OWASP Security Logging Project

The OWASP Security Logging project provides developers and ops personnel with APIs for logging security-related events. The aim is to let developers use the same set of logging APIs they are already familiar with from over a decade of experience with Log4J and its successors, while also adding powerful security features.

Description

Logging is often neglected by developers when thinking of security considerations. However, proper logging practice can provide the crucial forensics needed to investigate after a breach, and perhaps more importantly, a change to detect security issues as they happen. Most developers are already familiar with using logging for debugging and diagnostic purposes, so it should be easy for them to grasp the concept of security logging as well. The OWASP Security Logging project aims to give developers an easy way to get started with logging security events, tracking extra forensic information like the who (username), what (event type), and where (IP address, server name) needed for forensics. It also provides a means for classifying the information in log messages and applying masking if necessary.

Licensing

This library is free software: you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0. You can copy, distribute and transmit the work, and you can adapt it, and use it commercially, but all provided that you attribute the work and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

Project Resources

Source Code

Documentation

Issue Tracker


Project Leader

Sytze van Koningsveld

August Detlefsen

Milton Smith


Related Projects


Classifications

Project Type Files CODE.jpg
Incubator Project Owasp-builders-small.png
Owasp-defenders-small.png
Affero General Public License 3.0

News and Events

23 Dec 2014 Project Created and source code now available!

Many projects have "Frequently Asked Questions" documents or pages. However, the point of such a document is not the questions. The point of a document like this are the answers. The document contains the answers that people would otherwise find themselves giving over and over again. The idea is that rather than laboriously compose and post the same answers repeatedly, people can refer to this page with pre-prepared answers. Use this space to communicate your projects 'Frequent Answers.'

How can I participate in your project?

All you have to do is make the Project Leader's aware of your available time to contribute to the project. It is also important to let the Leader's know how you would like to contribute and pitch in to help the project meet it's goals and milestones. There are many different ways you can contribute to an OWASP Project, but communication with the leads is key.

If I am not a programmer can I participate in your project?

Yes, you can certainly participate in the project if you are not a programmer or technical. The project needs different skills and expertise and different times during its development. Currently, we are looking for researchers, writers, graphic designers, and a project administrator.

Volunteers

The success of OWASP is due to a community of enthusiasts and contributors that work to make our projects great. This is also true for the success of your project. Be sure to give credit where credit is due, no matter how small! This should be a brief list of the most amazing people involved in your project. Be sure to provide a link to a complete list of all the amazing people in your project's community as well.


Today many logging technologies are available providing powerful application logging capabilities. But while powerful, these technologies are not designed for specific use-cases like security and auditing. The generalized approach to logging platforms makes these platforms more useful to the widest possible audience but it also places more responsibility on designers. In short, we don't consider our desire for additional improvement for security and audit logs is no oversight on the part of logging platform designers.

It's the OWASP Security Logging Project desire to leverage existing technologies and apply them to improve security, audit, in addition to diagnostic logging. We understand logging is mostly an afterthought on many project schedules, if it's included at all. We believe a logging solution embracing this project will help the community produce better logs, a better understanding of our information systems, and higher quality software.

Getting involved

Are you passionate about logging? Are you motivated share your time and knowledge with the community? Send the project leads an email, listed on project home page, and explain your ideas and how you can help. Don't be discouraged if we don't immediately respond. We occasionally get distracted with life but rest assured we will respond.

What is the OWASP Security Logging Project?

OWASP Security Logging Project purpose is to deliver a suitable logging solution for general-purpose security, audit, and diagnostics log messaging. Beyond code and technology, the project provides architectural and implementation considerations you may find useful in your own projects, or technologies you may not have previously considered.

Project goals

  • Develop a set of logging requirements for key domains like security, auditing, and diagnostics
  • Develop interface specifications that support the projects requirements
  • Develop a base implementation supporting project interface specifications
  • Develop documentation artifacts (described later)

Considerations and restraints

  • Ease of use
  • Compelling value on initial deployment (without any refactoring). Increased value for refactoring
  • Compatibility with existing industry standard logging technologies (e.g., log4*, logback, FluentD, etc)
  • Typical scenarios considered, 1) stand-alone applications on mobile or desktop, 2) enterprise applications, and 3) cloud-based applications.

Anticipated support

  • Java 1.7 and Java 1.8
  • .NET (tbd)

We have considered other platforms for the future but everything depends upon community interest and support.

Proposed features

Following is a list of numbered features.

1. MDC metadata improvements
a. process id (TBD)
b. application id and application instance id
c. server time\date in UTC
d. client time\date in UTC
e. client IP address
f. username or ID
g. global client session ID
h. security policy identifier
i. transaction id
2. Log system properties on startup
3. Log command line options on startup
4. Log application server properties on startup
5. Log HTTP request parameters
6. Log HTTP session attributes
7. Internationalization considerations
8. Redirect system streams like system.out and system.err security logging framework
9. Asynchronous message logging, store and forward
10. Message correlation
11. Performance options for transport compression
12. Authenticated client logging
13. Secure log message transport
14. Signed log messages
15. Guaranteed log message delivery

Delivery phases

Alpha 1, some features code complete.
Alpha 2, more features code complete.
Beta, release code complete. Public encouraged to test and respond with comments.
Early Availability(EA), includes improvements to beta based upon public and team recommendations.

Use-case applicability & delivery schedule

The following table shows a proposed applicability of each feature to the projects areas of concern, diagnostics, security, and audit logging along with a suggested delivery phase.

  Diagnostics Security Audit Delivery
Feature 1a, process id X X X Alpha 1
Feature 1b, application id and application instance id X X X Alpha 1
Feature 1c, server time\date in UTC X X X Alpha 1
Feature 1d, client time\date in UTL X X X Alpha 1
Feature 1e, client IP address X X X Alpha 1
Feature 1f, username or ID X X X Alpha 1
Feature 1g, global client session ID X X X Alpha 1
Feature 1h, security policy identifier   M X Alpha 1
Feature 1i, transaction id X X X Alpha 1
Feature 2, Log system properties on startup X X X Alpha 1
Feature 3, Log command line properties on startup X     Alpha 1
Feature 4, Log application server properties on startup X     Alpha 2
Feature 5, Log HTTP request parameters X X   Alpha 2
Feature 6, Log HTTP session attributes X X ? Alpha 2
Feature 7, Internationalization considerations X X X Alpha 1
Feature 8, Redirect system streams like System.out and System.err to logging framework X     Alpha 2
Feature 9, Asynchronous message logging, store and forward X X X Alpha 2
Feature 10, Message correlation X X X Alpha 2
Feature 11, Performance options for transport compression X X X Alpha 2
Feature 12, Authenticated client logging   X X Alpha 2
Feature 13, Secure log message transport   X X Alpha 2
Feature 14, Signed log messages   X X Alpha 1
Feature 15, Guaranteed log message delivery   X X Alpha 2

Legend, X=applicable use-case, M=maybe useful, ?=tbd

Project delivery artifacts

Logging primer, architectural considerations for security, audit, and diagnostics for community projects. Provide information how logging project can be leverage to address concerns provided by each use case, general logging best practices, template for using message levels (e.g., INFO, WARN, etc).
Logging design, specific technical details to apply project logging to community logging projects.
Code, software program code that implements project feature goals.

Code areas

Logging layouts, at the moment this is Common Event Format(CEF) and Common Log File System(CLFS).
MDC filter, include system information handy for most deployments into logbacks Mapped Diagnostics Context(MDC).
MDC marker,
Unit testing, various software code we use (and you can also use) to test project code.

Detailed use-case descriptions

Following are detailed use-case descriptions for each feature. The purpose of this section is to help readers to understand more about each feature and it's potential benefits.

Feature 1, MDC metadata improvements

This feature adds certain metadata useful for security purposes to logback’s Mapped Diagnostics Content. The following metadata will be mapped where available.

process id (feature 1a)

This is the process id of the application as assigned by the operating system at execution. On *nix and Windows environments this the PID. Depending upon the language platform process id may not be readily available. As an alternative, server hostname or IP may be used.

application id and application instance id (feature 1b)

This an identifier set by the application designer to identify a unique application instance. This identifier is useful to identify applications uniquely where many instances of the same program (e.g., web application) are hosted on 1 or more physical servers. The application id is useful visual indicator of the type of application component. The instance id is useful to identify the application instance. The instance is particularly useful where the same process may host 2 or more application instances. An instance id may be a generated hash (e.g., VMID) or unique index where size is a concern. Once the id is used it should persist between process restarts. A suggested format: {APP ID}:{APP INSTANCE ID}. An sample POS:ace22c02aa858f670e3c227fbab141e2d8d6bea6 or POS:14563.

server time\date in UTC (feature 1c)

Time, date, and day, on the server with timezone offset at the time the message was logged. A suggested format[1], {yyyy-MM-dd'T'HH:mm:ss.SSSZ}. An example, 2001-07-04T12:08:56.235-0700

client time\date in UTC (feature 1d)

Time, date, and day, on the client with timezone offset at the time the message was logged. A suggested format[1], {yyyy-MM-dd'T'HH:mm:ss.SSSZ}. An example, 2001-07-04T12:08:56.235-0700

client ip address (feature 1e)

MDC property for the IP address of the client host where the log message originated. An example, 192.168.1.30

user name or ID (feature 1f)

This MDC property to property is an application account name associated with a human (if available) this is associated with this log message. This property may not be available if the log message is not specifically related to an individual's activity. An example, milton.smith

global client session id (feature 1g)

This MDC property is a session id assigned by an application designer that is shared across multiple application instances. Usually this is a secure hash to avoid reverse engineering. An example, ace22c02aa858f670e3c227fbab141e2d8d6bea6

security policy identifier (feature 1h)

MDC property that identifies activities associated with a sites security policy. The value is site defined and can be useful when producing information for audits. An example, Violation:SEC.5.2a

transaction id (feature 1i)

MDC property to identify activities associated with a single user action. For example, execution of a single application user feature may require many activities from the main application program along with components like LDAP servers and databases. The transaction id is useful to correlate all the related system activities that support a specific user request. Each subsequent user request receives a new transaction id. An example, TRX:1005862

Feature 2, Log system properties on startup

The requirement is to log all system properties on application startup. Often it’s difficult to perform an investigation without understanding the initial state of the system. An example how properties may appear in logs (without MDC information). 

 *******************************************
 JAVA PROPERTY SETTINGS
 *******************************************
Setting, java.runtime.name=Java(TM) SE Runtime Environment
Setting, sun.boot.library.path=C:\Program Files\Java\jre6\bin
Setting, java.vm.version=14.0-b16
Setting, java.vm.vendor=Sun Microsystems Inc.
Setting, java.vendor.url=http://java.sun.com/
Setting, path.separator=;
Setting, java.vm.name=Java HotSpot(TM) Client VM
Setting, file.encoding.pkg=sun.io
Setting, sun.java.launcher=SUN_STANDARD
 Setting, user.country=US
Setting, sun.os.patch.level=
Setting, java.vm.specification.name=Java Virtual Machine Specification
Setting, user.dir=C:\Users\Milton\workspace\MyProject
Setting, java.runtime.version=1.6.0_14-b08
Setting, java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
Setting, java.endorsed.dirs=C:\Program Files\Java\jre6\lib\endorsed
Setting, os.arch=x86
Setting, java.io.tmpdir=C:\Users\Milton\AppData\Local\Temp\
Setting, line.separator=
    
Setting, java.vm.specification.vendor=Sun Microsystems Inc.
Setting, user.variant=
Setting, os.name=Windows 7
Setting, sun.jnu.encoding=Cp1252
Setting, java.library.path=C:\Program Files\Java\jre6\bin;.;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:/Program Files/Java/jre6/bin/client;C:/Program Files/Java/jre6/bin;C:\Program Files\JavaFX\javafx-sdk1.2\bin;C:\Program Files\JavaFX\javafx-sdk1.2\emulator\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;c:\usershellcommands;C:\Program Files\QuickTime\QTSystem\
 Setting, java.specification.name=Java Platform API Specification
Setting, java.class.version=50.0
Setting, sun.management.compiler=HotSpot Client Compiler
Setting, os.version=6.1
Setting, user.home=C:\Users\Milton
Setting, user.timezone=
Setting, java.awt.printerjob=sun.awt.windows.WPrinterJob
Setting, file.encoding=Cp1252
Setting, java.specification.version=1.6
Setting, java.class.path=C:\Users\Milton\workspace\SDA\bin;C:\Java-Libs\jmx-1_2_1-bin\lib\jmxri.jar;C:\Java-Libs\apache-log4j-1.2.15\log4j-1.2.15.jar
Setting, user.name=Milton

Feature 3, Log command line options on startup

The requirement is to log all command line arguments on application startup. All command line arguments must be logged. In Java, the entire arg array passed into the main(String args[]) method should be logged. Any whitespace or special characters should be filtered before logged. For example a small program that echos the input to the command line may produce an output that looks like the following. 

 *******************************************
    COMMAND LINE ARGS
 *******************************************
java testapp “Hello World!”
Hello World!

Feature 4, Log application server properties on startup

The requirement is to log all key\value pairs that influence application behavior upon execution. In Java, there parameters are defined by HttpServlet.getInitParameterNames() An example of logged J2EE properties may look like the following. 

 *******************************************
    J2EE PROPERTIES
 *******************************************
Setting, thread.pool.size=1000
Setting, request.ttlms=30000

Feature 5, Log HTTP request parameters

The requirement is to log all key\value pairs associated with all application HTTP requests. Raw HTTP requests parameters across the cloud may generate significantly increase log volume. The goal is to define a request log that overwrites itself (e.g., a ring buffer) at a small designer specified interval or a default of 15 mins. This allows highly granular diagnostic messages over a short duration.

An ancillary requirement is that sensitive key\value pairs will be masked. A default set of masking rules will be included with the project with an option for designers to assign their own masking rules specific for their applications.

(TODOMS: need to insert some raw http requests from zap in a suitable log format)

Feature 6, Log HTTP session attributes

The requirement is to log all key\value pairs associated with a users HttpSession instance. These properties should be logged once upon user session initialization. In Java, key\value pairs from HttpSession.getAttributeName() should be logged when the HttpSession is created.

(TODOMS: need to insert some sample HTTP session attributes)

Feature 7, Internationalization considerations

The action is to use string resources so that logs are compatible across languages. The project will initially define US English. Designers are encouraged to translate resources to different languages. If the translations are made available to us we may include them.

Feature 8, Redirect system streams like System.out and System.err to security logging framework

This requirement captures any legacy messaging from older code without refactoring. The approach redirects any messages to system defined streams into the logging framework. Log messages will not be a content rich since since the caller, old code in this case, does not calling the Security API directly. The advantage is instant out of the box compatibility with no refactoring. In Java, the action is to capture calls like System.out.println(“My wife loves security.”) and System.err() reroute them to the logging framework without modification to legacy programs.

An ancillary requirement is that sensitive key\value pairs will be masked. A default set of masking rules will be included with the project with an option for designers to assign their own masking rules specific for their applications.

Feature 9, Asynchronous message logging, store and forward

The requirement for this feature one of performance. Log messages sent to a remote location (e.g., central log server) can take some time to send over networks. It may be desirable in some deployments for the caller not to block when logging these messages. The goal is to log the message locally, freeing the caller, then send the message in a background thread to the remote server. See Feature 15 also.

Feature 10, Message correlation

A problem with logs today is that it’s often difficult to reconstruct a series of activities leading to an event of interest. System logs are often out of order with messages originating from different threads and hosts. The goal of message correlation is to provide identifier(s) so that all log messages can be sequenced into a narrative of system activities leading to an event of interest. For example, with correlation it will be possible to separate log entries to see the activities involved in a single administrative user operation like Add User. Log entries to add a user may begin with HTTP posts from the clients browser, system permission checks, next a log message describing the insert of the new user into the user table, a log message of positive confirmation a SMTP message was sent to indicate the users new account is ready for initial signon.

Feature 11, Performance options for transport compression

Where log message will transit networks facilities will be provided to compress traffic to remote hosts.

Feature 12, Authenticated client logging

This feature is useful to ensure each message logged is attributable to a known source and trusted source. Messages from anonymous sources may still be allowed, depending upon system preferences, but authenticated messages will clearly indicate the identity of the source.

Feature 13, Secure transport

To facilitate secure transport a TLS 1.2 compliant connection be negotiated. Options must be provided to allow designers to control ciphersuite negotiation. Negotiation options must include provision for, a) the name of each ciphersuite permitted, b) order of negotiation which is ideally strongest suites first as a default but can be changed by the designer. The trust roots will be those supplied by the supporting language platform (e.g., Java, .NET, etc).

Feature 14, Signed log messages

To facilitate tamper resistant log messages log messages will be signed by the client. Each field of the log message will be included in the signing process. The signature will be included with the log message entry along with strongest fingerprint included within signing certificate. The fingerprint of the signing certificate is an aid to identify the signing certificate and may be important for enterprise or cloud environments where many clients are logging. Signed logs may or may not be encrypted.

Feature 15, Guaranteed log message delivery

This feature builds upon the Feature 9, Asynchronous message logging, store and forward to include guaranteed delivery. The goal is that no messages are lost. Messages received from the caller will be queued for delivery. Clients logging messages must block until their log message is committed to a queue. For simplicity, the queue will exist on the client computer. The function is somewhat analogous to a local print spooler. If committing to a queue is not possible an instance of a RuntimeException must be thrown to the caller. Once committed to a queue, worker threads will send the message in the background to the remote server. On the client, worker threads will not remove the log message from the queue until the server has acknowledged receipt.

From the server side, the server must maintain the client connection until the message is logged. If the message cannot be logged an instance of an Exception must be thrown. Using this system no message will ever be lost. A message will exist in only 3 states, 1) with the blocked client, 2) within the client’s log queue, 3) logged on the server. For a completely reliable solution, HA hardware and RAID media are required which is a consideration for system designers.

Feedback

Please report any concerns, correction, or other feedback to any of the project leads listed on the main project page.

This page is where you should indicate what is the minimum set of functionality that is required to make this a useful product that addresses your core security concern. Defining this information helps the project leader to think about what is the critical functionality that a user needs for this project to be useful, thereby helping determine what the priorities should be on the roadmap. And it also helps reviewers who are evaluating the project to determine if the functionality sufficiently provides the critical functionality to determine if the project should be promoted to the next project category.


This page is where you need to place your legacy project template page if your project was created before October 2013. To edit this page you will need to edit your project information template. You can typically find this page by following this address and substituting your project name where it says "OWASP_Example_Project". When in doubt, ask the OWASP Projects Manager. Example template page: https://www.owasp.org/index.php/Projects/OWASP_Example_Project


PROJECT INFO
What does this OWASP project offer you? 		RELEASE(S) INFO
What releases are available for this project?
what is this project?
Name: N/A
Purpose: N/A
License: N/A
who is working on this project?
Project Leader(s): N/A
how can you learn more?
Project Pamphlet: Not Yet Created
Project Presentation:
Mailing list: N/A
Project Roadmap: Not Yet Created
Key Contacts
  • Contact the GPC to contribute to this project
  • Contact the GPC to review or sponsor this project
current release
pending
last reviewed release
pending


other releases


Retrieved from "https://wiki.owasp.org/index.php?title=OWASP_Security_Logging_Project&oldid=188138"