Category:OWASP Application Security Metrics Project Roadmap

Subject to Contributor feedback, the following Roadmap is proposed. Phase One tasks are more specifically defined. Phase Two tasks will be developed and defined over time based on Phase One experience.

Proposed Project Phases:

Phase One - Collect and Provide Proven Metrics

The objective is to provide useful current state metrics to the OWASP community in the near-term.

Phase Two - Develop “Next Generation” metrics

Develop new metrics using Contributor feedback on Phase One metrics and metrics that organizations have asked for but do not currently exist.

Summary of Proposed Phase One Tasks

• Comment Period for Proposed Project Approach, Solicit Contributor Support
• Develop Metric Collection Survey Instrument
• Solicit Organizational Participants
• Conduct Metric Collection Survey
• Organize and Analyze Collected Survey Data
• Prepare Draft Findings and Provide to Reviewers
• Comment Period for Published Draft Findings
• Publish Final Findings, Metrics and Resources
• Conduct Phase One Project Post-Mortem

Detailed Phase One Tasks

Task 1 – Comment Period for Proposed Project Approach

Solicit Contributor feedback to ensure the most effective and widely supported approach.

Target Time Frame: Completed Current Status: Call for Contributors

Task 2 – Develop Metric Collection Survey Instrument

Develop a survey instrument that can be used by Contributors to gather metrics data in a uniform fashion. Design considerations will include “organizational” demographic data required (e.g., industry vertical), the format of the metric description, etc. Ideally, we can categorize the metric types using a standard nomenclature. The final survey instrument will be based on the 80/20 principle – developing the “perfect” instrument will excessively delay the Project. Contributor support in developing a survey form that will allow efficient data aggregation and analysis would be appreciated (or at least ideas how this could be accomplished).

Target Time Frame: August, 2006 Current Status: Completed Contributors: Bob Austin

Task 3– Solicit Organizational Participants

Contributors will be asked to approach organizations that are known to have effective metrics in use and request their (anonymous) participation in the survey. It may be wise to limit the number of organizations participating in the survey to 30 or so organizations. One incentive to participate is the sharing of current “best practice” metrics. From a confidentiality perspective, each Contributor would ensure that data provided by an organization is sanitized to ensure anonymity.

Target Time Frame: Complete by August 15, 2006 Current Status: Need more survey participants!

Task 4 – Conduct Metric Collection Survey

Using the survey instrument, collect survey data. It may be wise to conduct a “pilot” survey with 1 or 2 organizations, make fine-tuning adjustments to the survey instrument, and then complete the surveys.

Target Time Frame: Complete by September 15, 2006 (this date is particularly dependent upon Contributor support) Current Status: Need more survey participants to complete!

Task 5 – Organize and Analyze Collected Survey Data

This will involve merging and organizing the collected data to allow effective analysis and presentation of the data.

Target Time Frame: Current Status: Need more survey data to complete!

Task 6 – Prepare Draft Findings and Provide to Reviewers

We envision capturing a number of key data points including a description of metrics used, consumers of the metrics, length of time used, barriers to metric collection, metrics needed, planned metrics initiatives, useful tools/resources that facilitate metrics collection/analysis, etc. We also will attempt to provide insight into management’s interest and support for the metrics program.

Target Time Frame: Current Status: Call for Volunteers

Task 7 – Comment Period for Published Draft Findings

Solicit feedback from the OWASP community, address errors/ambiguity. Make edits based on feedback.

Target Time Frame: Current Status: Call for Volunteers

Task 8 – Publish Final Findings, Metrics, and Resources

Self-explanatory. Ideally, present the metrics in a way that Contributors can continue to add to and comment on over time. Create a Resources Page that incorporates the resources recommended by survey participants.

Target Time Frame: Current Status: Call for Volunteers

Task 9 – Conduct Phase One Project Post-Mortem

Solicit feedback and lessons learned on Phase One to improve Phase Two approach.

Target Time Frame: Current Status: Call for Volunteers