Push for Quality

This is a DRAFT page still under review by the Global Projects Committee

This page is maintained by the Global Projects Committee to help assist Project Leaders with information about successfully running an OWASP Project. It will be updated from time to time, and changes will be discussed and announced on the OWASP-Leaders list.

Overview

OWASP created the project assessment criteria to define the quality levels for OWASP Projects with the purpose of evaluating all OWASP projects. The overall goal was to ensure that consistent quality levels are maintained by OWASP projects. This benefits both the external audience and those working on projects. The criteria allows the external audience to determine the quality of any OWASP project they are considering. For project members, it provides a method to measure the quality of their project in relation to other OWASP projects. Additionally, the criteria allows for excellent contributions to be recognized and projects which need further work to be identified.

Currently, OWASP projects fall into three primary categories:

• Tools
• Documents
• Activities and Research

The Tools and Documents categories are easily understood. The Activities and Research category is less obvious and is used for projects which either have multiple sub-projects or have deliverables which fall into both the tools and documents category. Thus, Activities and Research can be used for parent projects that cover multiple smaller sub-projects. Some examples will make this clearer:

• OWASP ESAPI
  • Java
  • .Net
  • PHP
  • ...

• OWASP Guides
  • Testing Guide
  • Development Guide
  • Code Review Guide
  • ASDR (Application Security Desk Reference)

• OWASP OpenPGP Extensions for HTTP - Enigform and mod_openpgp

All existing projects and their current ratings are here. Any new OWASP project and its deliverables will be assessed based on the criteria below as well as any Season of Code project. The goal is to eventually have all OWASP project deliverables, past and future, assessed under a version of this criteria. The initial set of assessment criteria was created for the OWASP Summer of Code 2008 and was designated version 1.0. The current version below was derived from version 1.0 and is version 2.0. Labelling any new criteria with a version number allows for graceful transitions to occur should any criteria change.

Assessing a project

Any OWASP project will consist of two critical pieces:

• the project's OWASP Wiki page
• one or more project deliverables

Each of these pieces will be have different methods with which they are reviewed. For OWASP project wiki pages, please see the Project Wiki Pages section of the Guidelines for OWASP Projects.

For project deliverables, OWASP has established the assessment criteria with three designations of quality: Alpha, Beta and Release. As project deliverables move up the quality ladder from Alpha to Beta and finally to Release quality, the amount of rigor required increases. In general, the project lead will determine the goal quality level of their project and work towards fulfilling the criteria for that level. Once a project lead has completed the prerequisites and criteria for the goal level, they request that their project be reviewed. The level will determine who and how those reviews occur.

• Alpha quality: The review consists of the Global Project Committee (GPC) verifying that the project pre-assessment check-list is complete. Alpha projects are the most free and open since anyone with solution to an application security problem can propose a new OWASP project. Alpha is where all new projects begin.
• Beta quality: The review will first be conducted by the project's reviewer (more on this below). After the reviewer completes the reviewer action items, the GPC will validate the project's assessment.
• Release quality: The two project reviewers will complete their action items (more on this below). After the reviews are complete, the Global Projects Committee and OWASP Board will validate the project's review.

Pre-Assessment Checklists versus Reviewer Action Items:

• Pre-Assessment checklists should be completed by the project lead prior to asking for an assessment.
• Pre-Assessment checklists were designed to be completed in minutes if all items are complete.
• Reviewer Action Items should be completed by project reviewer(s) after the project lead indicates the project is ready to be assessed.
• Reviewer Action Items were designed to require some significant time commitment from the reviewer since the questions are subjective and require a good deal of understanding of the project.

Prerequisites for project assessment

Depending on the quality level criteria, the project lead may have prerequisites before the project deliverable(s) can be assessed by the criteria below

• Alpha quality: No prerequisites.
• Beta quality: 1 reviewer is required.
• Release quality: 2 reviewers are required. Second review has special requirements.

Notes on reviewers

• Ideally, the project lead will suggest project reviewer(s).
• Ideally, reviewers should be an existing OWASP project lead or Chapter leader.
• If the project lead is unable to find the required reviewer(s), the Global Projects Committee can assist in identifying reviewers for the project.
• It is recommended that an OWASP board member or Global Projects Committee member be the second reviewer on Release quality deliverables. The board has the initial option to review a project, followed by the Global Projects Committee.
• The Global Projects Committee confirms the assignment of reviewers to a project.

Tool Assessment Criteria

Alpha Quality Tool Criteria

Pre-Assessment Checklist:

1. Is your tool licensed under an open source license? (see Project Licensing section of the Guidelines for OWASP Projects)
2. Is the Project Identification template on the project complete and accurate?
3. Is the source code and any documentation available in an online project site? (e.g. Google Code or Sourceforge site)
4. Is there an OWASP mail list for the project?
5. Is there a statement of the application security issue the tool addresses on the OWASP project page?

Beta Quality Tool Criteria

Pre-Assessment Checklist:

1. Is there an installer or stand-alone executable?
2. Is there user documentation on the OWASP project wiki page?
3. Is there an "About box" or similar help item which lists:
   1. Project Name
   2. Short Description
   3. Project Lead and contact information (e.g. email address)
   4. Project Contributors (if any)
   5. License
   6. Project Sponsors (if any)
   7. Release status and date assessed as Month-Year e.g. March 2009
   8. Link to OWASP Project Page (NOTE: I am unsure if this is needed --Mtesauro 21:06, 30 March 2009 (UTC))
4. Is there documentation on how to build the tool from source including obtaining the source from the code repository?
5. Is the tool documentation stored in the same repository as the source code?
6. Are the Alpha pre-assessment items complete?

Reviewer Action Items:

1. Is an installer for the tool available and easy to use? How close does it reach the goal of a fully automated installer?
2. Is the end user documentation complete, relevant and presented on the OWASP wiki page?
3. Does the tool have an “About box” or similar help item which allows the end user to get an overview of the state of this tool? Is this information readily available and easy to find?
4. Does the documentation on building the source provide the necessary information and detail to allow someone to build the tool? Is there sufficient detail and information for the target user? Is there any domain specific knowledge that is assumed and not provided?
5. Is the tool's documentation available with the source code and would it readily discoverable by a new user of the tool?

Release Quality Tool Criteria

Pre-Assessment Checklist:

1. Does the tool include documentation built into the tool?
2. Does the tool include build scripts to automate builds?
3. Is there a publicly accessible bug tracking system?
4. Have any limitations of the tool been documented?
5. Is there a conference style presentation that describes the tool in at least 3 slides?
6. Is there a one sheet overview document about the project?
7. Are the Alpha and Beta pre-assessment items complete?

Reviewer Action Items:

1. Does the tool substantially address the application security issues it was created to solve?
2. Is the tool reasonably easy to use?
3. Does the documentation meet the needs of the tool users and is easily found?
4. Do the build scripts work as expected? Can you build the tool? The goal is a “One-click” build.
5. Is the bug tracking system usable? Is it hosted at the same place as the source code? (e.g. Google Code, Sourceforge)
6. Have you noted any limitations of the tool that are not already documented by the project lead.
7. Have all the Beta Reviewer Action Items been completed? These will need to be completed if they have not already occurred during a previous assessment.

Document Assessment Criteria

Alpha Quality Document Criteria

Pre-Assessment Checklist:

1. Is your document licensed under a free and open license? (see Project Licensing section of the Guidelines for OWASP Projects)
2. Is the Project Identification template on the project wiki page complete and accurate?
3. Is the document compiled into an exported (PDF) and editable (.Doc) format available on-line?
4. Are all articles that constitute the project properly tagged within project category and available from main project Wiki page? (NOTE: Leo: I believe it’s good to have all project’s contents in the wiki, not only one-piece document.)
5. Is there an OWASP mail list for the project?
6. Is there a statement of the application security issue the document addresses on the OWASP project wiki page?

Beta Quality Document Criteria

Pre-Assessment Checklist:

1. Are all document contents (articles) present and listed on the OWASP project wiki page?
2. Is there an “About This Document” section in the document listing: (NOTE: Leo - Need to create a general template to be used by all docs projects while Ivan Ristic’s proposal is still under elaboration/evaluation)
   1. Document Name
   2. Author(s)
   3. Contributor(s) (NOTE: Leo - How is this going with the sponsorship model (company and names))
   4. Contact email address
   5. Current version and/or release date
   6. Project's web page address (NOTE: Leo - This should exactly match the Wiki project page)
3. Is the document compiled into an exported (PDF) and editable (.Doc) format available on-line?
4. Are all articles that constitute the project properly tagged within project category and available from main project Wiki page? – ‘’’Leo: I believe it’s good to have all project’s contents in the wiki, not only one-piece document.’’’
5. Are the Alpha pre-assessment items complete?

Reviewer Action Items:

1. Does the document consider the OWASP Writing Style?
2. Do contents from wiki articles match compiled document?
3. Does the document have an “About This Document” section which allows the end user to get an overview of the state of the document?

Release Quality Document Criteria

Pre-Assessment Checklist:

1. Have any limitations been documented?
2. Is there a conference style presentation that describes the document in at least 3 slides?
3. Is there a one sheet overview document about the project?
4. Does the document considering OWASP Writing Style and OWASP Template for Docs?
5. Is there a one sheet overview document about the project?
6. Are the Alpha and Beta pre-assessment items complete?

Reviewer Action Items:

1. Does the document substantially address the application security issues it was created to solve?
2. Does the document respecting OWASP Writing Style and OWASP Template for Docs?
3. Have you noted any limitations of the document that are not already documented by the project lead.
4. Have all the Beta Reviewer Action Items been completed? These will need to be completed if they have not already occurred during a previous assessment.