This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit https://owasp.org

Difference between revisions of "Proposal Project Review QA Approach"

From OWASP
Jump to: navigation, search
(Created page with "=Proposal : Project Reviews Quality Assurance approach= ==Background== OWASP has a Project inventory of more than 150 projects. Many of these projects are in Incubator phase...")
 
 
(14 intermediate revisions by the same user not shown)
Line 4: Line 4:
  
 
OWASP has a Project inventory of more than 150 projects. Many of these projects are in Incubator phase and a selected group of approximate 30 projects are part of Lab or Flagship phase.
 
OWASP has a Project inventory of more than 150 projects. Many of these projects are in Incubator phase and a selected group of approximate 30 projects are part of Lab or Flagship phase.
−
Keeping track and verifying activity level has been a challenging task for the organization. In February 2013 an initiative to set a board of Project Advisors did produce a Project criteria by the end of September 2013, unfortunately, very few volunteers have the time and true commitment to actually do reviews. Another problem is that reviewing these projects at a quality level requires proper understanding of technologies therefore not anyone can actually execute a review, let alone that it is unbiased.
+
Keeping track and verifying activity level has been a challenging task for the organization. In February 2013 a new initiative to set a Board of Project Advisers delivered an assessment methodology using a set of criteria, which was ready by the end of September 2013, unfortunately, very few volunteers have the time and true commitment to actually do reviews. Another problem is that reviewing these projects at a quality level requires proper understanding of technologies therefore not anyone can actually execute a review, let alone that it is unbiased.
  
 
==Proposal approach==
 
==Proposal approach==
  
−
Code and Tool projects are software products produced by the efforts of volunteers. Incubators are a sandbox for innovation and experimentation, but once a project has reached a maturity level such as flagship, the project becomes an integral part of OWASP image. As a flagship project is linked to this view of how the organization operates, OWASP should provide a better platform to help project leaders reaching a higher quality level, but also, help the project leader sustaining the projects at another level such as financially and marketing initiatives. The review quality of a project as it actually is, does not deliver a realistic view of the real status and it is not able to keep proper track of the activity level or quality.
+
Code and Tool projects are software products produced by the efforts of volunteers. Incubators are a sandbox for innovation and experimentation, but once a project has reached a maturity level such as flagship, the project becomes an integral part of OWASP image. The main purpose of this project is to determine the quality of actual flagship projects and set a standard procedure for measuring the quality of projects who want or claim to have "Flagship" status
 +
 
 +
The actual review process does not deliver a realistic view of the real status and it is not able to keep proper track of the activity level or quality.
  
 
==Quantitative & Quality Assurance approach==
 
==Quantitative & Quality Assurance approach==
−
Measuring activity level is a time consuming task, however this can be solved with tools such as ohloh, making possible to automate this part. This works as long as Project leaders configure properly their project repositories in this system. This allow us to collect quantitative data about a project such as
+
Measuring activity level is a time consuming task, therefore it is imperative to automate this process.
 +
With the help of a new tool proposed by Enrico Branca, this could be achieved: http://www.pythonsecurity.org/stats
  
−
- Amount of commits
+
Some of the measurement that will be consider to measure activities are:
−
- Amount of contributors
+
 
−
- Ratings and reviews by the community
+
*Amount of commits
−
- Code data such as Language distribution used in the project
+
*Amount of contributors
−
- Kudos for project leaders
+
*Ratings and reviews by the community
 +
*Code data such as Language distribution used in the project
 +
*Kudos for project leaders
 
   
 
   
 +
We are using Ohloh to gather activity and reviews
 +
[[File:Ohloh.PNG]]
  
 +
==Code Analysis and Continuous Assurance using SWAMP==
 +
The possibilities to integrate this project together with the SWAMP initiative are of great benefit for OWASP. The pilot program will indeed integrate this Continuous Assurance process for long term automated testing and furtehr development of OWASP tools.
 +
 +
More info:
 +
https://www.owasp.org/index.php/SWAMP_OWASP
 +
 +
 +
==Qualitative Approach==
 
In order to verify the quality of projects but also, support flagship projects in this process, this proposal advises to use a Quality Assurance approach. This means that projects that are candidates to become flagship, will be tested at a functional level using known methodologies such as Tmap.
 
In order to verify the quality of projects but also, support flagship projects in this process, this proposal advises to use a Quality Assurance approach. This means that projects that are candidates to become flagship, will be tested at a functional level using known methodologies such as Tmap.
  
 
===Candidate Flagship projects (Tools & Code)===
 
===Candidate Flagship projects (Tools & Code)===
 
A candidate flagship project should fill the following criteria:
 
A candidate flagship project should fill the following criteria:
−
- A solid Health activity  as described in the Project Health Criteria
+
*A solid Health activity  as described in the Project Health Criteria :https://www.owasp.org/index.php/Assessing_Project_Health
−
- A measurable activity in Ohloh
+
Excluding the following:
−
- At least one contributor excluding project leader
+
 
 +
*Industry participation==> Very difficult to measure
 +
*Usability==> Will be replaced by Functional testing
 +
*Creation and quality of training/tutorial material for the project.==> Quality again is also difficult to measure so we should have at least materials as criteria
 +
*At least one contributor excluding project leader
  
  
 
===Focusing on Quality instead of popularity===
 
===Focusing on Quality instead of popularity===
−
Ohloh system might be vulnerable to “popularity” votes instead of quality. In order to be able to measure quality at a functional level, the candidate flagship project will be functionally tested using Tmap as methodology.  For this part , OWASP will hire a team of professional testers guided by a Project Advisor task force, in order to monitor and set the plan.
+
Ohloh system might be vulnerable to “popularity” votes instead of quality. In order to be able to measure quality at a functional level, the candidate flagship project will be functionally tested using Tmap as methodology.  For this part , OWASP will hire one or two testers to work part-time on this project including a group of volunteers to help test.
 +
==PLAN==
 +
This is a rough planning for testing the projects
 +
[[File:PilotProjectQA-approachPlan.pdf]]
 +
 
 +
==Pilot project==
 +
This plan will be executed as a pilot project, using the actual flagship projects, as mentioned before, we can determine if indeed these projects have the necessary production quality to be categorized as "Flagship".
 +
 
 +
Candidate projects to be tested for pilot project:
 +
 
 +
Code
 +
*OWASP AntiSamy Project
 +
*OWASP Enterprise Security API
 +
*OWASP ModSecurity Core Rule Set Project
 +
*OWASP CSRFGuard Project
 +
 
 +
Tools
 +
*OWASP Web Testing Environment Project
 +
*OWASP WebGoat Project
 +
*OWASP Zed Attack Proxy
 +
 
 +
 
 +
==Budget and Planning Pilot project==
 +
 
 +
[[File:PilotProjectQA-approachPlan.pdf]]
 +
 
 +
Resources
 +
* Jira account has been donated by Atlassian (thanks to Norman Yue)
 +
* Ranorex Tool (1) license ==> USD 2,706.4
 +
* 2 Virtual Servers(1 Linux/1 Windows) (Leaseweb) USD 130/server for 6 months period per server/ ==> USD 262,-
 +
* 1 tester @USD25/hour ==> Maximum hours 160==> Total USD 4,000-
 +
 
 +
<b>Total: USD 6968.4,-</b>
  
−
==PLAN==
 
 
===Approach to Tools and Code project===
 
===Approach to Tools and Code project===
−
1st Phase- focus is on Functionality Testing  
+
1st Phase- Focus is on Functional Testing  
 
Assuming we agree on using Tmap as testing methodology and as working methodology Scrum.
 
Assuming we agree on using Tmap as testing methodology and as working methodology Scrum.
 
We will apply a known QA methodology such as Tmap, but we will focus on creating test cases.  
 
We will apply a known QA methodology such as Tmap, but we will focus on creating test cases.  
 
   
 
   
 
Therefore we continue to:
 
Therefore we continue to:
−
Create a test cases for these projects. This plan focuses on creating specific test cases for the main and most important functionality described by each project using  (scrum methodology) in a system like JIRA
+
*Create a test cases for these projects. This plan focuses on creating specific test cases for the main and most important functionality described by each project using  (scrum methodology) in a system like JIRA
−
Focusing on at least 70 to 85% Coverage of all main functionality for each project. Project leaders will have access to the system to feedback during the testing period. All test results are log in JIRA and automated reports will be placed for the community to follow and feedback.
+
*Focus on at least 70 to 85% Coverage of all main functionality for each project. Project leaders will have access to the system to feedback during the testing period. All test results are log in JIRA and automated reports will be placed for the community to follow and feedback.
  
 
===Hiring of Testers===
 
===Hiring of Testers===
−
The test team will work based on a contract or per hour based. Ideally we want to keep working with a team that knows the applications and can easily regress test them anytime we need. Preferably a company and not individuals so we make this sustainable. Volunteers can also join us to test anytime. We can look for university students looking for an internship at OWASP.  
+
The test team will work based on a contract or per hour based. Ideally we want to keep working with a team that knows the applications and can easily regress test them anytime we need. Volunteers can also join us to test anytime. We can look for university students looking for an internship at OWASP.  
  
 
===Testing environment===
 
===Testing environment===
 
Some tools are very specific to a testing environment including hacking/attacking a website, as an example, OWASP ZAP is installed and the functionality test is done against WebGoat in the test environment. A VM containing all the projects installed and ready for testing will be configured.
 
Some tools are very specific to a testing environment including hacking/attacking a website, as an example, OWASP ZAP is installed and the functionality test is done against WebGoat in the test environment. A VM containing all the projects installed and ready for testing will be configured.
  
−
Approach for Reviewing OWASP Code & Tool projects
+
===Approach for Reviewing OWASP Code & Tool projects===
 
It is more complex because it requires the testers to know the programming language. For this part testers must know and understand the code. The main focus will be to follow the instructions as described by the code project and test how well they are integrated as they claim to be.
 
It is more complex because it requires the testers to know the programming language. For this part testers must know and understand the code. The main focus will be to follow the instructions as described by the code project and test how well they are integrated as they claim to be.
 
Then again here, we need to create code reviews, therefore, if we set an testing environment, the main focus of the testers is to test functionality and not into configuring the project, however configuration in case of a code project is also by itself a test. The Project Review Task force will monitor this part.
 
Then again here, we need to create code reviews, therefore, if we set an testing environment, the main focus of the testers is to test functionality and not into configuring the project, however configuration in case of a code project is also by itself a test. The Project Review Task force will monitor this part.
Line 56: Line 106:
 
This is more subjective and that's why I think that we better leave documentation to the Rating system. Hiring a technical writer and editor to provide his opinion could be an option, however, it is subjective.
 
This is more subjective and that's why I think that we better leave documentation to the Rating system. Hiring a technical writer and editor to provide his opinion could be an option, however, it is subjective.
  
−
===Sustainability & Automation==
+
===Sustainability & Automation===
 
The initial work will be to set the entire environment an infrastructure but once is done, is question of maintenance. The idea is that test cases are continuously reused and we can create automated test cases with Selenium (for web apps) and code review tools and use Automated tools such as Ranorex.
 
The initial work will be to set the entire environment an infrastructure but once is done, is question of maintenance. The idea is that test cases are continuously reused and we can create automated test cases with Selenium (for web apps) and code review tools and use Automated tools such as Ranorex.
  
 
==Time==
 
==Time==
 
First it must be approved and decided:
 
First it must be approved and decided:
−
-Testers==> provide a quotation for 2 ideally 3 testers. They must be able to configure tools and code projects.
+
*Testers==> provide a quotation for 2 ideally 3 testers. They must be able to configure tools and code projects.
−
-Buying an account JIRA as platform. So far this is one of the best I have worked with. It's on the cloud and very cheap for low amount of users on the cloud SAAS it costs USD 15/month or https://www.atlassian.com/software/jira/pricing
+
*Buying an account JIRA as platform. So far this is one of the best I have worked with. It's on the cloud and very cheap for low amount of users on the cloud SAAS it costs USD 15/month or https://www.atlassian.com/software/jira/pricing
−
-Automated tools as Ranorex that can automate test for multiple kinds of apps (web or smart) costs EUR 3500,- is one of the best for all type of apps
+
*Automated tools as Ranorex that can automate test for multiple kinds of apps (web or smart) costs EUR 3500,- is one of the best for all type of apps
−
-A bug cloud account for VM machine (dropbox with 100GB will be fine) maybe OWASP already have something like this
+
*A bug cloud account for VM machine (dropbox with 100GB will be fine) maybe OWASP already have something like this
  
 
Once this is decided then this is a rough planning:
 
Once this is decided then this is a rough planning:
−
- Setting up and installations of OWASP projects on VM machine and Setup of Jira accounts==> +/- 5 days
+
*Setting up and installations of OWASP projects on VM machine and Setup of Jira accounts==> +/- 5 days
−
- Create test cases and start testing, including  creation of automated test cases==>3 to 5 days average per project  
+
*Create test cases and start testing, including  creation of automated test cases==>3 to 5 days average per project  
−
- Results are logged into system so is easy to create reports from there and follow up with project leaders and tester team. Reports will be published on OWASP website
+
*Results are logged into system so is easy to create reports from there and follow up with project leaders and tester team. Reports will be published on OWASP website
  
−
==Rating ==
+
==Final Score Rating ==
 
The project should score at least 75% of the QA tests in order to become flagship
 
The project should score at least 75% of the QA tests in order to become flagship
 
These tests will be executed every 6 months period to verify and help the project leader achieve better quality
 
These tests will be executed every 6 months period to verify and help the project leader achieve better quality
  
 
=So I’m a flagship, what’s next=
 
=So I’m a flagship, what’s next=
−
Flagship is a status that OWASP as organization should maintain and support. Its reputation also depends on how well these projects endure and progress. The major objective  is a that a flagship project should get a package that will help them sustain their projects at many levels.
+
Flagship is a status that OWASP as organization should maintain and support. Its reputation also depends on how well these projects endure and progress. The major objective  is a that a flagship project should get a package that will help them sustain at many levels. For example
 +
*Marketing campaigns
 +
*Paying for traveling cost to OWASP conferences
 +
*Getting sponsors
 +
*A budget for side activities

Latest revision as of 11:33, 10 June 2014

Proposal : Project Reviews Quality Assurance approach

Background

OWASP has a Project inventory of more than 150 projects. Many of these projects are in Incubator phase and a selected group of approximate 30 projects are part of Lab or Flagship phase. Keeping track and verifying activity level has been a challenging task for the organization. In February 2013 a new initiative to set a Board of Project Advisers delivered an assessment methodology using a set of criteria, which was ready by the end of September 2013, unfortunately, very few volunteers have the time and true commitment to actually do reviews. Another problem is that reviewing these projects at a quality level requires proper understanding of technologies therefore not anyone can actually execute a review, let alone that it is unbiased.

Proposal approach

Code and Tool projects are software products produced by the efforts of volunteers. Incubators are a sandbox for innovation and experimentation, but once a project has reached a maturity level such as flagship, the project becomes an integral part of OWASP image. The main purpose of this project is to determine the quality of actual flagship projects and set a standard procedure for measuring the quality of projects who want or claim to have "Flagship" status

The actual review process does not deliver a realistic view of the real status and it is not able to keep proper track of the activity level or quality.

Quantitative & Quality Assurance approach

Measuring activity level is a time consuming task, therefore it is imperative to automate this process. With the help of a new tool proposed by Enrico Branca, this could be achieved: http://www.pythonsecurity.org/stats

Some of the measurement that will be consider to measure activities are:

  • Amount of commits
  • Amount of contributors
  • Ratings and reviews by the community
  • Code data such as Language distribution used in the project
  • Kudos for project leaders

We are using Ohloh to gather activity and reviews Ohloh.PNG

Code Analysis and Continuous Assurance using SWAMP

The possibilities to integrate this project together with the SWAMP initiative are of great benefit for OWASP. The pilot program will indeed integrate this Continuous Assurance process for long term automated testing and furtehr development of OWASP tools.

More info: https://www.owasp.org/index.php/SWAMP_OWASP


Qualitative Approach

In order to verify the quality of projects but also, support flagship projects in this process, this proposal advises to use a Quality Assurance approach. This means that projects that are candidates to become flagship, will be tested at a functional level using known methodologies such as Tmap.

Candidate Flagship projects (Tools & Code)

A candidate flagship project should fill the following criteria:

Excluding the following:

  • Industry participation==> Very difficult to measure
  • Usability==> Will be replaced by Functional testing
  • Creation and quality of training/tutorial material for the project.==> Quality again is also difficult to measure so we should have at least materials as criteria
  • At least one contributor excluding project leader


Focusing on Quality instead of popularity

Ohloh system might be vulnerable to “popularity” votes instead of quality. In order to be able to measure quality at a functional level, the candidate flagship project will be functionally tested using Tmap as methodology. For this part , OWASP will hire one or two testers to work part-time on this project including a group of volunteers to help test.

PLAN

This is a rough planning for testing the projects File:PilotProjectQA-approachPlan.pdf

Pilot project

This plan will be executed as a pilot project, using the actual flagship projects, as mentioned before, we can determine if indeed these projects have the necessary production quality to be categorized as "Flagship".

Candidate projects to be tested for pilot project:

Code

  • OWASP AntiSamy Project
  • OWASP Enterprise Security API
  • OWASP ModSecurity Core Rule Set Project
  • OWASP CSRFGuard Project

Tools

  • OWASP Web Testing Environment Project
  • OWASP WebGoat Project
  • OWASP Zed Attack Proxy


Budget and Planning Pilot project

File:PilotProjectQA-approachPlan.pdf

Resources

  • Jira account has been donated by Atlassian (thanks to Norman Yue)
  • Ranorex Tool (1) license ==> USD 2,706.4
  • 2 Virtual Servers(1 Linux/1 Windows) (Leaseweb) USD 130/server for 6 months period per server/ ==> USD 262,-
  • 1 tester @USD25/hour ==> Maximum hours 160==> Total USD 4,000-

Total: USD 6968.4,-

Approach to Tools and Code project

1st Phase- Focus is on Functional Testing Assuming we agree on using Tmap as testing methodology and as working methodology Scrum. We will apply a known QA methodology such as Tmap, but we will focus on creating test cases.

Therefore we continue to:

  • Create a test cases for these projects. This plan focuses on creating specific test cases for the main and most important functionality described by each project using (scrum methodology) in a system like JIRA
  • Focus on at least 70 to 85% Coverage of all main functionality for each project. Project leaders will have access to the system to feedback during the testing period. All test results are log in JIRA and automated reports will be placed for the community to follow and feedback.

Hiring of Testers

The test team will work based on a contract or per hour based. Ideally we want to keep working with a team that knows the applications and can easily regress test them anytime we need. Volunteers can also join us to test anytime. We can look for university students looking for an internship at OWASP.

Testing environment

Some tools are very specific to a testing environment including hacking/attacking a website, as an example, OWASP ZAP is installed and the functionality test is done against WebGoat in the test environment. A VM containing all the projects installed and ready for testing will be configured.

Approach for Reviewing OWASP Code & Tool projects

It is more complex because it requires the testers to know the programming language. For this part testers must know and understand the code. The main focus will be to follow the instructions as described by the code project and test how well they are integrated as they claim to be. Then again here, we need to create code reviews, therefore, if we set an testing environment, the main focus of the testers is to test functionality and not into configuring the project, however configuration in case of a code project is also by itself a test. The Project Review Task force will monitor this part.

Approach for Reviewing Document projects

It is difficult to quantify how well written and accurate or not a document is, unless the reviewer has a broad body of knowledge on the subject. Finding the right reviewer is more challenging. This one will have to spend a time reading and creating a report. An Alternative might be to hire a freelance technical editor/writer that can provide his input from the Documentation and editing point of view and how well written and consistent the document is. This is more subjective and that's why I think that we better leave documentation to the Rating system. Hiring a technical writer and editor to provide his opinion could be an option, however, it is subjective.

Sustainability & Automation

The initial work will be to set the entire environment an infrastructure but once is done, is question of maintenance. The idea is that test cases are continuously reused and we can create automated test cases with Selenium (for web apps) and code review tools and use Automated tools such as Ranorex.

Time

First it must be approved and decided:

  • Testers==> provide a quotation for 2 ideally 3 testers. They must be able to configure tools and code projects.
  • Buying an account JIRA as platform. So far this is one of the best I have worked with. It's on the cloud and very cheap for low amount of users on the cloud SAAS it costs USD 15/month or https://www.atlassian.com/software/jira/pricing
  • Automated tools as Ranorex that can automate test for multiple kinds of apps (web or smart) costs EUR 3500,- is one of the best for all type of apps
  • A bug cloud account for VM machine (dropbox with 100GB will be fine) maybe OWASP already have something like this

Once this is decided then this is a rough planning:

  • Setting up and installations of OWASP projects on VM machine and Setup of Jira accounts==> +/- 5 days
  • Create test cases and start testing, including creation of automated test cases==>3 to 5 days average per project
  • Results are logged into system so is easy to create reports from there and follow up with project leaders and tester team. Reports will be published on OWASP website

Final Score Rating

The project should score at least 75% of the QA tests in order to become flagship These tests will be executed every 6 months period to verify and help the project leader achieve better quality

So I’m a flagship, what’s next

Flagship is a status that OWASP as organization should maintain and support. Its reputation also depends on how well these projects endure and progress. The major objective is a that a flagship project should get a package that will help them sustain at many levels. For example

  • Marketing campaigns
  • Paying for traveling cost to OWASP conferences
  • Getting sponsors
  • A budget for side activities
Retrieved from "https://wiki.owasp.org/index.php?title=Proposal_Project_Review_QA_Approach&oldid=176729"