This site is the archived OWASP Foundation Wiki and is no longer accepting Account Requests.
To view the new OWASP Foundation website, please visit https://owasp.org
Difference between revisions of "Talk:Testing Guide Introduction"
m |
(→Reviewer Comments 20080829) |
||
| (19 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| + | __TOC__ | ||
| + | |||
| + | == Thoughts == | ||
| + | |||
The main issue I have is that the concept of "testing" proposed by this introduction ("testing is a process of comparing the state of something against a set of criteria") is significantly different from the generally accepted idea of testing. For example, the IEEE glossary defines test "An activity in which a system or component is executed under specified conditions, the results are observed or recorded and an evaluation is made of some aspect of the system or component." (http://www.fda.gov/ora/inspect_ref/igs/gloss.html). Most evidently (and for me, confusingly), this contrasts with the definition of manual inspection and reviews, threat modeling and source code review as testing techniques. | The main issue I have is that the concept of "testing" proposed by this introduction ("testing is a process of comparing the state of something against a set of criteria") is significantly different from the generally accepted idea of testing. For example, the IEEE glossary defines test "An activity in which a system or component is executed under specified conditions, the results are observed or recorded and an evaluation is made of some aspect of the system or component." (http://www.fda.gov/ora/inspect_ref/igs/gloss.html). Most evidently (and for me, confusingly), this contrasts with the definition of manual inspection and reviews, threat modeling and source code review as testing techniques. | ||
However, I do understand that the objective of the guide is to provide a testing framework, rather than a simple testing technique/methodology (even if it seems to me that the text shows here and there the shift from the narrower to the wider meaning). In this context, those techniques could certainly be presented as techniques that aid (focus, guide, etc.) the proper testing activity. | However, I do understand that the objective of the guide is to provide a testing framework, rather than a simple testing technique/methodology (even if it seems to me that the text shows here and there the shift from the narrower to the wider meaning). In this context, those techniques could certainly be presented as techniques that aid (focus, guide, etc.) the proper testing activity. | ||
| Line 26: | Line 30: | ||
"A security unit testing framework might consist on a place holder for security test cases and used to wrap the functions, methods and classes that need to be security tested." [[User:Rick.mitchell|Rick.mitchell]] 11:06, 15 July 2008 (EDT) | "A security unit testing framework might consist on a place holder for security test cases and used to wrap the functions, methods and classes that need to be security tested." [[User:Rick.mitchell|Rick.mitchell]] 11:06, 15 July 2008 (EDT) | ||
| − | == Documentation of my editorial changes == | + | == Documentation of my (editorial) changes == |
Changes: | Changes: | ||
* The intro paragraphs now have the following structure: what is this guide, what were the challenges, the result is good, roadmap for the rest of the guide | * The intro paragraphs now have the following structure: what is this guide, what were the challenges, the result is good, roadmap for the rest of the guide | ||
* The next paragraphs titles have been changed to match the programmatic intent of explaining the what, why, when, where, and how of security testing | * The next paragraphs titles have been changed to match the programmatic intent of explaining the what, why, when, where, and how of security testing | ||
| + | * I've reworded the paragraph on economics so that (hopefully) it's more clear. The gist now is: you have to measure both the technical aspects and the economics one. I've added some data from the NIST report, which otherwise was a rather unclear reference. | ||
| + | Questions: | ||
| + | * Is there a format for citations/references? | ||
| + | * Threat modeling in Testing Techniques Explained refers to a detailed threat modeling methodology in part 2. Does this really exist? | ||
| + | * What is a good reference for STRIDE? http://msdn.microsoft.com/en-us/magazine/cc163519.aspx ? | ||
| + | * The part on source code analysis tools (A Note about Static Source Code Review Tools) is not clear. What is the code? What is the undefined variable temp? This part should probably be substantially improved or go. | ||
| + | * Before the section on Security requirements, there is a link to the TOC. Is that supposed to be there? | ||
| + | * What is "application security frame categorization"? | ||
| + | * Not clear: "From the defect management and reporting perspective, software quality and security testing can leverage similar issue categorizations and metrics." | ||
| − | + | Comments: | |
| + | * There's a big disconnect between Section 3 and Section 4, both in terms of the language and in the level of details used. Also, it probably would be good to have a paragraph to introduce Section 4, so that it is clear why it's there. For example, something along the lines of "if you want to have a successful testing program, you need to know what are the objectives of the testing. These objectives are specified by security requirements. This section discusses..." | ||
| + | * Some parts of the last section significantly overlap with the content of the next chapter | ||
| − | + | Open: | |
| − | * | + | * The part on automated tools being bad at finding vulnerabilities should probably be clarified (Mindset paragraph in principles). See also https://lists.owasp.org/pipermail/owasp-testing/2008-July/001586.html |
| + | |||
| + | TODO: | ||
| + | * The paragraph on penetration testing could reference http://www.ranum.com/security/computer_security/editorials/point-counterpoint/pentesting.html and http://www.schneier.com/blog/archives/2007/05/is_penetration.html | ||
| + | |||
| + | |||
| + | [[User:Marco|Marco]] 21:28, 29 July 2008 (EDT) | ||
| + | |||
| + | == Reviewer Comments 20080829 == | ||
| + | Within the "Why Test" section.<br> | ||
| + | "This document does not go into the technical details of how to test an application, as the intent is to provide a typical security organizational framework. The technical details about how to test an application, as part of a penetration test or code review will be covered in the remaining parts of this document. "<br> | ||
| + | Seems contradictory.<br> | ||
| + | |||
| + | Within the "When to Test" section. The "maintain" slice of the pie chart is hard to read, can the viewing angle of the pie be changed so that it appears the reads is looking down a bit more from the top (not necessarily straight down, but just angled slightly more). | ||
| − | [[User: | + | Within "A Note about Static Source Code Review Tools" some exemplar code is provided but not discussed. This really leaves the reader hanging, especially if they're unfamiliar with C/C++ code.<br> |
| + | [[User:Rick.mitchell|Rick.mitchell]] 10:20, 29 August 2008 (EDT) | ||
Latest revision as of 15:02, 29 August 2008
Thoughts
The main issue I have is that the concept of "testing" proposed by this introduction ("testing is a process of comparing the state of something against a set of criteria") is significantly different from the generally accepted idea of testing. For example, the IEEE glossary defines test "An activity in which a system or component is executed under specified conditions, the results are observed or recorded and an evaluation is made of some aspect of the system or component." (http://www.fda.gov/ora/inspect_ref/igs/gloss.html). Most evidently (and for me, confusingly), this contrasts with the definition of manual inspection and reviews, threat modeling and source code review as testing techniques. However, I do understand that the objective of the guide is to provide a testing framework, rather than a simple testing technique/methodology (even if it seems to me that the text shows here and there the shift from the narrower to the wider meaning). In this context, those techniques could certainly be presented as techniques that aid (focus, guide, etc.) the proper testing activity. To make a long story short, I guess this boils down to renaming "testing techniques explained" to something like "techniques of the testing framework explained" and reframing the presentation of those techniques.
Second, even if it is hinted somewhere, it is never clearly stated that, in general, testing cannot find all vulnerabilities in an application. I would add a paragraph on this in the principles of testing section.
Third, the final section of source code review tools doesn't seem to be on a par with other parts of the guide: it promises to highlight fundamental issues but only briefly mentions the problem of false positives (what about false negatives?) and presents an (unexplained) example in C/C++ (which is not exactly the first language one would think of when discussing source code review tool for web applications). I would either expand it or move it as a paragraph of the section of "need of balanced approach", or merge it with the section titled "a note about web application scanner" to make it a more general "note about automated tools".
More down-to-the-text comments:
- Is there any guideline for references: sometimes they are used inline, sometimes they presented at the end of the paragraph.
- The first paragraph is somewhat disorganized. I would restructure it to say (in this order):
- objectives (first sentence, the line starting with "This framework aims", and the discussion on the economics of insecure software)
- two challenges (from "Writing the testing project has proven" to "integrated in the software development lifecycle")
- The guide is good: "However, OWASP is able to take" to the end of the paragraph and "Many industry experts..."
- Comments: the paragraph before "Principles of testing"
- I would rename some titles:
- Scope of this Document -> Why having a testing program?
- The software development life cycle process -> When to apply testing?
- The scope of what to test -> What to test?
Marco 04:58, 9 February 2007 (EST)
v3 Review Comments
Under "Developers' Security Tests" the following sentence doesn't read clearly but I'm not 100% sure what the author was trying to say so I haven't edited it: "A security unit testing framework might consist on a place holder for security test cases and used to wrap the functions, methods and classes that need to be security tested." Rick.mitchell 11:06, 15 July 2008 (EDT)
Documentation of my (editorial) changes
Changes:
- The intro paragraphs now have the following structure: what is this guide, what were the challenges, the result is good, roadmap for the rest of the guide
- The next paragraphs titles have been changed to match the programmatic intent of explaining the what, why, when, where, and how of security testing
- I've reworded the paragraph on economics so that (hopefully) it's more clear. The gist now is: you have to measure both the technical aspects and the economics one. I've added some data from the NIST report, which otherwise was a rather unclear reference.
Questions:
- Is there a format for citations/references?
- Threat modeling in Testing Techniques Explained refers to a detailed threat modeling methodology in part 2. Does this really exist?
- What is a good reference for STRIDE? http://msdn.microsoft.com/en-us/magazine/cc163519.aspx ?
- The part on source code analysis tools (A Note about Static Source Code Review Tools) is not clear. What is the code? What is the undefined variable temp? This part should probably be substantially improved or go.
- Before the section on Security requirements, there is a link to the TOC. Is that supposed to be there?
- What is "application security frame categorization"?
- Not clear: "From the defect management and reporting perspective, software quality and security testing can leverage similar issue categorizations and metrics."
Comments:
- There's a big disconnect between Section 3 and Section 4, both in terms of the language and in the level of details used. Also, it probably would be good to have a paragraph to introduce Section 4, so that it is clear why it's there. For example, something along the lines of "if you want to have a successful testing program, you need to know what are the objectives of the testing. These objectives are specified by security requirements. This section discusses..."
- Some parts of the last section significantly overlap with the content of the next chapter
Open:
- The part on automated tools being bad at finding vulnerabilities should probably be clarified (Mindset paragraph in principles). See also https://lists.owasp.org/pipermail/owasp-testing/2008-July/001586.html
TODO:
- The paragraph on penetration testing could reference http://www.ranum.com/security/computer_security/editorials/point-counterpoint/pentesting.html and http://www.schneier.com/blog/archives/2007/05/is_penetration.html
Marco 21:28, 29 July 2008 (EDT)
Reviewer Comments 20080829
Within the "Why Test" section.
"This document does not go into the technical details of how to test an application, as the intent is to provide a typical security organizational framework. The technical details about how to test an application, as part of a penetration test or code review will be covered in the remaining parts of this document. "
Seems contradictory.
Within the "When to Test" section. The "maintain" slice of the pie chart is hard to read, can the viewing angle of the pie be changed so that it appears the reads is looking down a bit more from the top (not necessarily straight down, but just angled slightly more).
Within "A Note about Static Source Code Review Tools" some exemplar code is provided but not discussed. This really leaves the reader hanging, especially if they're unfamiliar with C/C++ code.
Rick.mitchell 10:20, 29 August 2008 (EDT)
