Talk:Testing Guide Introduction

The main issue I have is that the concept of "testing" proposed by this introduction ("testing is a process of comparing the state of something against a set of criteria") is significantly different from the generally accepted idea of testing. For example, the IEEE glossary defines test "An activity in which a system or component is executed under specified conditions, the results are observed or recorded and an evaluation is made of some aspect of the system or component." (http://www.fda.gov/ora/inspect_ref/igs/gloss.html). Most evidently (and for me, confusingly), this contrasts with the definition of manual inspection and reviews, threat modeling and source code review as testing techniques. However, I do understand that the objective of the guide is to provide a testing framework, rather than a simple testing technique/methodology (even if it seems to me that the text shows here and there the shift from the narrower to the wider meaning). In this context, those techniques could certainly be presented as techniques that aid (focus, guide, etc.) the proper testing activity. To make a long story short, I guess this boils down to renaming "testing techniques explained" to something like "techniques of the testing framework explained" and reframing the presentation of those techniques.

Second, even if it is hinted somewhere, it is never clearly stated that, in general, testing cannot find all vulnerabilities in an application. I would add a paragraph on this in the principles of testing section.

Third, the final section of source code review tools doesn't seem to be on a par with other parts of the guide: it promises to highlight fundamental issues but only briefly mentions the problem of false positives (what about false negatives?) and presents an (unexplained) example in C/C++ (which is not exactly the first language one would think of when discussing source code review tool for web applications). I would either expand it or move it as a paragraph of the section of "need of balanced approach", or merge it with the section titled "a note about web application scanner" to make it a more general "note about automated tools".

More down-to-the-text comments:

• Is there any guideline for references: sometimes they are used inline, sometimes they presented at the end of the paragraph.
• The first paragraph is somewhat disorganized. I would restructure it to say (in this order):
  • objectives (first sentence, the line starting with "This framework aims", and the discussion on the economics of insecure software)
  • two challenges (from "Writing the testing project has proven" to "integrated in the software development lifecycle")
  • The guide is good: "However, OWASP is able to take" to the end of the paragraph and "Many industry experts..."
  • Comments: the paragraph before "Principles of testing"
• I would rename some titles:
  • Scope of this Document -> Why having a testing program?
  • The software development life cycle process -> When to apply testing?
  • The scope of what to test -> What to test?

Marco 04:58, 9 February 2007 (EST)

v3 Review Comments

Under "Developers' Security Tests" the following sentence doesn't read clearly but I'm not 100% sure what the author was trying to say so I haven't edited it: "A security unit testing framework might consist on a place holder for security test cases and used to wrap the functions, methods and classes that need to be security tested." Rick.mitchell 11:06, 15 July 2008 (EDT)

Documentation of my (editorial) changes

Changes:

• The intro paragraphs now have the following structure: what is this guide, what were the challenges, the result is good, roadmap for the rest of the guide
• The next paragraphs titles have been changed to match the programmatic intent of explaining the what, why, when, where, and how of security testing
• I've reworded the paragraph on economics so that (hopefully) it's more clear. The gist now is: you have to measure both the technical aspects and the economics one. I've added some data from the NIST report, which otherwise was a rather unclear reference.

Questions:

• Is there a format for citations/references?
• Threat modeling in Testing Techniques Explained refers to a detailed threat modeling methodology in part 2. Does this really exist?

Open:

• The part on automated tools being bad at finding vulnerabilities should probably be clarified (Mindset paragraph in principles)
• The part on source code analysis tools (A Note about Static Source Code Review Tools) is not clear. What is the code? What is the undefined variable temp?

TODO:

• The paragraph on penetration testing could reference http://www.ranum.com/security/computer_security/editorials/point-counterpoint/pentesting.html and http://www.schneier.com/blog/archives/2007/05/is_penetration.html

Marco 05:16, 17 July 2008 (EDT)