Difference between revisions of "Limitations of Fully Automated Scanning"

Revision as of 07:08, 17 August 2007

Author: Jeff Williams

I've heard some wild misunderstandings of what is possible with automated tools over the past few days, and so I think this question is really quite important.

You just can't scan anything that's not exposed by the web interface - and even for things that are exposed, you can't scan anything that requires any knowledge of how the application is supposed to behave. Some things that scanners just can't find are:

- business layer access control issues
- internal identity management issues
- lack of a structured security error handling approach
- improper caching and pooling
- failure to log critical events
- logging sensitive information
- fail open security mechanisms
- many unusual forms of injection
- improper temporary storage of sensitive information
- encryption algorithm choice and implementation problems
- hardcoded credentials, keys, and other secrets
- backdoors, timebombs, easter eggs, and other malicious code
- all concurrency issues (race condition, toctou, deadlock, etc...)
- failure to use SSL to connect to backend systems
- lack of proper authentication with backend systems
- lack of access control over connections to backend systems
- lack of proper validation and encoding of data sent to and received from backend systems
- lack of proper error handling surrounding backend connections
- lack of centralization in security mechanisms
- other insane design and implementation patterns for security mechanisms
- code quality issues that lead to security issues (duplication, dead code, modularity, complexity, style)
- improper separation of test and production code
- lots more...

Most of the above list applies to static analysis tools as well, but there are a few differences.

I'm pretty disheartened hearing that people are buying automated tools and building an application security program around them. It's completely backwards to me. First you should figure what security controls are the most important to your organization and then figure out how to verify those cost-effectively.

Now I'm completely in favor of tools that help you verify those things that are really important to your organization. But it just doesn't make sense to limit your security efforts to finding the (relatively small percentage of) vulnerabilities that accidentally happen to be easy to find automatically.

Automated security tools can be very seductive and distracting. They produce an endless stream of work to run and handle findings, and you get to watch the vulnerability numbers decreasing. But it reminds me of the old joke about the guy looking for his keys in a different place from where he lost them because the light is better there.

--Jeff Williams