CISO AppSec Guide: Metrics For Managing Risks & Application Security Investments

< Back to the Application Security Guide For CISOs

Part IV: Metrics For Managing Risks & Application Security Investments

IV-1 Executive Summary

CISOs need metrics to report to Senior Management the effectiveness of the application security program investment and its impact on business risk. CISOs also need metrics to manage and monitor the people, processes, and technologies that make up the application security program.

These metrics are made up of three categories. CISOs should be able to answer these questions based on the metrics and push her team to provide these in a near real-time basis through automated means. Critical questions include:

• Application security process metrics - How well is the organization meeting security policies, technical standards, and industry practices? How consistently are we executing security SLAs? By application? By division? By channel?
• Application security risk metrics
  • Vulnerability risk management metrics - What is the Mean Time to Repair on an annual basis? On a monthly basis? By application? By division? What are the known security issues in production?
  • Security incident metrics - What security issues have been exploited? Were they known issues that were released in production? What was the cost to the business?
  • Threat intelligence reporting and attack monitoring metrics - Which applications are receiving more attacks than others? Which applications have upcoming expected peak usage?
• Security in the SDLC metrics
  • Metrics for risk mitigation decisions - What is the Mean Time to Repair by an application's risk category? Does it meet expectations? What is the risk heat map by application? By division? By channel?
  • Metrics for vulnerability root causes identification - What are the root causes of vulnerabilities for each application? Is there a systemic issue? Which security practices have been best adopted by each development team? Which development teams need more attention?
  • Metrics for software security investments - Which SDLC phase have identified the most security issues? What is the maturity of the corresponding security practices in each SDLC phase? What is the urgency for more security people, process, and technologies in each SDLC phase?

Note OWASP provides several projects and guidance for CISOs to help measuring and monitor security and risks of application assets within the organization develop and implement an application security program. Besides reading this section of the guide, please consult the Appendix B: Quick Reference to OWASP Guides & Projects for more information on OWASP projects within the application risk management metrics & monitoring domain.

IV-2 Introduction

The aim of this part of the guide is to help CISOs manage the many aspects of an application security program – specifically risk and compliance, as well as application security resources such as processes, people and tools. One of the goals of application security metrics is to measure application security risks as well as compliance with application security requirements mandated by information security legislation, regulation and standards. Among critical application security processes that CISOs need to report on and manage are development processes and operational aspects such as application vulnerability management. It is often CISOs' responsibility to report the status of application security activities to senior management such as, for example, the status of application security testing and software security activities in the SDLC.

From a risk management perspective, it is important that application security metrics include reports on technical risks such as un-mitigated vulnerabilities for applications that are developed and managed by the organization. Another important aspect of the these metrics is to measure coverage, such as the percentage of the application portfolio regularly assessed in the application security verification program, the percentage of internal apps vs. external apps covered, the inherent risk of these applications and the type of security assessments performed, and when in the SDLC they are performed. These types of metric help CISOs in reporting on application security process compliance and application security risks to the head of the information security as well as to the application business owners.

Since one of CISOs' responsibilities is to manage both information security and application security risks and to make decisions on how to mitigate them, it is important for these metrics to be able to measure these risks in terms of risk exposure to the organization’s assets that include application data and functions.

IV-3 Application Security Process Metrics

Metrics and measurements goals

The goal of the application security process metrics is to determine how well the organization's application security processes meet the security requirements defined by application security policies and technical standards. For example, an application vulnerability process might include requirements to execute vulnerability assessments on internet facing applications every six or twelve months depending on the inherent risk rating of the application. Another vulnerability process requirement would be to execute security in multiple SDLC processes such as architecture risk analysis/threat modeling, static source code analysis/secure code reviews and risk based security testing on applications that store individuals' confidential information and whose business functionality is a critical service to citizens, clients, customers, employees, etc.

From the perspective of process coverage, one of the goals of these metrics might be to report on the coverage of application security process such as to measure how applications fall in scope for application security assessments to identify potential vulnerability assessment gaps based upon application type and the application security requirements. These help CISOs provide visibility on process coverage as well as the status of the operational execution of the application security programs. For example the metrics might show (e.g. in red status) that some of the application security processes in the SDLC such as secure code reviews are being not executed for a number of high risk rated applications and flag this as an out of compliance issue with the security testing requirements. This type of metric allows the CISO to prioritize resources by allocating them on where is most needed to comply with the standard process requirements.

Another important measurement for application security verification is to measure the time when the application security processes are scheduled versus when they are actually executed, to identify potential delays in security code review, static source code analysis, ethical hacking and application penetration testing processes.

IV-4 Application Security Risk Metrics

Vulnerability risk management metrics

CISOs' responsibilities include the management of application security risks. From a technical risk perspective, application security risks might be due to vulnerabilities in the applications that expose the application assets such as data and critical functions to potential attacks that seek to compromise these. Typically, technical risk management comprises mitigating the risks posed by vulnerabilities by applying fixes and countermeasures. The mitigation of the risk of these vulnerabilities is typically prioritized based upon the qualitative measurement of risks. For example, for each application that is developed and managed by the organization there would be a certain number of vulnerabilities identified at ranked by severity (e.g. high, medium and low). The greater the number of high and medium risk vulnerabilities, the higher is the risk to the application. The higher the value of the data assets protected by the application and the criticality of the functions supported, the higher the impact of these vulnerabilities on the application assets.

One important data point in vulnerability metrics is the number of vulnerabilities that remain unfixed. A given number of application vulnerabilities might still be “open”, that is not yet fixed in the production environment, and these represent a risk to the organization and require the CISO to prioritize the risk mitigating actions such as “closing” the vulnerability within the compliance timeframes that is deemed acceptable by the application vulnerability management standards.

Security incident metrics

Another important metrics for CISOs managing information security risks is the reporting of security incidents relating to applications that are developed and/or managed by the organization. The CISO might gather this data reported from SIRT (Security Incident Response Team Incidents) that affect a given application due to the exploitation of a vulnerability. The correlation of the security incidents reported for a given application with the vulnerabilities reported by security testing allows the CISO to prioritize the risk mitigation effort on mitigating vulnerabilities that might cause the most impact to the organization. Obviously, waiting for a security incident to occur to decide which vulnerabilities to mitigate is symptomatic of a reactive rather than proactive approach toward risk management, but feedback is important.

Threat intelligence reporting and attack monitoring metrics

Risk proactive organizations do not wait for security incidents to occur but instead learn from other attacks, published vulnerabilities and threat intelligence, using the information to adjust severity and risk assessments, take proactive risk mitigation measures such as to develop and implement countermeasures, or prioritizing the mitigation of known higher risk known vulnerabilities that might potentially be exploited in an incident to cause the most impact to the organization. The CISO can use threat intelligence reports as well as metrics from monitored application layer security events such as from SIEM (Security Incident Event Management) systems and honeypot applications to assess the level of risk. Unfortunately today, most of security incidents are discovered and reported only months after the initial intrusion or data compromise. Security metrics that are actionable and targeted towards preventing risks of attacks is of critical importance for CISOs since it might facilitate deciding which applications to put under stricter monitoring and alerting in able to be able act more quickly in the case of an attack, reducing the impact of an event. For example,:

• A threat alert of a possible distributed denial of service against online banking applications might allow the CISO to put the organization on alert and prepare to roll out prepared countermeasures to prevent a subsequent outage
• A reported threat of malware targeting e-commerce applications to steal user credentials and conduct un-authorized purchases allows the CISO to issue monitoring alerts for the incident event monitoring management team.
• A published vulnerability in a software framework or library may alter the CISO's scheduling of patch testing, deployment and verification.

IV-5 Security in SDLC Management Metrics

Metrics for risk mitigation decisions

Once vulnerabilities are identified at any stage of development and operation, the next step is to decide which should be fixed, when and how. The first question can be answered by the vulnerability assessment process compliance requirements that might state for example that high risk vulnerabilities are remediated in shorter time frames than medium and low risk vulnerabilities. The requirement might also vary depending on the type of application, being for example a totally newly developed application versus a new release of an existing application. As new applications have not been security tested previously, they represent a higher risk than existing applications and therefore this might require that high risk vulnerabilities to be mitigated prior to release of the application into the production. Once the issues are identified and prioritized for mitigation based upon the risk severity of the vulnerability, the next step is to determine how to fix the vulnerability. This depends on factors such as the type of the vulnerability, the security controls/measures that are affected by the vulnerability, and where the vulnerability is most likely being introduced. This type of metric allows the CISO to identify the root causes of vulnerabilities and present the case for remediation to the application development teams.

Metrics for vulnerability root causes identification

When the vulnerability metrics are reported as a trend, it allows CISOs to assess improvement. For example, in the case of a single issue measured over time for the same type of application, it is possible for the CISO to point to potential root causes. With trend vulnerability metrics and categorization of the type of vulnerabilities, it is possible for CISOs to make the business case for investing in certain types of security activities such as process improvements, adoption of testing tools as well as training and awareness. For example, the metrics showed in figure 1 shows positive trends of certain type of vulnerabilities by comparing two quarterly releases of the same applications. Application security improvements measured as a reduced number of vulnerabilities identified from one quarterly release to another is observed for most vulnerability types except for authentication and user/session management issues.

CISOs might use these metrics to discuss with CIOs and development directors on whether the organization is getting better or worse over time in releasing more secure application software, and to direct the application security resources (e.g. process, people and tools) to where they are most needed for reducing risks. With the metrics shown in Figure 1 for example, assuming the application changes introduced between releases do not differ much in term of type and complexity, as well as the number and the type of software developers in the development team, and the tools used, a case can be made on focusing on the type of vulnerabilities that the organization is having trouble fixing such as better design and implementation of authentication and user/session management controls. The CISO might then coordinate with the CIO and the development directors to schedule a targeted training on these types of vulnerabilities, document development guides for authentication and session management and adopt specific security test cases. Ultimately this coordinated effort will empower software developers in designing, implementing and testing more secure authentication and session management controls and show these as improvements in the vulnerability metrics.

Metrics for software security investments

Another important aspect of the S-SDLC security metrics is to decide where in the SDLC to invest in security testing and remediation. To know this, it is important to measure in which phase of the SDLC the most of vulnerabilities (higher percentage of issues) originate, when these vulnerabilities are tested and how much it cost the organization to fix them in each phase of the SDLC. A sample metric that measure this is shown in the figure below based upon a case study on the costs of testing and managing software bugs (Ref Capers Jones Study).

These metrics can also be used to assist the proper allocation of security investment between application and infrastructure.

A similar type of security defect management metrics can be used by CISOs for managing security issues effectively by reducing overall security costs. Assuming the CISO has rolled out security throughout all SDLC process and has budget allocated for investment in security in the SDLC activities such as secure coding training and secure code review process and static code analysis tools, these metrics allows the CISO to make that case for investing in testing and fixing security issues in the early phases of the SDLC. This is based upon the following measurements from this case study:

• Most of the vulnerabilities are introduced by software developers during coding,
• The majority of these vulnerabilities are tested during field tests prior to production, and
• Testing and fixing vulnerabilities late in the SDLC is the most inefficient way to do it since is approx. ten times more expensive to fix issues during pre-production tests than during unit tests.

CISOs can use vulnerability case studies like these or use their own metrics to make the case for investing in secure software development activities since these will save the organization time and money.