Remember Whac-A-Mole? The arcade classic where you frantically hammer plastic moles that are popping out of holes? In software security we have a similar version of this game named vulnerability management. It’s a game where you’re not just competing for high scores, you’re battling to keep your company’s data safe and reputation intact.
Except, unlike smiling plastic moles, these vulnerabilities are far from funny. They lurk in the depths of your codebase waiting for an attacker to find them, exploit their weakness, and wreak havoc. And the worst part? The game never ends. New vulnerabilities pop up like clockwork, leaving you scrambling to fix the defects and patch before the next one surfaces.
So, why does this game seem so rigged against product teams and software engineers? Let’s dive into the challenges and uncover the reasons why patching vulnerabilities can feel like a losing battle.
Challenges
1. Vulnerability Overload
The ever-growing complexity of software, coupled with the relentless pace of development, has created a perfect storm of security vulnerabilities. Security researchers, vendors, vulnerability scanning tools, and penetration testers constantly bombard teams with reports of a large volume of flaws. It’s not an exaggeration to say most large companies and software engineering teams are drowning in a sea of security issues.
Let’s quantify this overwhelming reality. Veracode’s 2023 State of Software Security report paints a stark picture with a staggering 85% of all applications contain at least one vulnerability, and 13% harbor critical severity flaws. This translates to thousands vulnerabilities identified annually, each posing a potential entry point for attackers to exploit. The National Institute of Standards and Technology (NIST) National Vulnerability Database (NVD) serves as a sobering testament to this, currently tracking over 200,000 vulnerabilities, with new ones added daily.
The volume of vulnerabilities has profound implications. It stretches security and engineering teams thin, forcing them to prioritize their efforts, often leaving vulnerabilities unaddressed. Resources become scarce, and the pressure to deliver features quickly can overshadow security best practices. This creates a vicious cycle, where vulnerabilities accumulate, the attack surface expands, and the risk of compromise grows often exponentially.
But the problem goes beyond just the numbers. The diversity of vulnerabilities adds another layer of complexity. From the ever-present injection flaws (SQL, XSS) to the growing threat of logic flaws and supply chain attacks, security professionals and software engineers must stay abreast of a constantly evolving landscape. This requires ongoing training, good threat intelligence, and access to the latest security tools and techniques.
The true weight of the problem lies in the potential consequences. Exploited vulnerabilities can lead to data breaches, financial losses, reputational damage, and even physical harm. The 2021 Cost of a Data Breach Report by IBM Security estimates the average cost of a breach at $4.24 million, a figure that underscores the gravity of the situation.
2. Legacy Code & Security Debt
Remember the first time you tried untangling Christmas lights? That’s kind of like patching vulnerabilities in legacy code. Different developers with varying coding styles and security knowledge built systems over years, resulting in a poorly documented tangled mess. They wrote the code at a time when security practices were less established and threats were not as complex. It often lacks fundamental security principles, encryption standards, and secure coding practices that are prevalent today. This inherent insecurity creates a vast attack surface, ripe for exploitation by attackers armed with readily available exploits targeting known vulnerabilities in aging software. Like those tangled Christmas lights, the whole process to unravel, understand, and fix the code is an exercise in frustration that cause regression and compatibility issues that send software engineers screaming.
Further compounding the issue is the problem of security debt. This type of technical debt, accrued through neglecting essential security practices manifests as a backlog of unaddressed vulnerabilities, further expanding the attack surface and amplifying risk. The sheer volume of vulnerabilities within legacy code, coupled with the burden of security debt, can overwhelm even the most well-resourced teams.
3. Balancing Speed and Security
Most software engineering teams must strike a delicate balance. On one hand, fierce competition and volatile markets necessitate rapid delivery of new features to keep users engaged and businesses ahead of the curve. On the other hand, the ever-evolving threat landscape demands robust security to protect data and tech infrastructure from exploitation. This creates a constant tug-of-war with product owners and engineering teams, who grapple with the challenge of reconciling the need for speed with the imperative of security.
The pressure to deliver quickly can lead to several security risks. Tight deadlines and demanding release cycles can incentivize engineers to cut corners, inadvertently introducing vulnerabilities into the code. The allure of new features can also overshadow the importance of following security best practices and addressing security debt. This can lead to vulnerabilities being missed or ignored before deployment.
4. Dependency Hell
Software development has entered an era of rapid iteration and collaboration, this has been accelerated by the proliferation of third-party code and open-source dependencies. While these tools offer many benefits in terms of speed and functionality, they also introduce a complex web of nested security risks and challenges that software engineers must navigate.
Integrating third-party code and open-source dependencies can introduce unexpected security risks. External entities often maintain those dependencies, unlike internally developed code. The inherent trust placed in external parties adds another layer of complexity to the security equation. These external entities may have varying levels of commitment to security and responsiveness. This lack of direct control can be problematic when vulnerabilities arise. Communication with the maintainers may be non-existent and not providing vulnerability fixes fast will leave the software exposed.
A single vulnerability in a seemingly innocuous dependency can serve as a springboard for attackers, granting them access to sensitive data or disrupting critical functionalities. The 2017 Equifax breach, where attackers exploited a vulnerability in the open-source Apache Struts framework, serves as a stark reminder of the potential consequences.
The interconnected nature of modern software development and the prevalent use of open source also increases the risk of sophisticated supply chain attacks. Malicious actors can target popular open-source projects, injecting vulnerabilities into widely used libraries that then ripple downstream, impacting numerous applications.
Teams facing unmaintained or slow-to-patch external dependencies must choose between fixing and maintaining the software themselves, waiting for someone else to create a fix, or replacing the dependency with a secure alternative.
5. The Security and Engineering Disconnect
A security team, siloed from engineering teams, sits atop high horse, shouting through a megaphone at product and IT teams around them: “Don’t you see all these vulnerabilities? Meanwhile, software engineers, buried in code, simply shrug and yell back, “We’ll get to it when we can!” This is the security and engineering disconnect, a communication chasm that can leave security vulnerabilities festering and molding like leftovers you forgot about in the fridge. Without collaboration and shared responsibility, patching stays a game of Whac-A-Mole played by two teams with different perspectives and misaligned goals.
How to Improve Your Game
So, what can we do to start winning this game? Here are some tips on how to take a different approach and stop swinging at so many moles.
1. “Shift Everywhere” Security
We must proactively prevent vulnerabilities from reaching production and strengthen our protection of deployed applications. This means implementing security throughout every phase of the software development lifecycle (SDLC) in planning, development, testing, and ongoing monitoring after deployment. Static code analysis, automated vulnerability scanners, penetration testing, runtime protection tools, and security training for engineers, can all be weapons in our arsenal, helping us identify, prevent, and fix vulnerabilities before they become real threats. For the vulnerabilities that do make it into production implement controls to reduce the likelihood of attackers exploiting them.
For many years the concept of “shift left” security has been quite popular and the phrase dominates the marketing of software security solutions. By detecting and mitigating security problems as early as possible in the SDLC we prevent creating new security issues and slow the rate of vulnerabilities hitting the backlog. The focus should not only be on shifting left, we also need to focus on monitoring and protecting applications after they have been deployed to production.
Don’t stop at “shift left”; “shift right” as well. Better yet think of the SDLC as being circular not linear.
2. Prioritize Based on Risk
Not all vulnerabilities are created equal. With limited resources and an ever-expanding attack surface, prioritizing the remediation of these vulnerabilities becomes crucial for meaningful progress. To effectively address security concerns, we must adopt a systematic approach that prioritizes vulnerabilities based on risk. This ensures that we focus on the most important issues first, maximizing effectiveness.
Some key factors that you should consider in your approach are:
- Asset Criticality – The importance and value of the affected asset or system within the organization’s ecosystem need to be assessed. Vulnerabilities in mission-critical systems or those storing sensitive data naturally warrant higher priority.
- Business Impact – The potential consequences of a successful attack leveraging the vulnerability. This includes the impact on critical systems, sensitive data, financial losses, and reputational damage.
- Vulnerability Severity – Established scoring systems, such as the Common Vulnerability Scoring System (CVSS), provide an initial assessment of vulnerability severity based on factors like exploitability and potential impact.
- Exploitability – The likelihood of a vulnerability being actively exploited is crucial. Factors like the existence of public exploits, attacker motivation, and the ease of exploitation play a significant role
Effective prioritization requires collaboration across different parts of the organization. Security teams need to work closely with software engineering teams to understand the technical context and feasibility of remediation efforts. Additionally, communication with business stakeholders is crucial to ensure alignment with overarching business goals and risk tolerance
Prioritization is an ongoing process, not a one-time event. Continuously review and update as the threat landscape evolves and the organization’s risk profile changes. This ensures the framework reflects the latest threats and vulnerabilities we face.
To help your customers better prioritize based on risk, provide them Vulnerability Disclosure Reports and Vulnerability Exploitability eXchange information. You can read more on that in this blog post.
3. Reserve Capacity
Engineering team leaders need to reserve capacity throughout the SDLC, not just give it lip service. By reserving additional time and resources, engineers can address unexpected security vulnerabilities that may be discovered in testing or after release. This buffer allows for thorough investigation, analysis, and implementation of necessary patches without jeopardizing planned release dates or compromising on quality.
Engineers can work on vulnerability remediation with a sense of calm and focus, knowing they have the time and resources necessary to complete the task effectively. The reserved capacity alleviates some pressure and stress which may also help to improve morale and productivity.
4. Automate, Automate, Automate
Automation can significantly enhance a team’s ability to remediate vulnerabilities. It achieves this by offloading repetitive, time-consuming tasks, allowing security professionals and software engineers to focus on what is more important.
While automation is a powerful tool, it is crucial to remember that it serves as a complement, not a replacement, for human expertise. Security teams should carefully consider the specific needs of their organization and the nature of their vulnerabilities before implementing automation solutions.
5. Bridge the Security and Engineering Disconnect
Let’s tear down the silos and start working together! Effectively addressing security vulnerabilities requires a collaborative effort between security and software engineering teams. This requires moving beyond siloed operations and fostering a culture of shared responsibility for organizational security.
Open and transparent communication is crucial. Security teams should avoid overwhelming engineers with inaccurate or irrelevant findings. Instead, they should explain vulnerabilities in a clear and concise manner, providing engineers with the context and resources necessary to understand the issue and its potential impact.
It’s important for both teams to recognize that absolute security is not the goal. Security teams should acknowledge the inherent trade-offs between speed and security. This creates a space for realistic expectations and fosters trust between the teams.
Instead of only reporting vulnerabilities and asking for status, Security should position themselves as partners, offering expertise and support to engineers throughout the remediation effort. This shift in approach helps build positive relationships and encourages engineers to actively seek security input during the development lifecycle.
Another approach to explore is the use of a Security Champions program. You can read more on that in this blog post.
AI Prompts:
Generative AI prompt used while creating this blog post.
Microsoft Bing Image Generator
"whack-a-mole", smiling plastic moles popping out of holes in an arcade game, 1 adult frustrated male software developer and 1 adult frowning woman software developer frantically swing foam hammers, photo realistic, 4kResources:
- Veracode State of Software Security Report 2023 – https://info.veracode.com/report-state-of-software-security-2023.html
- IBM Security Cost of a Data Breach Report 2021- https://www.ibm.com/security/data-breach
- National Institute of Standards and Technology National Vulnerability Database – https://nvd.nist.gov/vuln
- Wikipedia Article on Whac-A-Mole – https://en.wikipedia.org/wiki/Whac-A-Mole
