<img src="https://ws.zoominfo.com/pixel/0nVRFDqEc4KEsx6wmKaS" width="1" height="1" style="display: none;">

Can Metrics Help You Evaluate Your Security Program’s Effectiveness?


In today’s fast-paced cybersecurity landscape, it should go without saying that time is of the essence. After all, major industry publications like the Ponemon Institute’s Cost of a Data Breach Report are constantly reminding us that the damages and costs associated with cybersecurity incidents increase with the length of time that the attackers maintain a presence within the victims’ environment. In 2020, data breaches that took longer than 200 days to identify and contain cost an average of $4.87 million, $1.26 million more than those that were identified and contained in less than 200 days.

It’s only logical: the longer that cybercriminals have access to your systems and networks, the more time they have for accomplishing their nefarious deeds — exploring your computing ecosystem, gathering privileged credentials, locating and preparing the most valuable information assets, and removing them stealthily from your databases, all while covering their tracks as they work.

As a result, several industry vendors have hyped the concept of “breakout time,” stressing that defenders may have a few hours at most (or as little as 18 minutes in the worst-case scenario) before threat actors advance from the compromise of a single computer to accessing an entire enterprise network. The implications are obvious: crackerjack security operations teams — those who rank among the best of the best — should be able to detect and respond to attacks almost immediately.

We’ve all heard that what gets measured, gets improved. But does this really hold true for security operations programs? Or, managed detection and response (MDR) and other types of outsourced cybersecurity services? Are there metrics that can tell you — in a way that’s quantified, and thus objective — how well your security team is performing.

Unfortunately, the reality isn’t so simple. While detection and response metrics may be useful as an internal benchmark, it’s difficult if not impossible to define the relevant terms without ambiguity or leaving room for error. So, it’s difficult to make an apples-to-apples comparison across different vendors’ products or services.

Defining the Relevant Metrics

Though there are a few different performance metrics that vendors and service providers may tout, there’s no industry-wide standard. The National Institute of Standards and Technologies (NIST) doesn’t endorse any of these terms, nor do other government agencies, regulatory bodies or independent industry organizations. This means there’s no commonly agreed-upon definition.

With that said, mean time to detect (MTTD) and mean time to respond (MTTR) are the metrics you’ll see most often.

What is MTTD?

Just like it sounds, MTTD indicates the average amount of time it takes your team to detect a security threat or discover an incident.

Though organizations ranging from the SANS Institute to the authors of the Verizon Data Breach Investigations Report do use this term, it is by nature challenging to define. After all, it’s a metric that can be determined only retrospectively. If your environment has been compromised without your knowledge, how will you figure out how long the attackers have been there? What criteria do you use to determine when to start and stop the clock? And how can you promise that you’re figuring out this “incident start time” with complete accuracy?

In some forensic investigations, it’s relatively simple to determine when attackers may have gained a foothold in an environment; in others, it can be near-impossible. When security teams conduct regular threat hunting, they may find previously undetected indicators of compromise that would otherwise have remained undiscovered: if they don’t threat hunt, time to detection might appear to be low, but this won’t accurately reflect detection rates.

What is MTTR?

Though it’s usually taken to denote “mean time to respond,” MTTR can also signify “mean time to recover” or, even, less commonly “mean time to restore” (meaning bring user-facing applications back into service). This ambiguity is at the heart of the problem with the term.

Because there’s no industry-standard benchmark for what “response” or “recovery” means, each vendor, service provider or organization can create their own definition of the term. Restoring systems’ functionality may not mean that the attackers’ foothold in the environment has been completely eradicated. And, even if every trace of the attackers is gone, has the vulnerability through which they gained initial access been patched? Or could they follow the same path to return? Simply closing a ticket doesn’t mean that all problems have been solved.

What to Look at Alongside (or instead of) Metrics

A top-quality cybersecurity service provider will in fact respond to real-world threats and incidents quickly and effectively. But there’s no single number that can definitively prove that they have this ability. Instead of taking MTTD or MTTR as the gold standards, press vendors and providers about the details. How do they define these terms? And why did they choose the definitions that they did?

Comparing quantitative metrics across vendors is difficult if not impossible, but qualitative information can be extremely useful. Does the provider in question have strong testimonials from past and current customers? Are happy clients ready and able to serve as references?

Even more important is a sense of partnership: you’ll know you’re working with a high-quality provider when they provide you with well-researched alerts and actionable suggestions, rather than an endless barrage of false positives. You’ll also discover that the provider is coming alongside to help you improve your organization’s overall security program, rather than simply filling a predetermined role set out in service -level agreements (SLAs).

Want to know more about what it looks like to work with a best-in-class MDR provider? Check out BTB’s Rapid Advanced Detection and Response (RADAR) offering to learn more about our industry-leading platform and expert security monitoring services. Or set up a no-obligation demonstration today.

Contact Us

Related Posts