Announcing The Empirical Security Global Model
We just launched a product that I believe is fundamentally different from anything in the market today. A solution that combines the largest collection of real time exploitation activity with years of experience in advanced vulnerability modeling. If you’ve used EPSS before and had a thought that started out with, “I wish that EPSS…” then hopefully this announcement is going to make you excited as well.
But rather than just describing the product and why it’s cool, I need to say something a bit controversial, especially strange since I just announced that we have been focusing on exploitation telemetry available: Known exploitation lists are over-hyped and under-performing. I don’t say that lightly - this thought has been stuck in my head for years, even before the Cybersecurity and Infrastructure Security Agency (CISA) began publishing their Known Exploited Vulnerabilities list in the fall of 2021. I’ve been trying to figure out what to do about it, and I am very excited to share with you what we’ve been up to.
But why are known exploitation lists over-hyped? I’ve been studying and researching vulnerabilities for many years now and a recent study with Cyentia last summer found about 30% of the previously exploited vulnerabilities weren’t exploited month after month. These are false positives: vulnerabilities that may be prioritized but they weren’t actively exploited this month. These are not desirable as they waste valuable time and resources. But false positives are not actually my primary concern nor should they be yours, instead I think the false negatives are much worse and much more concerning for most practitioners.
The Non-Existent False Negatives
If we pick any random vulnerability and ask if it's been exploited in the wild, there are only two valid answers to that: “Yes” and “I don’t know”. Why do people think otherwise? First, it is really easy to under-estimate the incredibly enormous size of the Internet. And second, in the words of Carl Sagan, “Absence of evidence is not evidence of absence.” In support of that, I have two points of reference that make me think we have a lot more absence of evidence than we think:
- As of this morning, in Empirical.Models.Global, there is evidence of over 5,000 vulnerabilities with exploitation activity in the past week. You read that right, while the CISA KEV has just over 1300 vulnerabilities with exploitation activity at some time in the last 3 years, we have found evidence of exploitation for 4 times that just last week. If we compare it to all time exploitation activity, only 5% of the CVEs with exploitation activity are on the KEV List. Which doesn’t mean the KEV List from CISA is bad or invalid in any way, just the opposite as we see in the next point.
- The KEV folks have a high bar to add things to the list so we can have confidence in that exploitation activity being observed one or more times. We are collecting at least daily evidence of exploitation from primary sources and yet: the two sources do not completely overlap. CISA KEV has reported exploitation activity on 5.5% of those we have seen (all time), and we have seen only 68.4% of the vulnerabilities on the CISA KEV. The lack of overlap indicates there are more vulnerabilities being exploited and not being observed or captured for prioritization. To help you understand that, here is a the overlap between the two sources:
Learning from Exploitation (Enter the modeling!)
We’ve been building the models for the Exploit Prediction Scoring System (EPSS) for over 6 years now and we’ve learned quite a few things in that time. The biggest takeaway that we’ve learned is that modeling works and we need a lot of help with prioritization because EPSS has been integrated into well over a 100 different productsLink to https://www.first.org/epss/who_is_using/ and programs across the world.
There are several reasons modeling works:
- It’s built on feedback - it leverages real world exploitation to train the model. It’s not built on what security analysts think may be important or what they feel would be more bad or less bad. It’s built on real world, empirical data.
- It provides feedback - The scores aren’t just dumped into the world for others to figure out the value, we can measureLink to https://www.linkedin.com/posts/jayjacobs1_vulnerabilitymanagement-cybersecurity-infosecurity-activity-7179540602345123840-wH6-?utm_source=share&utm_medium=member_desktop&rcm=ACoAAACqFEQBVUEFy41q_7nMtvMlA7onIUTlBjk how the model is performingLink to https://www.linkedin.com/posts/jayjacobs1_vulncon2024-vulnerabilitymanagement-cybersecurity-activity-7179489961576984576-jE61?utm_source=share&utm_medium=member_desktop&rcm=ACoAAACqFEQBVUEFy41q_7nMtvMlA7onIUTlBjk. So ask yourself if your current process provides feedback on how it’s doing.
- It improves over time - not all modeling improves over time, but our models doLink to https://www.linkedin.com/posts/jayjacobs1_vulncon2024-vulnerabilitymanagement-cybersecurity-activity-7179489961576984576-jE61?utm_source=share&utm_medium=member_desktop&rcm=ACoAAACqFEQBVUEFy41q_7nMtvMlA7onIUTlBjk, because we are measuring performance, evaluating new data sources and improving how we extract information from our existing sources. We can and do help you make better decisions over time
Past exploitation activity can be a powerful indicator of future activity, but exploitation activity is, by its very nature, going to incomplete information. We need to move into the future of learning from exploitation through statistical analysis and modeling. The future of cybersecurity needs to be built on empirical evidence.
