I think it is worth clarifying one aspect of the paper that is (unintentionally) deceptive. In their attempt to compare availability of exploits and patches over time, the authors use different sampling techniques and populations that are not representative of the whole pool of vulnerabilities. This skews the data and ultimately leads to an incorrect conclusion.

On the surface, it makes sense to use vulnerabilities with known exploit dates as the sample population when analyzing exploit timing. And the authors correctly point out that the number of exploits they use is a minimum number. However, when they use the number as a ratio using the smaller "known-exploit" population in the denominator, and not the overall population of vulnerabilities, it is no longer conservative. In fact, because the ratio over time reaches 100% (inherent to the sample technique), it becomes a maximum ratio.

To illustrate: By my estimate, approximately 40% of the vulnerabilities in the time period being analyzed had known exploits. So conservatively, any distribution function should top off at 40% over time. Instead, the authors discuss numbers reaching "94% 30 days after disclosure."

The fatal flaw here is that the authors assume that there is an exploit for every single vulnerability. This is true for the sample they used, but it is unlikely to be true for the total population of vulnerabilities being assessed. More importantly, it is certainly not a minimum number.

The problem is exacerbated when the authors perform a similar analysis on patch data but select a different sample (top ten vendors) in conducting their analysis. I think it is reasonable to assume that these vendors are either more responsive or less responsive than the overall population of vendors, but I have no reason to believe it is representative.

The final issue here is that after conducting similar analyses on exploits and patches using different, non-random, sample populations, the authors then compare the two results: "We found that expoit availability has consistently exceeded patch availability since 2000." Given the weaknesses in methodology, this is a faulty conclusion.

The work is important and useful, but it needs to be redone using appropriate populations if it is going to be accurate.

I think this is significant to track because it is an indicator of two things:

1. It reminds us that even before vulnerabilities are disclosed they exist in the software, and it is possible that we should be working harder to provide protection.
2. It highlights alternative methods for identifying vulnerabilities other than the discover and disclose cycle that occur between bugfinders and vendors today.

There are a handful of other reasons why I think this tracking is important… but I am stuck wondering if this is useful for the community. It is remarkably difficult (and time consuming) to actually trace the origins of an announcement – it essentially involves taking all reports of "0days" and vendors being aware of attacks in the wild, and playing chicken and egg games to try to come to a conclusion.

Currently, the OSVDB has taken up the charge (thanks, Jericho), but I don't think they can do the full root cause analysis required to really get to a solid determination -My list has 21 undercover vulns since 1988 and theirs has 75 (10 in 2009), though I haven't updated mine since last October (did I mention the time consuming part? ).

While I have not executed an all-out full-court press on vendors, the times I did ask for follow-up to see how the vulnerability was discovered resulted in somewhat ambiguous answers about having "no information" or "disclosure agreements" that prevent any discussion about them.

So, if you think the list is helpful, please let me know. And if you are a conspiracy theorist (I can't help but wonder a bit myself) I would be curious to hear why you think vendors are reluctant to provide this information – personally, I think we should care MORE about these vulns and not less.

1. If finding vulnerabilities makes software more secure, why do we assert that software with the highest vulnerability count is less secure (than, e.g., a competitor)? (It is even more ironic that this has become more questionable an action as "favored" software has fared worse in this category).
2. If disclosure date is the day that software becomes "at risk" why don't we try our hardest to prolong that date?

Conclusion: nobody really knows what the heck they are talking about when it comes to "secure software."

An alternative measure:
The Spire Vulnerability Rating
Why We Need the Spire Vulnerability Rating

Not surprisingly, I have some comments (mostly reiterations) to some of his points, excerpted in italics:

It floors me to think that it is acceptable for
vulnerabilities to be left unpatched for a serious amount of time.

I can understand the emotion behind this comment but not the logic. As it is, there are many vulnerabilities that are left unpatched — all the ones we don't know about yet.

I
consider 90 days to be entirely too long to patch a vulnerability.

This sounds like an arbitrary number of days. It would be great to understand Danny's background with global software development to understand how he can conclude that this is "entirely too long."

You can disagree with full disclosure, but it is a useful
motivational tool. … Limited or closed
disclosure creates complacency, which amounts to willful neglect.

Again, here are conclusions that are unsupported. I can believe that full disclosure is motivational, but I am not sure this brings about more secure software. In addition, I think it would be much more motivational for software companies to have to deal with exploits in the wild. The final point here is a simple opinion with no basis.

This really is closed-minded thinking. With the process as broken as it is already, we need more people thinking outside of the box. There IS another way to motivate vendors – seek out undercover vulnerabilities being exploited in the wild. After all, these exploits are oft-mentioned and yet also ignored. These are the ones I worry about most.

Google has decided to have a bugfinding contest for its Native Client. After its Borg-like introduction of the "Android Security Team" (no names, collective only), it also introduced the contest.

For vendors, this is probably the best option to take to address public vulnerability disclosure. The idea is to increase the value to researchers in order to focus efforts and presumably find most of the security bugs, rather than the traditional alternative of submitting to random public bugfinding.

I wrote a post back in 2005 that mentioned this option:

It will be interesting to see if the vulnerability discovery curve for Android is significantly different from other solutions. (It would also be great to determine rediscovery rates for vulnerabilities).

Corrected [2/27/09]: Fixed per Ryan's clarification in the comments.

The arguments for disclosure first tug at the heartstrings with simplistic platitudes like "it is better to know than not to know" but then grounds itself a bit with the following logic:

1. The bad guys already have this information
2. The good guys need the information to protect themselves

There you have it – a classic fight between good and evil. But even this is incredibly simplistic. It treats two groups as holistic entities and not dynamic populations. And that matters a lot when evaluating risk.

Essentially, folks who support higher levels of quicker disclosure are betting that good guys can and will respond faster and more completely than the bad guys can attack. With discrete groups this may be true, but with dynamic populations I am not so sure.

Risk is a function of threats, vulnerabilities, and consequences. The variance in these elements is constrained by scarce resources on both the attacker side and the defender side.

The attacker makes his decisions based on a cost-benefit analysis that compares costs – skill, effort, and equipment – to the expected benefit discounted by potential penalties (the attacker's risk equation). The higher the result of this equation, the higher the risk to an organization (because threat is higher).

The defender makes a ROSI (return on security investment) assessment (typically ad-hoc) to determine her overall risk. The lower the cost of protection, the more likely that investment is a good one.

Finally, we shouldn't forget opportunity cost which compares these results to anything else the attacker or defender might want to do.

Looking again at the disclosure reasoning, the question is whether releasing more information helps the good guys or the bad guys more. The "bad guys already have this information" argument neglects the acquisition cost of this information and the skill level required to execute.

A basic illustration of cost associated with "effort" – some of you were no doubt a bit annoyed as you read my first paragraph above and wanted to see the source material in question – it didn't have links to the pertinent Sourcefire and Metasploit blogs. Of course, you "already had" this information in the form of your ability to use search engines to find it. Links are a part of the Web culture because we recognize that time is money and making things a bit easier for the reader lowers his/her costs.

So the short point is that distribution of information, and its corresponding ease of access, matters.

The "good guys need this information for protection" is perhaps a trickier. The huge majority of Internet users do not need the information provided because they have no capacity to leverage it for protection. (Guys like HD Moore can do wonders with it, of course). The users rely on the makers of products who DO need this information to provide protection.

It is clear from this case that many large security companies already had the information (they already had samples), so the added benefit to the "good guy" community must be adjusted with that information in mind.

In the end, I think it is less likely that good guys used this information for protection than it is that bad guys used it to compromise some user. I believe this is almost always the case, and my evidence is the aggregated number of exploits that occur after disclosure compared with the number of exploits of undercover vulnerabilities

With many security researchers supporting software liability (or at least some), it is useful to review alternative legal actions that may be taken throughout the discovery and disclosure process. I think the recent RapidSSL hack by Sotirov and Appelbaum is an interesting case.

In case you missed it, they demonstrated a hack where they could essentially spoof a Certificate Authority by taking advantage of known flaws in MD5 and (now exposed) weaknesses in RapidSSL's certificate issuance process (the serial number identification).

This is interesting primarily because it is uncommon. When Adobe and Cisco (and others) attempted legal action against security researchers, the researchers were demonstrating vulnerabilities in software that, if compromised, would impact their customers. This obviously is a concern, but in the end it is very difficult to show damages. That is generally true with most vulnerabilities.

Things change with cases like this (and potentially Software as a Service) where damages are borne by the vendor itself.

This vulnerability comes pretty close to demonstrating business fraud as far as I can tell. I don't know how far they went in their conference session, but if they actually issued a certificate, it seems to me that Verisign /RapidSSL could claim lost revenue (as minimal as it might be at this stage).

I am not sure what to make of their intentional secrecy to protect against legal action. I don't think it was a very smart move to tell people that you purposely didn't tell a vendor because they might have a legal right to keep you from presenting.

In any case, the approach by these folks completely disregards responsible disclosure (as anyone can at this stage) and the whole issue is couched in reasoning that could be applied to any vulnerability discovered — anyone can withhold information from vendors under the assertion that legal action may be taken.

These kinds of competitive rivalries really bring out the worst in security companies and highlight the house of cards that is vulnerability discovery and disclosure. Perhaps more importantly, you'd think ISS would act differently given its experience with the Witty worm and its somewhat strange circumstances… although they may hold the record for the number of vulnerabilities found in competitor products (hmm, maybe I am confusing cause and effect here).

In any case, I doubt it would pass my litmus test. I really don't understand why the profession facilitates arbitrary target practice. Pescatore cuts to the chase with his IBM point, and I am tempted to challenge for ISS to out IBM sometime soon, except that it would increase risk. In any case, IBM would be a target-rich environment in an arbitrary world.

So, if Microsoft is trending consistent with everyone else, then it is more difficult to see the benefit of SDL… This is one of the problems with using public disclosure data – it is inherently fickle and can't tell you nearly as much as, say, internal QA data.

I assume there is a different explanation since I haven't waded through the entire report yet. In any case, it seemed worth mentioning.