The Long-Term Impact of Vulnerability Research: Public Welfare

It seems pretty clear to everyone these days that vulnerability discovery and disclosure by “white hats” has a negative impact on risk in the short-term. This is illustrated time and again by the worms and exploits that get published, and most importantly, by the compromised systems that result.

The best argument for discovery/disclosure is its long-term impact on risk. Proponents assert that the short-term loss is worth the pain because the discovery process allows us to identify new attack techniques and strengthens existing software implementations.

Long-term impact is a very difficult assertion to test. Rescorla asserted that in order for this research to be valuable, the number of vulnerabilities in an application’s lifetime should decrease. His test cases showed mostly random incidents. Ozment countered Rescorla with data that showed otherwise, albeit weakly, and adds some information about rediscovery (anecdotal evidence shows about 7-8% overlap in known vulnerability discovery).

Counting known vulnerabilities, however, only tells a part of the story; they are an indicator of the possibility of a compromise, but leave out the undercover vulnerabilities. More importantly, it ignores the compromise itself, and since risk is a function of the frequency and volume of negative outcomes, the compromise is the most important element to evaluating risk.

If evaluating the impact on negative outcomes is a reasonable goal, then we need to understand (at least conceptually) how to evaluate the long-term impact of white hat research in those terms and determine whether public welfare was served. The equilibrium point, therefore, must be between actual negative outcomes (compromises) and foregone negative outcomes along a timeline that incorporates both short- and long- term effects.

In order to provide public welfare, then, White Hat vuln research must eliminate more compromises than it creates. As alluded to earlier, this is slightly different than public welfare simply being created through software with fewer bugs, especially given that there is more software in existence and the rate of bug finding overall does not appear to be diminishing.

Evaluating public welfare directly based on foregone compromises is obviously impossible, since we are defining the future. But we should at least be keeping in mind that future developments assisted by whitehat research must at a minimum overcome all of the costs associated with today’s worms.

[I think there is more here to work on, like the impact of software life and the additional costs of prevention, but I don't have specific conclusions yet.]