[This whole Microsoft Security Development Lifecycle issue is
really pretty surreal – if someone had told me five years ago that a bunch of
bugfinders would be defending Microsoft while I pointed out inconsistencies
with what they were saying, I would have thought they were crazy.]

There seems to be some confusion about my point regarding SDL so I feel compelled to reiterate my message: I am not
suggesting that SDL doesn’t work – on the contrary, I think it probably has had
a significant effect. I am simply saying that public vulnerability numbers do nothing
to prove it.

Let me see if I can explain my logic a little better and
then come up with some better ideas for Microsoft if they want to prove SDL
works.

As far as I can tell, the overall goal of SDL is to ship a
product with fewer bugs. I believe the most significant effort around SDL was
to train Microsoft employees, so that a) architects design more secure
products; b) developers write better code (fewer bugs); and c) QA testers find
more bugs (with the caveat that this last one may be difficult if the first two
succeed).

The three assertions above – driving higher quality from
each participant – are my interpretation of SDL objectives based on six years
of public reports about the MS initiative. Some have suggested that SDL is a
process and that simply increasing the amount of resources allocated to the
problem, is within the scope of SDL. I believe this is disingenuous.

If all we need are more resources, we don’t need SDL (or
anything with a special name to it). If all we need are more external reviews,
we don’t need SDL. If the truth of the matter is that programmers are all at
the top of their game already, we don’t need SDL. We simply need to work harder
and not smarter.

That is not what I perceive to be the spirit of SDL. And if
it is true, then it simply means that we’ve been cheated all these years and
SDL really was always a marketing exercise. (And a very clever one at that). I
choose to think more highly of Microsoft and believe their intentions were to
really make their people better at what they do. If you disagree, there is no
need to read further.

The major shortcoming of using public vulns as a measure of
SDL efficacy is that there are too many other variables involved. Just because
SDL’s goal is fewer bugs and there are fewer bugs disclosed in public doesn’t
mean that one caused the other. Here are some other candidate reasons that
could have contributed to a reduced public bug count in post-production:

1) Microsoft
simply increased the amount of resources allocated to development;

2) Microsoft
hired/contracted with all the major bugfinders before going GA;

3) External,
uncontrolled resources spent less time and effort on Vista than they did on XP
(due to unpopularity of Vista, more interest in applayer stuff, or simply
happenstance).

4) Vulnerabilities
have gone undercover and don’t get publicly disclosed anymore.

Now, if I were an external analyst (oh, wait…;-)), I might
be forced to consider public vulnerability data as my only source of
information to try to determine causation from SDL. In that case, I would use
the vuln numbers and then try to come up with estimates to address the factors
described above (at a minimum). But not only does MS have access (presumably)
to much better data, but they also don’t see a need whatsoever to control for these
other variables.

This is the core problem: Microsoft is much more confident
in their hypothesis than the evidence dictates. So, not only is the accuracy of
the information in question, but the approach itself uses one data point to
make its case. That turns science into religion and forces a second look at the
entire process. (I pause again to re-assert that I think SDL is probably
working, but the proof laid out is not convincing).

I really don’t expect Microsoft to go through a scientific
exercise to demonstrate the value of SDL. But I certainly feel like I am at a
religious revival when I hear how poorly everyone else is doing and that others
should adopt SDL and how insulting it is for people like me to question their
beliefs. It sounds solely like a marketing and PR exercise to me, and nothing
else.

And Microsoft can do better than that.

[Next Installment: What Microsoft SHOULD do to demonstrate
the value of SDL.]