In a recent comment on a TaoSecurity post about quantitative risk assessment I tried to explain the benefits of the quantitative approach:

Whatever inputs are decided on, presumably they reflect the
individual’s opinions accurately. In that case, if the inputs are
garbage, the opinions are garbage as well. Since the qualitative
approach relies on the same opinions, the qualitative approach is also
garbage.

What providing estimates does, at the very least, is
provide a level of scale and magnitude to the discussion, so that your
inputs (opinions) can be reviewed, evaluated, discussed, and MADE
BETTER. Eventually, they can even lead to objective numbers that have
been proven and collected over time (great upside).

What’s more,
if the modeling is done correctly, they are even testable over time.
And here comes the icing: you can more accurately reflect changes in
your opinions over time.

A mathematical model can be no worse
than a subjective, qualitative risk assessment. The process you must go
through itself is enlightening, and if you decide in the end that your
qualitative judgement should take precedence, then at least you’ll know
why.

If there is even one thing that you (or anyone else) think
security professionals have wrong (i.e. conventional wisdom that is
bad), you should be looking for ways to prove it. Quatitative models
provide that opportunity.

I wanted to circle around on the "can’t do any worse" comment. I recently bought the excellent book, Super Crunchers, by Ian Ayres that describes many, many examples where quantitative approaches are changing the face of all sorts of decisionmaking. No, it is never perfect, but it is almost always better. So "almost always" isn’t as absolute as my statement, but it is pretty close.

My backup is to suggest that in any case where the decisionmaker determines that the numbers don’t make sense, then the qualitative backup plan still exists. There is a problem with this, however, because qualitative backup plans are used too frequently (i.e. decisionmakers think the numbers are wrong but they aren’t).

In any case, Ayres’ book points to an important study done by Drs. Paul Meehl and William Grove - Comparative Efficiency of Informal (Subjective, Impressionistic) and Formal (Mechanical, Algorithmic) Prediction Procedures: The Clinical–Statistical Controversy.

Here is the summary:

Given a data set about an individual or group (e.g., interviewer ratings, life history or demographic facts, test results, self-descriptions), there are two modes of data combination for a predictive or diagnostic purpose. The clinical method relies on human judgment that is based on informal contemplation and, sometimes, discussion with others (e.g., case conferences). The mechanical method involves a formal, algorithmic, objective procedure (e.g., equation) to reach the decision. Empirical comparisons of the accuracy of the two methods (136 studies over a wide range of predictands) show that the mechanical method is almost invariably equal to or superior to the clinical method: Common antiactuarial arguments are rebutted, possible causes of widespread resistance to the comparative research are offered, and policy implications of the statistical method’s superiority are discussed.

See also, this paper on the same topic. I consider this paper and the 136 supporting papers to be fairly conclusive in support of the quantitative, objective approach in any area, including information security. There are lots of obstacles to overcome in information security to make quantitative, objective methods a reality (like finding sources of data) but we need first to get past the generic emotional objections of infosec pros that support subjective, qualitative approaches.

People often come up with anecdotal reasons why security is an art and not a science, but every area where these studies probed, they said the same thing. It is really eery reading Super Crunchers knowing that this same debate is going on in information security today.

Note that an initial exploration of this topic was written by Dr. Meehl in 1954. Fifty-three years ago.

I welcome citations of academic research that disproves or points out the weaknesses in this work.