Metrics

Outcomes, outputs, and evidence

by  •  • Comments Off

Adam Shostack has an excellent post on the importance of outcomes:

"Another way to put this is if you want to improve something, you have
to start by measuring it. Let’s start measuring security outcomes, so
we can start assessing the processes, errors or hostile acts that lead
to those outcomes."

Couple this with Richard Bejtlich's similarly excellent post on outputs:

"What I am trying to say is that implementing the controls in CAG does not tell you the score of the game. CAG is all about inputs. After implementing CAG you still do not know any outputs.
In other words, you apply controls (an "X"), but what is the outcome
(the "Y"). The controls may or may not be wonderful, but if you are
control-compliant you do not have the information produced by
field-assessed security."

I touched on similar issues in a post on evidence-based security. It is the "unwanted outcome" that drives natural frequencies and ultimately provides data for predicting risk.

In security, we focus too much on implementing controls that operate more frequently with more coverage and yet it isn't clear exactly what we are getting for it. It is common to think we should set our goals to, say, increase patch coverage within 30 days to 95% and be done with it. What people forget is the intent of patching to begin with – to prevent incidents (unwanted outcomes).

Richard talks about inputs and outputs, which I assume is in keeping with his engineering background and the tenets of systems theory.

(Evidence-based medicine is addressing similar issues in the medical world.)