Lots of talk about security  metrics in the blogosphere. Since I just got done "breaking all the (unofficial) rules" about security metrics in my one day metrics class, I feel compelled to comment.

Mike Rothman at Security Incite re-initiated the (security part) of the conversation by pointing to "Joel on Software" and suggesting that metrics are bad because you can game them. I think he can be swayed to the pro-metrics side of the house, but only time will tell. He (and Joel) is/are right that metrics can be gamed – my biggest "metric" pet peeve is when my kids’ teachers all spend the bulk of their time "teaching" kids how to take their aptitude tests so they’ll score higher…arrgh! However, the whole "metrics are bad because they can be gamed" argument is a pretty weak one when you consider the state of affairs without metrics – talk about being able to "game" the system – heck, there is no system, so you can do whatever you want. Now that’s real gaming.

Amrit Williams responds to Mike’s post with some pro-metrics fare (since I agree with him, no comments necessary). Then Rich Mogull (and his editor) responds twice to Amrit’s post and a comment (man, you guys should talk more in the office ) with the "economist" approach (you know "on the one hand…but on the other hand") which is more reasonable but also gets us no further along. I do have a quibble or two about what he said:

1. First, ROI is not a metric per se, but the calculations used get you closer to creating one. (I told my security metrics class this today…then proceeded to teach a section on ROI).

2. ROI is about efficiency, not effectiveness. ROI is about (in the case of security) cost savings; it doesn’t provide any data on (e.g.) risk reduction.

3. There is no reason you can’t collect real numbers for the half of ROI that incorporates existing data – in fact, if the real numbers don’t exist, then ROI isn’t the right measure. The alternative strategy being assessed is the one that requires decent estimates, and those aren’t that hard either.

More on security ROI.

3. ROSI (which also isn’t a metric) is the effectiveness metric. ROSI appears to be Rich’s true target, because it involves estimating potential losses and reduction in risk to get to hard dollars. I agree that it is hard, but I don’t think it is impossible. The real key to any estimates are that the folks involved (i.e. key management executives) buy into the assumptions. There is very little "absolute truth" here (particularly with potential loss) so estimates are okay.

More on ROSI.

4. While I agree that estimates can be dangerous, it is not clear to me why they create more of a problem than qualitative judgements.

Now, back to metrics. The problem with all of this discourse is simply that there are still no metrics to speak of. So, I will provide at least some structure around real metrics that can be used for various purposes. There are a handful of "standard" purpose metrics: cycle time, process efficiency, productivity, and cost effectiveness are some good ones (this APQC article provides a great summary). In the security space, we can tailor these to create metrics for risk and controls.

When you realize that there are many types of metrics for many purposes, it becomes apparent why a "top ten" list really won’t work – because people are often talking about different things when they are talking about metrics. But I won’t let that stop me.

There really are only a finite number of things you can count for metrics. Here are some basics:

For security admin processes (e.g. patching, password resets, user account management):

• Total units of work (e.g. total patches)
• Errors
• Time to complete (single unit)
• Time period
• FTEs
• Cost

I didn’t even use the word "risk" because it really is only ancillary to security admin functions.

For inline security mechanisms (firewalls, antivirus, authentication, etc), we employ an approach that provides sensitivity and specificity feedback as the mechanism "tests" the event along with the metrics from above:

• Total Events
• Total Events addressed
• Legitimate allows (true positive)
• Legitimate denies (true negative)
• False positives
• False negatives
• Time period
• Cost

If you are collecting this information, then you can work into prevalence/incidence numbers that (I believe) can help you quantify risk.

In order to use this data, it is helpful to normalize it. You may just normalize by standardizing on a time increment (this works for internal trending); you may use some set of resources (i.e. laptops or applications, etc.); or you may use volume of activities.

There is no magic here, but there is a lot of work. And all with the potential for real numbers.

I’ll have more some other time.

[p.s. Amrit, Rich - send me an email at petelind@spiresecurity.com if you'd like to see one of my all-time favorite use of swags.]