Ryan Barnett of ModSecurity has an excellent post on web security metrics. He really nails the fact that it is the outcome that matters:
"While some customers may be distracted by eye-catching graphical
displays of this information, the savvy ones will ask this key question
- Were there any successful attacks? The
answer to this question will tell you the score of the game – did the
opponent score any touchdowns??? All other data is corollary."
His metrics make good sense as well:
- Web transactions per day
- Attacks detected (true positives)
- Missed attacks (false negatives)
- Blocked traffic (false positives)
- Attack detection failure rate
I have some minor quibbles, like I think he should also count the true negatives, which should be the largest portion of the Web transactions (and can be easily derived if the metrics above are collected). Also, I would recast the failure rate in number 5 as a success rate (1 – the failure rate).
The cool thing about this model is that these results can be applied to any inline security mechanism – network firewalls, host intrusion prevention, etc. And with a bit of rejiggering can incorporate authentication and user access control as well.
(It is probably worth noting that these metrics are useful when the Web server is the target, so javascript and other mobile code attacks against the client don't fit well.)
You can’t count false negatives. You might count false negatives that you can detect through alternative means; but the total “false negatives” is unknown. That is why they are called false negatives; because you don’t detect them.
@Vicente -
I agree that you may never count all of them, but I also think you can get most of them through those alternate means. That’s why we have multiple layers of defense. Your best effort is all you can do.
Pete