Andy Jaquith was nice enough to send me to camp. But it wasn’t like the basketball camps that I went to in high school, it was a "camp" cleverly disguised as a "side" in some arena. This particular arena has modelers in one camp and measurers in another. (And two other incidental groups).
As Alex Hutton (who is in my "camp") points out – it is not clear that the "model vs. measure" characterization is anything more than a false dichotomy – the two need each other. I know this even more personally because I cringe at the characterization and am compelled to write this post simply to counter (and in some ways clarify) the parts that miss the point. In fact, if I had to pick a "side" (you can’t really) I would want to be considered a "measurer" and NOT a "modeler".
The big problem with measuring is simply that there isn’t enough data to go around, so I find myself doing a lot of modeling – here is a good one: risk{threat, vulnerabilities, consequences}. Yes, you must see that I am brilliant to come up with such a fancy algorithm for risk that nobody else has come up with in the past and why my career lay in building more and more complex models like these.
(Incidentally, this whole measure vs. model thread got started on the security metrics mailing list just before Metricon 1.0 simply because very few folks have any data. This dearth of data was troubling to me. Metricon 2.0 was no different, to the extent that at one point I was compelled to ask a panel "what to count" and was criticized for it. Strange that – for me to want numbers at a metrics conference. So, at that point, I was a measurer. Now, I am a modeler.. I think because I believe in the risk equation as a very straightforward way to classify operational metrics.)
While I thought the prose was great (it always is with Andy), I was really disappointed with a number of the assertions he’s made. I know for a fact that people have been talking about activity-based costing and process metrics, two fundamental components of ROI and ROSI, for over two years now and he should, too. I cringe at the notion that "my camp" is somehow not interested in reality but only imaginary things, given that my whole approach to quantifying risk involves collecting historical data.
Andy really highlights his own false dichotomy throughout the rest of his well-thought-out answers. They incorporate models, ROSI, ROI, risk, and everything else we "modelers" think about. The fact that he doesn’t seem to be able to put his data together should not be a curse upon the profession.
Here is a good starting point on security metrics.
[A private note to Andy - please leave me out of your constraining characterizations in the future. They are incorrect and inappropriate. Thanks.]
Hey Pete –
Thanks for your post about my interview with Hoff. It seems that the sins I committed were twofold: putting your name somewhere in my post, and attempting to place you into a camp. My point in doing both wasn’t to call you, Alex or anybody else out. I meant simply to break down the metrics space into styles of thought — modelers, measurers, and the like, and to give a few examples of people who seemed to me that they, more often than not, were inclined to think in those styles. To me, that’s Press Relations 101: put together a simple narrative with plain language, and give examples. We can certainly argue over whether I got the camps (or their members) right. But let’s not forget what Hoff asked: he wanted to know why things were contentious. I half-agreed, and provided a theory as to why I thought that might be the case. That’s it.
By the way, lest you think that I was trying to do the security metrics equivalent of ethnic cleansing — forcibly dividing people into camps — you’ll note that I also wrote this: “Metrics aren’t really that contentious. Just about everyone in the securitymetrics.org community is pretty friendly and courteous. It’s a ‘big tent.’ Most of the differences are with respect to inclination.” It’s that last word — inclination — that’s important. In the same way that a left-handed person doesn’t turn into a clumsy slobbering fool when confronted with a water glass at a state dinner, being inclined to think about metrics a certain way doesn’t negate other modes, either. Indeed, that is exactly why my post included allusions to things like models, cost and revenue calculations in the logistics industry. There is no “dichotomy”: Saying so is (grin) simplistic too.
With regard to activity-based metrics: you won’t get any argument from me that certain people are talking about this. My rant about that was more directed at security product companies: I should have specified that those are the “nobodies” I meant. Vendors rely almost solely on risk and threat “metrics” as the thing they report on. I think we need operations-oriented activity-based costing (wow, there’s a mouthful) supported directly in security products, not ever-more-polished Hamster Wheels of Pain.
@Andy -
Timing is everything. Thanks for the clarification – I figured it was something like this. I don’t object to you using my name, I object to you using my name in conjunction with a “camp” that you consider interested in imaginary things.
Pete