The value proposition of detective controls can be a bit trickier than it seems. From a Return on Security Investment (ROSI) perspective, the overall goal is to increase the costs associated with crimes and therefore:

1. Reduce the number of incidents that occur through a deterrent effect;
2. Increase the likelihood that bad guys will be identified;
3. Contribute to the body of evidence available for a crime; and
4. Shorten the time span of active investigation.

From a Return on Investment (ROI) perspective, the cameras may be much less expensive than alternative measures being considered (or replaced) for crime fighting.

Regardless of what side you are on, this is the right discussion to be having.

Assuming you are a conscientious security professional, you actually went through the process of risk assessment and completed it when you decided it was "worth it" to make the purchase. Informal, ad hoc, seat of the pants are all tried and true methods of success in this field.

In fact, you, as conscientious security professional, conducted an informal calculation of the consequences and the probability associated with some negative event that this $250k countermeasure is supposed to eliminate or reduce.

Now, I can't tell you exactly what consequence and probability values you implicitly decided on, but I can tell you that their product must be greater than $250,000. It could have been a 5% chance of losing $5 million or a 50% chance of losing $500,000. Perhaps it was a 1% chance of losing $25 million.

In order to get your money's worth (e.g. ROSI or
return on security investment) that $250,000 must be LESS than the
amount of reduced risk. If you assert otherwise, then it wasn't a good
decision. Let me say that another way – if you tell me I am wrong, then
you are making a poor decision for your organization.

Rich Mogull objects strenuously, saying that Andrew's term is factually incorrect.

while I disagree with Rich's reasoning – I don't believe made-up terms…or labels   can ever be "facts" – I do agree that Andrew's choice is an awkward one.

The problem with the word "labeling" is that it is too passive. Even the verb label reminds me of a jar of jam, not some gee-whiz technology that someone wants to buy (which, in the end, is the reason for product categories to begin with).

Of course, I also empathize with Andrew and his campaign against rights (not very politically, correct, that ). However, I think his ERM has more traction than McClean's ERM, as evidenced by his own admission that he expected the post to be about rights and not risk. So, I don't think it is the end of the world to continue with ERM, or just switch back to DRM. This has the benefit that everyone knows what it is, albeit perhaps without positive connotation.

If we want more accuracy in product category naming, I suggest considering something around the notion of "object control" or "data control" or maybe something a bit fancier like "data" or "content shield".

Ultimately, the thing that matters is that people know what you are talking about.

Remarkably, after making this statement, he follows it with:

he irony of this superficial take on security spending (and perhaps the biggest "if" as weak probability statement in the security profession) highlights the nature of the problem and gives us an opportunity to see the value of applying some quantitative measures. Some of you might be agreeing with the statement, but if I say, "Okay, I am going to spend a trillion dollars on it" you would likely suggest it is a ludicrous amount. It might even make you mad, even though I wasn't the one who said "it doesn't matter."

Here's the point: someone lost in ambiguity-land can say "it doesn't matter how much you spend" and not really mean it, while someone with quantitative measures would quickly be shown the door at $1 trillion, $1 billion, $100 million… until some point that is considered "reasonable" within the context of the situation.

In short, folks who use quantitative measures can be judged. The level of precision may not be accurate, but everyone understands it and can offer their own value judgments to the approach.

On the other hand, using subjective "expert" opinion, one can establish much more wiggle-room. I think of this as a cop-out.

No doubt, this stuff is hard. It will never be perfect. But it is ultimately more beneficial to an enterprise than existing guesswork, if only to get past the subjective joke of suggesting "it doesn't matter how much you spend on security." Clearly, it matters.

The arguments for disclosure first tug at the heartstrings with simplistic platitudes like "it is better to know than not to know" but then grounds itself a bit with the following logic:

1. The bad guys already have this information
2. The good guys need the information to protect themselves

There you have it – a classic fight between good and evil. But even this is incredibly simplistic. It treats two groups as holistic entities and not dynamic populations. And that matters a lot when evaluating risk.

Essentially, folks who support higher levels of quicker disclosure are betting that good guys can and will respond faster and more completely than the bad guys can attack. With discrete groups this may be true, but with dynamic populations I am not so sure.

Risk is a function of threats, vulnerabilities, and consequences. The variance in these elements is constrained by scarce resources on both the attacker side and the defender side.

The attacker makes his decisions based on a cost-benefit analysis that compares costs – skill, effort, and equipment – to the expected benefit discounted by potential penalties (the attacker's risk equation). The higher the result of this equation, the higher the risk to an organization (because threat is higher).

The defender makes a ROSI (return on security investment) assessment (typically ad-hoc) to determine her overall risk. The lower the cost of protection, the more likely that investment is a good one.

Finally, we shouldn't forget opportunity cost which compares these results to anything else the attacker or defender might want to do.

Looking again at the disclosure reasoning, the question is whether releasing more information helps the good guys or the bad guys more. The "bad guys already have this information" argument neglects the acquisition cost of this information and the skill level required to execute.

A basic illustration of cost associated with "effort" – some of you were no doubt a bit annoyed as you read my first paragraph above and wanted to see the source material in question – it didn't have links to the pertinent Sourcefire and Metasploit blogs. Of course, you "already had" this information in the form of your ability to use search engines to find it. Links are a part of the Web culture because we recognize that time is money and making things a bit easier for the reader lowers his/her costs.

So the short point is that distribution of information, and its corresponding ease of access, matters.

The "good guys need this information for protection" is perhaps a trickier. The huge majority of Internet users do not need the information provided because they have no capacity to leverage it for protection. (Guys like HD Moore can do wonders with it, of course). The users rely on the makers of products who DO need this information to provide protection.

It is clear from this case that many large security companies already had the information (they already had samples), so the added benefit to the "good guy" community must be adjusted with that information in mind.

In the end, I think it is less likely that good guys used this information for protection than it is that bad guys used it to compromise some user. I believe this is almost always the case, and my evidence is the aggregated number of exploits that occur after disclosure compared with the number of exploits of undercover vulnerabilities

Trust me, I have been there and done that with the whole ROI in security thing. And I honestly thought I would read the section that said you can't get ROI because security doesn't generate revenue (I don't think that matters), along with the "and another thing" that ROI isn't even a very good model to use anyway.

Instead, I found this (my comments in between italicized excerpts):

No, you don't attempt to quantify loss, you quantify your costs and hold the revenue constant. That means ROI comes from reduced costs. Typically, you don't factor in losses, though that is a possibility for the high-frequency recurring events.

The "always producing a negative result" doesn't really make any sense when you apply the model to two different alternatives (your current state and the modified state).

This is the standard 'no-revenue' objection which I disagree with but understand. The slip in the final part of the sentence perhaps explains the faulty reasoning – you aren't calculating revenue, you are calculating a return (based on the net between revenue – held constant – and costs).

This is incorrect.

ROI generally comes from increased productivity – the gain that comes from substituting computing for human labor over a period of time. This, by the way, is the goal of essentially all IT. The items discussed above are again associated with losses and though they can be quantified as well, that is where ROSI comes in, not ROI. 

I certainly agree with garbage in, garbage out.

Further reading:

Can you get ROI from reduced costs?

Ten Points about Security ROI and ROSI

What is threatening about ROI in security?

ROI is about value

Return on Security Investment (and a little ROI)

Chris over at Zero in a Bit has a thoughtful post on the timeline for the recent Real Player vulnerability found by Gleg. This strikes me as the type of thing we need to learn to live with. Though perhaps not optimal to some, it certainly is a better case than the undercover exploit. And in any case, it is pretty difficult to change, given the slow evolution of the discovery/disclosure cycle. There is so much resentment inherent throughout the process, that it ends up creating a hot potato of blame:

• Blame the developers (Chris: "Certainly, Real needs to increase its efforts to reduce security vulnerabilities in its shipping products." If perfect software is the goal, then I can see where he could make this statement. Else, I would love to know what objective measure of effort is expected. I guess this is a good use case for the WTF/Minute.)
• Blame the bugfinders. Actually, I think part of the problem with this evolving situation is that we haven’t blamed bugfinders in the past. Certainly, some high-profile folks have made good name and good money from this process (while increasing risk in the process). Now, bugfinders have to look for different ways to make the name and the money. Enter Gleg.
• Blame the users. Prevailing wisdom appears to be that once the patch is available, it is the users fault if they don’t apply it. This, of course, is prevailing only with folks who work with homogeneous, vanilla, small IT infrastructures. Anyone with significant enterprise experience immediately recognizes the inherent difficulties herein.

I don’t think any of the players actually empathize with the plight of the others and recognize how much misinformation and miscommunication occurs throughout.

You’d think by this stage nobody would rely so heavily on a process that can’t be controlled, especially given that the reliance completely ignores what is widely regarded as conventional wisdom that undercover exploits are common (ain’t no patch for them).

Nice job!

As Alex Hutton (who is in my "camp") points out – it is not clear that the "model vs. measure" characterization is anything more than a false dichotomy – the two need each other. I know this even more personally because I cringe at the characterization and am compelled to write this post simply to counter (and in some ways clarify) the parts that miss the point. In fact, if I had to pick a "side" (you can’t really) I would want to be considered a "measurer" and NOT a "modeler".

The big problem with measuring is simply that there isn’t enough data to go around, so I find myself doing a lot of modeling – here is a good one: risk{threat, vulnerabilities, consequences}. Yes, you must see that I am brilliant to come up with such a fancy algorithm for risk that nobody else has come up with in the past and why my career lay in building more and more complex models like these.

(Incidentally, this whole measure vs. model thread got started on the security metrics mailing list just before Metricon 1.0 simply because very few folks have any data. This dearth of data was troubling to me. Metricon 2.0 was no different, to the extent that at one point I was compelled to ask a panel "what to count" and was criticized for it. Strange that – for me to want numbers at a metrics conference. So, at that point, I was a measurer. Now, I am a modeler.. I think because I believe in the risk equation as a very straightforward way to classify operational metrics.)

While I thought the prose was great (it always is with Andy), I was really disappointed with a number of the assertions he’s made. I know for a fact that people have been talking about activity-based costing and process metrics, two fundamental components of ROI and ROSI, for over two years now and he should, too. I cringe at the notion that "my camp" is somehow not interested in reality but only imaginary things, given that my whole approach to quantifying risk involves collecting historical data.

Andy really highlights his own false dichotomy throughout the rest of his well-thought-out answers. They incorporate models, ROSI, ROI, risk, and everything else we "modelers" think about. The fact that he doesn’t seem to be able to put his data together should not be a curse upon the profession.

Here is a good starting point on security metrics.

[A private note to Andy - please leave me out of your constraining characterizations in the future. They are incorrect and inappropriate. Thanks.]