Engineers are focused on quality, so when they hear about vulnerabilities in software, their immediate reaction is to want to fix them… all of them. Regardless of whose software it is. Regardless of where it’s deployed. In fact, some of them care so much that they go out seeking vulnerabilities simply to fix them. They are the type of people who are great at solving problems, but not at understanding the downstream implications of their actions.

Economists, on the other hand (get it?), look at cause and effect, actions and reactions, and, most importantly, outcomes. The root of the economic problem lay in the ultimate unwanted outcome – the breach.Economics-oriented security pros understand that everything we do is intended to thwart the breach. It is easy to lose track of unwanted outcomes in the face of compliance needs and operational activities, but even those activities are all intended to minimize damages from attacks and exploits.

The engineer correctly believes that fixing vulnerabilities creates high quality (“stronger”) software. If the program starts with 300 vulnerabilities and you fix one, that obviously leaves 299 – one less than when it started. More importantly, if an enterprise has 1,000 systems that all have that same vulnerability and they apply a patch to 500 of them, they have decreased their attack surface by 500 vulnerabilities. From both perspectives, the level of vulnerability is, in fact, reduced.

But the economist knows that fewer vulnerabilities is not the ultimate objective. The ultimate objective is to reduce the likelihood of an incident.

The economist understands that there is a key missing ingredient to the engineer’s scenario – the intelligent adversary, aka the threat. And in pursuit of higher quality software the vulnerability details usually get published, leading to lower attack costs for the adversary. Given the scalability of technology, this typically leads to more attackers connecting to more targets, albeit in a (somewhat) smaller population of targets.

That is the key observation for this discussion – a breach requires both an attacker (threat) and a target (vuln), which manifests itself in the form of a connection between source and destination. Even though the population of targets may be reduced (perhaps even significantly so), if the threat is sufficiently motivated, more connections can be made with the vulnerable targets. The only way to guarantee reduced risk is to bring one of the populations (most likely the vulnerable targets) to zero. History shows us this is not likely with commercial software in enterprises. Interestingly, the increasingly common scenario for cloud-based software (e.g. Software-as-a-Service) may be able to do just that.

And there you have it – given the need for both threats and vulnerabilities, the reduction in one doesn’t force a reduction overall. And if the other element is increased in the process, the marginal difference in each population must be evaluated to truly understand the impact. Historically, this has led to scenarios where the vulnerability is reduced while the risk is simultaneously increased.

For reference:

More Sex is Safer Sex…

I believe the folks at Gartner put a lot of research and effort into their Magic Quadrant analysis. That said, I can’t help but conclude that “vision” and “execution” don’t quite do it for me when it comes to identifying appropriate candidate solutions to address a problem. They just seem to be too much about marketing, which is very important to the companies but only ancillary to an enterprise’s needs. Sure, they want a solution that will be viable for the long-term, but other than that it is pretty insignificant.

To address this issue, I have put together a set of questions in 4+1 evaluation categories that I believe provide more insight into the important attributes of a solution. The first round of categories was introduced at AMP NYC a month ago. Here is my second revision. Opinions and advice are welcome.

1. Company/Product Information: What level of confidence does the company information provide that the company and product will remain viable for your organization?

Consider:
• What year was the company founded?
• What is the background of the management team?
• How many employees does the company have?
• What is the funding status/source of finances?
• What is the product name and version?
• How many customers does the company have for the pertinent product?
• What certifications and tests were done on the product?
• What other 3rd party reviews, awards, or other supporting evidence exists about the product?
• What is the pricing model for the solution?

2. Functional Operation: What level of benefit does the functional operation of the product have?

Consider:
• Primary operation – scan memory state, scan configuration/file system/network state, monitor/record system call activity, monitor/record network traffic, isolate memory, isolate system activity, isolate network communications.
• Trigger action – detect “known good” execution, detect “known good” activity, detect “known bad” execution, detect “known bad” behavior, detect anomalous execution, detect anomalous behavior.
• Response options – allow, deny execution, kill process, kill network connection, reroute network communication, log event, notify user, notify admin (alert), other.
• Recovery options (post-infection) – Restore config to known good state, remove bad files/objects, identify similar issues across network, notify/update other control solutions.

3. Architecture & Administration: How well does the product’s architecture fit in with your organization’s existing security processes? How likely is it to provide benefits? What features does it have to support implementation and administration?

Consider:
• Where/how are any product sensors or agents deployed throughout an enterprise (endpoint, network, cloud, other)? How are they protected?
• Where/how does the product admin/management function work? How is it protected? (endpoint, network, cloud, other)
• Where/how does the product log/data/storage function work? How is it protected? (endpoint, network, cloud, other)
• How is information shared a) with the solution components; and b) with others?
• How does the solution get installed/implemented in the environment?
• How customizable is the configuration and interface?

4. Technical Integration: How well does the solution integrate into the IT ecosystem? How easy will it be to implement and maintain?

Consider:
• How does the solution integrate with other products from the same company?
• How does the solution integrate with 3rd party security solutions?
• How does the solution integrate into an IT architecture?
• What are the prerequisites for user directories, management servers, etc?
• What standards, communication protocols, platforms, languages, frameworks, etc. are supported?
• How robust is the API for third party access?

The final category is actually a rollup of the other four, since the differentiators and value come from the previous specifics being identified.

Key Differentiators / Overall Value Proposition
When looking at the complete picture of the solution, how strong are the overall benefits derived from the individual evaluation categories?

I believe these evaluation categories more properly reflect the needs of the enterprise. What do you think?

1. Need. I choose the word “need” with caution, since, as you will find out below, it does not necessarily mean there is “demand” for a better solution. However, I don’t think techrisk professionals can deny that the malware dropping attack vector is alive and well. It is highlighted as the key to the Aurora attacks that catalyzed the “advanced persistent threat” concern.
2. Varied Solutions. There are a number of vendors that have cropped up through the years with solutions to address the malware problem, and the techniques vary significantly. Whitelisters only allow identified executables to run; sandboxes isolate malware and/or identify actions; and real-time forensics track system calls and/or configured state.
3. Mature Market. Even with an identifiable need and newer interesting solutions, the most powerful security market in the world – antivirus (nee antimalware) – operates in pseudo-commodity mode and dominates in endpoint security.

As an industry analyst, I have had the opportunity to interview over a dozen solution providers and even more enterprise security architects and executives on the state of antimalware in the enterprise. Here are a few of my conclusions:

• Companies are moderately satisfied (and perhaps complacent) with their existing antimalware solutions. They acknowledge that these solutions are not blocking all malware but believe that every solution in the category has similar problems and so are reluctant to switch.
• The only factor that could affect existing signature-base antimalware is price – a lower-cost solution (which many agree is unlikely) could have a strong-enough value proposition. Notably, a few organizations are evaluating Microsoft’s free antimalware solution as one of these alternative options.
• Organizations are looking to gain more benefit from their existing antimalware solutions. Many are still focused on signature-based functionality and are now looking at more advanced capabilities. In addition, organizations are considering and employing new capabilities like Microsoft’s EMET functionality.
• For those times when malware gets through and infects a system, re-imaging is the standard approach, though some organizations are mildly reluctant to do it. Most of these malware infections are not classified as “incidents” per se – there is an ad hoc evaluation process to decide whether any infection should be escalated into being classified as an incident.
• Organizations are looking at architectural changes and not product changes when it comes to endpoint client-side security. This means they are focusing on BYOD and/or VDI (or even dumb terminals) as options in their client security strategies.
• Control over (physical) clients continues to relax, with certain “pockets” of exceptions (kiosks or manufacturing systems). For some, this was after a long period of control strengthening (e.g. finally taking away local administrative rights).

As I mentioned at the start, the market dynamics fascinate me here. I don’t think there is a techrisk professional left that believes signature-based antimalware is “good enough” and yet we see its dampening impact everywhere. At this stage, it has simply become the “checkbox compliant” easiest approach.

As someone extremely interested in cybersecurity economics I am encouraged by the attention being given to the bottom line – organizations should be very careful about cost-benefit in their security programs. While some of the organizations I interviewed had done a comprehensive analysis, it appeared to me that a number of organizations had not undergone a thorough review of their strategies.

I will be addressing these issues at my “Drinking from the AMP Firehose” workshop in New York City in a couple of weeks. The workshop concept was driven by these ideas and aims to break through the logjam brought on by complacency and confusion. Regardless of the conclusions that individual organizations come to, I think the entire field will be better off for it.

Pete Lindstrom is Principal and VP of Research for Spire Security, LLC, a research and advisory firm. Learn more about Advanced Malware Protection by “Drinking from the Firehose” in New York City on 9/17/13. Details at www.regonline.com/AMPFirehoseNYC.

On the protection side of the equation, although antimalware solutions provide a basic (and compliant) level of protection, security professionals are well aware of the limitations of signature-based approaches. Solutions that have been around for a while, such as host intrusion prevention and whitelisting, have gained renewed interest. Other approaches like network or endpoint sandboxes for isolation and/or analysis or active forensics for near-real-time analytics are coming on strong.

The challenge is determining whether the additional cost is worth it, deciding whether a new solution will significantly reduce the problem, and identifying which type of solution(s) are best.

While it is easy for security professionals to claim they will spend “whatever it takes” to address technology-related risk, that assertion is easily deflated through extreme examples (millions? billions? trillions?). While the intentions are valiant (and I get the point), no organization has an unlimited supply of money to spend on security. Therefore, it is crucial to make good decisions about how and where to spend money.

Any business decision is accompanied by some sort of justification, and cybersecurity is no different. In security, we typically evaluate total cost of ownership of the solution and compare it to our notion of how much risk is reduced. At the very least, every purchasing decision is supported by a claim that the spending “is worth it.” At best, a more formal cost-benefit approach should be employed.

Evaluating the cost-benefit of an “advanced malware protection” solution can be extremely challenging. Dropping malware (in the form of viruses and worms) onto systems is one of the oldest methods of attacking and compromising computing environments. Because of this, all enterprises already have controls in place that attempt to protect against malware infection. In addition, there are a number of techniques that can be used to address the problem.

Regardless of the challenge, conducting an economic analysis of newer AMP solutions may lead to some surprising conclusions. Here are six key considerations for conducting your analysis.

1. Ignore the “Advanced” Part of Advanced Malware Protection

The first distinction you should make in reviewing your needs for “advanced” malware protection is that the “advanced” part is extremely nebulous – the bar keeps changing in defining exactly which techniques are advanced and which aren’t advanced. Accordingly, the first takeaway is “evaluate your AMP solutions in concert with all antimalware efforts in your organization.”

This should not be a radical thought.

2. Cover All the Antimalware Bases

Diving a bit deeper into costs, enterprises should consider the costs of all capabilities – the capital investments made on hardware and software, maintenance costs, and personnel costs. The vendor solution (capital investment) side of antimalware protection can include endpoint antimalware, email or gateway-based antimalware, intrusion detection (potentially), and secure web gateways. On the operational expense side, organizations should consider the personnel costs associated with identification, prevention, mitigation, response, and recovery activities associated with malware infections and incidents.

3. Allocate Partial Costs of Broader Solutions

Focusing on the costs associated with one type of threat – in this case, malware – can be challenging. Some solutions, like endpoint antimalware, focus directly on the problem while others provide varying levels of accompanying support. In my research, for example, secure web gateways were cited as a means for detecting malware infections that were undetected by endpoint antimalware solutions, but secure web gateways provide more capability than malware infection detection.

The key in the analysis is to allocate costs based on the proportional value provided by the broader solution. If 10% of the ongoing value of the solution comes from antimalware detection, then 10% of the future costs should be allocated to antimalware.

4. Ignore Sunk Costs

The maturity level of antimalware makes it likely that capital investments to address the problem have already occurred. Any spending that occurred in the past should be excluded from the analysis, though any current and future operational expenses should be included. In contrast, a decision involving a future capital investment should include that amount allocated (either amortized or depreciated) over its lifetime as well as the operational costs.

5. Factor in Employee Productivity

The second economic issue to consider is the productivity of employees. The productivity costs associated with the impacted worker should be considered along with the costs associated with the IT triage person. If it takes four hours to recover an infected system, then four hours of the worker’s lost productivity should be included (nothing fancy here – use a single average number based on salary for all workers).

6. Use a Breakeven Approach

Perhaps a bigger challenge in justifying antimalware spending is in determining the amount of potential losses. That “it is worth it” decision means that the security professional spending $100,000 on a security solution believes the solution will offset at least $100,000 in risk.

While some cringe a bit at the realization that spending reveals the minimum expectation of risk reduction, it also provides an opportunity to conduct a standard financial breakeven analysis. Rather than attempting to figure out the exact amount of financial loss at stake, you need only consider the total amount being spent and determine whether it is less than the potential losses. So as long as you’ve properly accounted for all costs, an appropriate decision can be made.

The AMP solution decision is not an easy one – with a handful of controls at different maturity levels in the organization already and a variety of newer solutions vying for attention. Some organizations may even come to the conclusion they don’t need to augment their existing capabilities. Others will find out they really do. Regardless, enterprises should be conducting the necessary analysis to make the best decision for its needs.

Pete Lindstrom is Principal and VP of Research for Spire Security, LLC, a research and advisory firm. Learn more about Advanced Malware Protection by “Drinking from the Firehose” in New York City on 9/17/13. Details at www.regonline.com/AMPFirehoseNYC.

I am not so sure. (And, to be fair, I am not clear how strong Phil’s opinion is on this.)

The idea of some sort of “cyberSwitzerland” sounds like a direction we could head in, but we immediately run into questions of trust, oversight, and technical capability.

• Trust – The first step is to identify an entity (presumably in the cloud service-providing business) that we trust more than the U.S. Government (since they are the bad guys with this NSA spying scenario). This doesn’t seem particularly onerous – any of the big players might do – Amazon, Google, etc… Some privacy-supporting org like the Electronic Frontier Foundation might also consider getting into the business or endorsing some service. (Come to think of it, maybe we just need James Earl Jones to or Martin Sheen to endorse a no-name. William Shatner, too ). But the problem really comes with oversight.
• Oversight – Any entity we decide to trust more than the U.S. Government would also have to be willing to snub the U.S. legal system. This is the problem area, because no large entity with U.S. operations would snub the U.S. legal system – heck, that’s probably part of the reason you trust them – they follow rules. So the linchpin problem is that any trustworthy entity also will (ultimately) obey the law and so we are right back where we started from. Like they say, there is no honor among thieves.
• Technical Skill – The final nail in the coffin is that, even if you found an entity you trust who is willing to snub the U.S. legal system, they need to be able to protect you from the NSA. Any successful entity in this endeavor would obviously become a prime target for them. It is unclear whether this is possible in the long run, especially given the many ways to compromise systems. At the very least, it would be quite expensive.

Ultimately, I believe anyone going through this analysis will come to the conclusion that a “CyberSwitzerland” cloud service provider is highly unlikely to be able to address the needs of those concerned enough to make a change (who aren’t breaking the law in some way). That is, for the average U.S. citizen, a “CyberSwitzerland” is not a way out.

There are ways, however, that could significantly help the average citizen concerned about privacy. The real answer here has got to be some form of obfuscation – at the very least encrypted data, perhaps augmented by more unique schemes of data dispersal and split-key techniques. And the super-paranoid might even throw in some “chaff” generation along the way to add noise to whatever analysis is putting you in the ‘results’ list to begin with. Heck, you could even hire 20 people around the world to impersonate you and encrypt random data uploaded to random sites with a “firewall” between you and each of them (sort of joking here).

This is an important announcement because it highlights the very real problem of “in-the-wild-exploits of undercover vulnerabilities.” This strain of “0day” is the most significant given that active exploits are already happening when they are discovered. In these scenarios, the threats (malicious actors) and vulnerabilities have already collided in the real world and losses are being actively incurred. Thus, this type of situation is the most important type that technology risk (techrisk) managers must deal with in their environments.

The announcement itself highlights some important, underappreciated aspects of the techrisk profession:

- That exploits/breaches/incidents are the fundamental “unwanted outcome” that we are trying to prevent. It is not uncommon for techrisk pros to focus efforts on software quality, control weaknesses, or compliance violations – all useful intentions to the extent that they address the aforementioned incidents.

- That techrisk professionals can identify attacks even when the vulnerability is unknown. Much of our profession’s focus revolves around the notion that we must find vulnerabilities in order to protect ourselves, yet time and again we succeed in identifying these types of attacks using behavioral analysis and other techniques. With the growth in popularity of forensic archiving, we can now also determine to what extent we have been victims in the past to assist with understanding the risks of the future.

• That much of the profession’s effort associated with vulnerability management is ineffective. Our efforts to identify each vulnerability prior to exploit are simply overwhelmed by scale and can simply be shown through a thought exercise – consider how many vulnerabilities are created every day (in the aggregate) as compared with how many are found. Perhaps more importantly, it is worth noting that the vast majority of vulnerabilities that are found are never known to be actively exploited [pdf].
• That there is a variance in how different types of attacks – namely, targeted vs. opportunistic – manifest themselves online. Google’s primary cited reason for its new policy involves political activists as victims of targeted attacks that may lead to physical harm. The history of infosec and techrisk highlight other scenarios – the NIMDA worm, WMF exploit, WebDAV, etc – that involve opportunistic exploits across a multitude of targets.
• That the most significant way to “move the marker” in security is through the identification of exploits and not vulnerabilities. As with Code Red and Nimda in the Fall of 2001 leading to Bill Gates’ well-known “Trustworthy Computing Memo,” active exploits are the best drivers of change in the techrisk profession.

 While Google’s new policy offers and opportunity to assess the state of security on the Internet overall, it also demonstrates significant deficiencies in its approach:

• The 7-day deadline has no risk basis. With the significant variance in number of affected parties and speed of compromise associated with opportunistic attacks versus targeted ones, the number is an arbitrary one. In the primary example cited (activists at risk of physical harm), speed is highly unlikely to have a significant impact on risk reduction.
• The capabilities of enterprises and/or users to protect themselves can vary significantly. There are many reasons why some parties choose to remain vulnerable to certain types of attacks – system complexities, legacy support needs, lack of technical skill, competitive priorities, etc. Through the years some security researchers (including some employees of Google) have expressed disdain for those that cannot protect themselves. A company the size of Google should be held to a higher standard in its willingness to protect those online that can’t always protect themselves.
•  No consideration of economics. The policy completely ignores tradeoffs like the risk of breaking systems when taking precautionary measures (e.g. patch failures), the well-known increase in exploits that occur after the disclosure of many new vulnerabilities [Arbaugh, McHugh, 2000 pdf; Bilge, Dumitras 2013 pdf], and the opportunity costs associated with new requirements. When Google says, for example, “each day an actively exploited vulnerability remains undisclosed to the public and unpatched, more computers will be compromised” they neglect the significant likelihood that computers will be compromised regardless of the state of disclosure to the public and fall back on the age-old myth that only patches can protect systems.
•  It can lead to even more exploitations and incidents. Anyone paying close attention to the vulnerability research community knows that there is wide variance in how researchers disclose their information and some decisions are made based on annoyance, frustration, spite and sometimes even malice. If a vulnerability will get “noticed” more quickly, researchers may be tempted to “test” it in the wild in order to increase its priority level.

A company with the talent and resources at Google can do better. Here are some opportunities for improving the state of security on the Internet and addressing the real, significant risk associated with actively exploited 0days:

• Encourage and train political activists in obfuscation and evasion techniques. It is challenging to discuss a blanket policy across all scenarios simply by highlighting arguably the most important one – that involving physical harm. It seems highly unlikely that this case is a common one and the best way to discuss the overall implications of the policy itself is to remove this scenario from the discussion as it tends to cause an emotional reaction. As many of us know, there are many ways political activists can protect themselves online that would be much more effective than a 7-day disclosure policy which comes after they have been compromised.
•  Increase focus on actively exploited 0days. Since these are the most important scenarios the techrisk profession has to deal with, Google should be making every effort to identify these exploits and employ or invent new ways to protect against them. Google researchers still participate in random, ineffective vulnerability research that simply distracts from this very real problem.
•  Provide more insight into the “dozens” of 0days identified “through the years” that was mentioned in the blog announcement. If there is one thing Google has, it is great data. As evidenced by past reports [Provos, 2008 pdf], Google could very easily provide more specific evidence on the number of 0days they have identified, the volume of exploits, and their disposition by vendors. The fact that they haven’t yet, especially in the face of this policy announcement, is disappointing and makes it difficult to evaluate the measure.
•  Take a risk-based approach to disclosure. Fast-moving worms do most of their damage in hours and days – in those cases, seven days is too long. Targeted attacks are unlikely to get repeated in a way that demands immediate attention for most environments – in those cases, seven days is too short. A risk-based approach would take into account the frequency of exploit, probability of future exploit within a target population, and impact of the exploit while evaluating the changes to these variables over time – in particular before and after disclosure.
•  Monitor the situation closely. Google’s unique ability to gather data in this regard is worth mentioning again as a function of its ability to assess its own policy. Collecting and publishing data on actual 0days throughout their exploit lifecycle would be a boon to the entire profession.
•  Initiate or participate in discussions to create new ways to address this very real problem. Commercial, community, and government mechanisms already exist for sharing data publicly and privately that could be used as models for minimizing the risks associated with these types of attacks. For example, a (private) process similar to federal wiretap capabilities in secrecy and opportunity may be more effective in addressing targeted attacks. There are countless other approaches that could be leveraged to address this problem.

Make no mistake, the Google 7-day policy announcement sheds light on a real and significant issue in technology-related risk. While it highlights some of the challenges techrisk professionals face on a daily basis, it also demonstrates significant deficiencies in its approach to address the problem. This is a great opportunity to evaluate the existing state of the Internet from a risk and security perspective to determine where inconsistencies or weaknesses lay and map out a risk-based program that has the highest likelihood of success.

“If you want a bug fixed quickly, sell it on the Russian black market. It’ll be so heavily abused that the vendor will patch out of cycle.”

Now, it could be the joke’s on me and the 126 people who retweeted this message (a large number for security tweets) were in on it. Or, they all don’t realize how ludicrous this is. In the infosec/techrisk field, this kind of thinking is not unheard of so I will treat this as if it is legitimate.

The tweet highlights just how biased people can be when they get caught up in a notion without understanding the implications. Apparently, this tweeter wants bugs fixed quickly. At first blush this seems like a simple enough concern, shared by many. But peel one small layer deeper and the statement often ends up being “want bugs that you know about (or worse, that you discovered) fixed quickly after your discovery?” It becomes easier to see how certainty bias and the focusing illusion come into play.

there is plenty of evidence to demonstrate that it is unlikely that the bug in question is the only bug that remains unfixed – we have any number of bugs in various stages of discovery and disclosure all the time. If we assume that the average bug takes 120 days from discovery (or at least vendor notification) to patch release, and vendors generally release patches on a monthly cycle, then there are four months of undisclosed (typically) vulns on your systems that remain upatched.

Now, you might assert that this makes the point – of course we want them patched “quickly.” But that completely ignores the tradeoffs. If your patch is prioritized, that means another one must be de-prioritized. I suppose you could say that security developers aren’t operating at capacity and therefore can absorb the workload for both bugs, but that seems farfetched to me and doesn’t scale in any case.

Of course, the worst part of the tweet is the part that purposely increases risk by increasing the threat of compromise. No need for a soapbox/high horse here to recognize that purposely inflating risk to get attention in spite of how detrimental it is to Internet users is certainly unprofessional and really kind of pathetic.

Too often, folks get caught up in some perceived solution to a problem and neglect the bigger picture. Many times, the bugfinder is sincerely concerned. But it is important to understand the cost/benefit and risk dynamics involved if you really want to positively affect Internet risk.

When estimating losses, it isn’t entirely unreasonable to do the high-level straight-line math like IT World did here:

“Amazon.com’s latest earnings report showed that the company makes about $10.8 billion per quarter, or about $118 million per day and $4.9 million per hour.”

It’s really quick and dirty – and in a general sense legitimate – but can we do better? There are other ways to look at this that might shed some light on impact assessment. First, the assessment above makes no mention of costs. That might be the biggest weakness since costs are more under the control of Amazon and (probably) don’t fluctuate as much as revenue.

Luckily for us, Amazon just released its quarterly earnings report and this report asserts that its operating margin is about 3%. So right off the bat, we could suggest that Amazon lost 97% of $5 million or $4.85 million in costs. A more conservative estimate might try to determine whether the costs were unrecoverable or not, etc. Hopefully, you get the idea. A cost-oriented approach also works well as an example in infosec since that is often a big piece of the losses we face.

It is important to note here that these costs are additive to the lost revenue estimate – not only did we lose the $4.85 million in operating costs, but also (presumably) we lost that initial $4.9 million in revenue, for a total of (let’s say) $10 million.

Now, let’s look again at that lost revenue estimate. As mentioned earlier, coarse numbers like those used in the calculation above are certainly justifiable but we can probably do better. A quick thought exercise can help here – by creating the experience of an “average customer” of Amazon’s we can better assess the impact of the outage. This is harder than it sounds because we’ll have to second guess our own biases, but let’s try anyway. Let’s call him Joe.

Given that the outage was simply a “denial-of-service” of sorts, the big variable we must evaluate is time. More specifically for our scenario, we need to answer the question “How timely does Joe’s interaction with Amazon need to be, or, how likely is Joe to wait an hour to complete his purchase?” At the very least, we know Joe is willing to wait two days (maybe more – not sure what the average delivery time is for Amazon) to receive whatever goods he purchases. Throw in what we might assume (my bias) about Amazon’s low prices and the corresponding brand loyalty that comes with it and it seems reasonable to conclude that Joe will wait an hour to make the purchase, and therefore the lost revenue is actually only deferred revenue to be recognized in the future.

But not everyone is average (usually nobody is), and so once we cover a generic case, it is useful to consider the impact of the outliers. Now, we can imagine scenarios where even though a customer can wait for delivery, she can’t wait to place the order – too many other things going on in life. Or even a case where the customer would actually lose a full day due to delivery cutoff times. These are the types of cases that warrant more attention. Certainly it is reasonable to factor these cases into a loss scenario. Let’s say this is true 10% of the time.

The goal here is to be conservative in our estimates (even though it is sometimes beneficial for companies to be liberal after the fact – can hide other problems) so we should remember that these scenarios are typically useful in identifying some sort of discount factor to apply to the initial $5 million estimate. Though it is possible to come up with scenarios where there is a multiplier – maybe holiday seasons – it is less common.

Our lost revenue evaluation has led us to conclude that 90% of purchases will still be made in the future, so the remaining 10% of cases will discount our $5 million loss down to $500,000. Add that to our lost costs and we are back to the initial $5 million estimate, though from a different perspective. While it might be attractive to decide all was for nought, it is worth considering the situations where the costs are much lower, or the revenue is more likely to be lost to see the value in the exercise.

Now, some might suggest (essentially) that the above analysis is really not worth it because a loss is a loss. Not only that, but Amazon’sown numbers have shown (?) that there is no discernible uptick in sales in the period following the outage. As mentioned earlier, it is easier to see how costs are fairly static and therefore turn into losses. On the revenue side, however, it is not clear at all.

In assessing lost revenue in this case, one must do two things: first distinguish between necessity and convenience and second evaluate the impact of buyer’s capacity. The purported lack of a noticeable uptick in sales in the short term could easily be explained if purchases are more oriented around convenience than necessity. Measures associated with shopping carts might be of assistance here (I sometimes leave items in my shopping cart for days if not weeks). Again, this information can be factored into the estimates if need be.

It is uncommon to consider a “buyer’s capacity” but especially with convenience purchases, one might decide that the rate of purchase is a determining factor and even though the shopper returns, she will be buying other items, etc. This justification is easier to believe in cases where capacity is high – that is, the shopper is buying at a rate where fitting in the “lost” purchases is unlikely (and when it happens is noticeable in the numbers). My assessment is that this scenario is unlikely; people are more casual in their shopping experience and will therefore wait to make their purchases. (A similar capacity limit could have an effect on the Amazon side, but that is even more farfetched).

My conclusion is that $5 million is a reasonable loss estimate for Amazon’s outage, but not for the reasons initially believed.

So, for example, using available data they calculated that automatic fire extinguishers in airplane lavatory trash receptacles cost $16,000 per life year saved. (This was in 1993 – maybe smoking was still allowed then?)

Interestingly, these costs ranged from “those that save more resources than they consume to those costing more than 10 billion dollars per year of life saved.” The median cost per life year saved was $42,000. The paper also breaks down amounts by type of intervention, prevention stage, and even provides some data on proposed govt regulations by regulatory agency (FAA median $23,000; EPA median $7,600,000).

As a quick aside, the existence of this data helps one understand that even though circumstances where “success means nothing happened” (in this case, death didn’t happen), there is still plenty of opportunity to assess the benefit of some particular intervention.

These types of “revealed preference” study results can be eye-opening to those that suggest we should spend “whatever it takes” to address some particular concern. In looking at the large variance in costs, perhaps that isn’t the best course of action. It is nice to think we have unlimited resources, but at some point they run out. When they do, not only does that impact overall effectiveness, but opportunity costs come into play.

What does this mean for cybersecurity? Though it is not fair any more to say there is no data available to our profession, it certainly is difficult to leverage the data coming out in ways that are helpful to an organization. However, we can start thinking in terms of estimates and measures that make sense. In particular, we can evaluate and compare costs of various controls to each other and factor in some notion of anticipated risk reduction.

We can learn a lot from studies like these.

“After multivariate adjustment for major lifestyle and dietary risk factors, the pooled hazard ratio (HR) (95% CI) of total mortality for a 1-serving-per-day increase was 1.13 (1.07-1.20) for unprocessed red meat and 1.20 (1.15-1.24) for processed red meat.”

As with many studies about diet, lifestyle, and death, this one has sparked discussion. The Numbers Guy from the Wall Street Journal, Carl Bialik, wrote two articles on the study itself and the difference between absolute risk and relative risk numbers that often create confusion and annoyance. That article led me to the always excellent Understanding Uncertainty blog post by Dr. David Spiegelhalter’s fuller treatment of exactly what a 13% increased risk of death actually means (dying about a year younger, in case you are wondering). It also provides discussion on correlation/causation caveats and the practical application of the numbers.

All this discussion is interesting and should be useful for any IT risk professional interested in quantitative treatments of risk. But these details are not the reason I am writing this. As I was reviewing the information, it struck me just how difficult this is in the physical world. This quote from Dr. Willett in the L.A. Times article really highlights the problem:

“In principle, the ideal study would take 100,000 people and randomly assign some to eating several servings of red meat a day and randomize the others to not consume red meat and then follow them for several decades. But that study, even with any amount of money, in many instances is simply not possible to do.”

What struck me was not only how hard this is, but also the rigor of the results in the face of the described obstacles. And, even more importantly how much easier this would be for IT risk professionals in the virtual world.

In the virtual world, we actually could design and conduct a study that controlled for almost every variable to quantify risk. We could, for example, deploy 10,000 or 100,000 virtual machine clients around the Internet that were all configured exactly alike with the exception of some specified difference – patched vs. non-patched, different anti-malware solutions and/or signature updates, open vs. closed ports, other configuration changes, etc. About the hardest part would be determining how/where to deploy the VMs and coming up with a “honeymonkey” algorithm to mimic user activity.

Perhaps the biggest challenge would be recognizing and characterizing the intelligent adversary contribution to the variance in the numbers – the popularity of vulnerabilities, exploit techniques, 0days, etc. And that would be the good stuff, as well.

Conducting an experiment like this seems so easy to me that I wonder if somebody is already doing it. I am pretty sure some group (ISC?) used to do some sort of “time-to-compromise” metric for unpatched systems. And I suspect there may be others. Does anyone know of experiments/studies being done similar to this? If so, I’d love to hear about them. If not, why not?