I believe the folks at Gartner put a lot of research and effort into their Magic Quadrant analysis. That said, I can’t help but conclude that “vision” and “execution” don’t quite do it for me when it comes to identifying appropriate candidate solutions to address a problem. They just seem to be too much about marketing, which is very important to the companies but only ancillary to an enterprise’s needs. Sure, they want a solution that will be viable for the long-term, but other than that it is pretty insignificant.

To address this issue, I have put together a set of questions in 4+1 evaluation categories that I believe provide more insight into the important attributes of a solution. The first round of categories was introduced at AMP NYC a month ago. Here is my second revision. Opinions and advice are welcome.

1. Company/Product Information: What level of confidence does the company information provide that the company and product will remain viable for your organization?

Consider:
• What year was the company founded?
• What is the background of the management team?
• How many employees does the company have?
• What is the funding status/source of finances?
• What is the product name and version?
• How many customers does the company have for the pertinent product?
• What certifications and tests were done on the product?
• What other 3rd party reviews, awards, or other supporting evidence exists about the product?
• What is the pricing model for the solution?

2. Functional Operation: What level of benefit does the functional operation of the product have?

Consider:
• Primary operation – scan memory state, scan configuration/file system/network state, monitor/record system call activity, monitor/record network traffic, isolate memory, isolate system activity, isolate network communications.
• Trigger action – detect “known good” execution, detect “known good” activity, detect “known bad” execution, detect “known bad” behavior, detect anomalous execution, detect anomalous behavior.
• Response options – allow, deny execution, kill process, kill network connection, reroute network communication, log event, notify user, notify admin (alert), other.
• Recovery options (post-infection) – Restore config to known good state, remove bad files/objects, identify similar issues across network, notify/update other control solutions.

3. Architecture & Administration: How well does the product’s architecture fit in with your organization’s existing security processes? How likely is it to provide benefits? What features does it have to support implementation and administration?

Consider:
• Where/how are any product sensors or agents deployed throughout an enterprise (endpoint, network, cloud, other)? How are they protected?
• Where/how does the product admin/management function work? How is it protected? (endpoint, network, cloud, other)
• Where/how does the product log/data/storage function work? How is it protected? (endpoint, network, cloud, other)
• How is information shared a) with the solution components; and b) with others?
• How does the solution get installed/implemented in the environment?
• How customizable is the configuration and interface?

4. Technical Integration: How well does the solution integrate into the IT ecosystem? How easy will it be to implement and maintain?

Consider:
• How does the solution integrate with other products from the same company?
• How does the solution integrate with 3rd party security solutions?
• How does the solution integrate into an IT architecture?
• What are the prerequisites for user directories, management servers, etc?
• What standards, communication protocols, platforms, languages, frameworks, etc. are supported?
• How robust is the API for third party access?

The final category is actually a rollup of the other four, since the differentiators and value come from the previous specifics being identified.

Key Differentiators / Overall Value Proposition
When looking at the complete picture of the solution, how strong are the overall benefits derived from the individual evaluation categories?

I believe these evaluation categories more properly reflect the needs of the enterprise. What do you think?

This is an important announcement because it highlights the very real problem of “in-the-wild-exploits of undercover vulnerabilities.” This strain of “0day” is the most significant given that active exploits are already happening when they are discovered. In these scenarios, the threats (malicious actors) and vulnerabilities have already collided in the real world and losses are being actively incurred. Thus, this type of situation is the most important type that technology risk (techrisk) managers must deal with in their environments.

The announcement itself highlights some important, underappreciated aspects of the techrisk profession:

- That exploits/breaches/incidents are the fundamental “unwanted outcome” that we are trying to prevent. It is not uncommon for techrisk pros to focus efforts on software quality, control weaknesses, or compliance violations – all useful intentions to the extent that they address the aforementioned incidents.

- That techrisk professionals can identify attacks even when the vulnerability is unknown. Much of our profession’s focus revolves around the notion that we must find vulnerabilities in order to protect ourselves, yet time and again we succeed in identifying these types of attacks using behavioral analysis and other techniques. With the growth in popularity of forensic archiving, we can now also determine to what extent we have been victims in the past to assist with understanding the risks of the future.

• That much of the profession’s effort associated with vulnerability management is ineffective. Our efforts to identify each vulnerability prior to exploit are simply overwhelmed by scale and can simply be shown through a thought exercise – consider how many vulnerabilities are created every day (in the aggregate) as compared with how many are found. Perhaps more importantly, it is worth noting that the vast majority of vulnerabilities that are found are never known to be actively exploited [pdf].
• That there is a variance in how different types of attacks – namely, targeted vs. opportunistic – manifest themselves online. Google’s primary cited reason for its new policy involves political activists as victims of targeted attacks that may lead to physical harm. The history of infosec and techrisk highlight other scenarios – the NIMDA worm, WMF exploit, WebDAV, etc – that involve opportunistic exploits across a multitude of targets.
• That the most significant way to “move the marker” in security is through the identification of exploits and not vulnerabilities. As with Code Red and Nimda in the Fall of 2001 leading to Bill Gates’ well-known “Trustworthy Computing Memo,” active exploits are the best drivers of change in the techrisk profession.

 While Google’s new policy offers and opportunity to assess the state of security on the Internet overall, it also demonstrates significant deficiencies in its approach:

• The 7-day deadline has no risk basis. With the significant variance in number of affected parties and speed of compromise associated with opportunistic attacks versus targeted ones, the number is an arbitrary one. In the primary example cited (activists at risk of physical harm), speed is highly unlikely to have a significant impact on risk reduction.
• The capabilities of enterprises and/or users to protect themselves can vary significantly. There are many reasons why some parties choose to remain vulnerable to certain types of attacks – system complexities, legacy support needs, lack of technical skill, competitive priorities, etc. Through the years some security researchers (including some employees of Google) have expressed disdain for those that cannot protect themselves. A company the size of Google should be held to a higher standard in its willingness to protect those online that can’t always protect themselves.
•  No consideration of economics. The policy completely ignores tradeoffs like the risk of breaking systems when taking precautionary measures (e.g. patch failures), the well-known increase in exploits that occur after the disclosure of many new vulnerabilities [Arbaugh, McHugh, 2000 pdf; Bilge, Dumitras 2013 pdf], and the opportunity costs associated with new requirements. When Google says, for example, “each day an actively exploited vulnerability remains undisclosed to the public and unpatched, more computers will be compromised” they neglect the significant likelihood that computers will be compromised regardless of the state of disclosure to the public and fall back on the age-old myth that only patches can protect systems.
•  It can lead to even more exploitations and incidents. Anyone paying close attention to the vulnerability research community knows that there is wide variance in how researchers disclose their information and some decisions are made based on annoyance, frustration, spite and sometimes even malice. If a vulnerability will get “noticed” more quickly, researchers may be tempted to “test” it in the wild in order to increase its priority level.

A company with the talent and resources at Google can do better. Here are some opportunities for improving the state of security on the Internet and addressing the real, significant risk associated with actively exploited 0days:

• Encourage and train political activists in obfuscation and evasion techniques. It is challenging to discuss a blanket policy across all scenarios simply by highlighting arguably the most important one – that involving physical harm. It seems highly unlikely that this case is a common one and the best way to discuss the overall implications of the policy itself is to remove this scenario from the discussion as it tends to cause an emotional reaction. As many of us know, there are many ways political activists can protect themselves online that would be much more effective than a 7-day disclosure policy which comes after they have been compromised.
•  Increase focus on actively exploited 0days. Since these are the most important scenarios the techrisk profession has to deal with, Google should be making every effort to identify these exploits and employ or invent new ways to protect against them. Google researchers still participate in random, ineffective vulnerability research that simply distracts from this very real problem.
•  Provide more insight into the “dozens” of 0days identified “through the years” that was mentioned in the blog announcement. If there is one thing Google has, it is great data. As evidenced by past reports [Provos, 2008 pdf], Google could very easily provide more specific evidence on the number of 0days they have identified, the volume of exploits, and their disposition by vendors. The fact that they haven’t yet, especially in the face of this policy announcement, is disappointing and makes it difficult to evaluate the measure.
•  Take a risk-based approach to disclosure. Fast-moving worms do most of their damage in hours and days – in those cases, seven days is too long. Targeted attacks are unlikely to get repeated in a way that demands immediate attention for most environments – in those cases, seven days is too short. A risk-based approach would take into account the frequency of exploit, probability of future exploit within a target population, and impact of the exploit while evaluating the changes to these variables over time – in particular before and after disclosure.
•  Monitor the situation closely. Google’s unique ability to gather data in this regard is worth mentioning again as a function of its ability to assess its own policy. Collecting and publishing data on actual 0days throughout their exploit lifecycle would be a boon to the entire profession.
•  Initiate or participate in discussions to create new ways to address this very real problem. Commercial, community, and government mechanisms already exist for sharing data publicly and privately that could be used as models for minimizing the risks associated with these types of attacks. For example, a (private) process similar to federal wiretap capabilities in secrecy and opportunity may be more effective in addressing targeted attacks. There are countless other approaches that could be leveraged to address this problem.

Make no mistake, the Google 7-day policy announcement sheds light on a real and significant issue in technology-related risk. While it highlights some of the challenges techrisk professionals face on a daily basis, it also demonstrates significant deficiencies in its approach to address the problem. This is a great opportunity to evaluate the existing state of the Internet from a risk and security perspective to determine where inconsistencies or weaknesses lay and map out a risk-based program that has the highest likelihood of success.

“If you want a bug fixed quickly, sell it on the Russian black market. It’ll be so heavily abused that the vendor will patch out of cycle.”

Now, it could be the joke’s on me and the 126 people who retweeted this message (a large number for security tweets) were in on it. Or, they all don’t realize how ludicrous this is. In the infosec/techrisk field, this kind of thinking is not unheard of so I will treat this as if it is legitimate.

The tweet highlights just how biased people can be when they get caught up in a notion without understanding the implications. Apparently, this tweeter wants bugs fixed quickly. At first blush this seems like a simple enough concern, shared by many. But peel one small layer deeper and the statement often ends up being “want bugs that you know about (or worse, that you discovered) fixed quickly after your discovery?” It becomes easier to see how certainty bias and the focusing illusion come into play.

there is plenty of evidence to demonstrate that it is unlikely that the bug in question is the only bug that remains unfixed – we have any number of bugs in various stages of discovery and disclosure all the time. If we assume that the average bug takes 120 days from discovery (or at least vendor notification) to patch release, and vendors generally release patches on a monthly cycle, then there are four months of undisclosed (typically) vulns on your systems that remain upatched.

Now, you might assert that this makes the point – of course we want them patched “quickly.” But that completely ignores the tradeoffs. If your patch is prioritized, that means another one must be de-prioritized. I suppose you could say that security developers aren’t operating at capacity and therefore can absorb the workload for both bugs, but that seems farfetched to me and doesn’t scale in any case.

Of course, the worst part of the tweet is the part that purposely increases risk by increasing the threat of compromise. No need for a soapbox/high horse here to recognize that purposely inflating risk to get attention in spite of how detrimental it is to Internet users is certainly unprofessional and really kind of pathetic.

Too often, folks get caught up in some perceived solution to a problem and neglect the bigger picture. Many times, the bugfinder is sincerely concerned. But it is important to understand the cost/benefit and risk dynamics involved if you really want to positively affect Internet risk.

When estimating losses, it isn’t entirely unreasonable to do the high-level straight-line math like IT World did here:

“Amazon.com’s latest earnings report showed that the company makes about $10.8 billion per quarter, or about $118 million per day and $4.9 million per hour.”

It’s really quick and dirty – and in a general sense legitimate – but can we do better? There are other ways to look at this that might shed some light on impact assessment. First, the assessment above makes no mention of costs. That might be the biggest weakness since costs are more under the control of Amazon and (probably) don’t fluctuate as much as revenue.

Luckily for us, Amazon just released its quarterly earnings report and this report asserts that its operating margin is about 3%. So right off the bat, we could suggest that Amazon lost 97% of $5 million or $4.85 million in costs. A more conservative estimate might try to determine whether the costs were unrecoverable or not, etc. Hopefully, you get the idea. A cost-oriented approach also works well as an example in infosec since that is often a big piece of the losses we face.

It is important to note here that these costs are additive to the lost revenue estimate – not only did we lose the $4.85 million in operating costs, but also (presumably) we lost that initial $4.9 million in revenue, for a total of (let’s say) $10 million.

Now, let’s look again at that lost revenue estimate. As mentioned earlier, coarse numbers like those used in the calculation above are certainly justifiable but we can probably do better. A quick thought exercise can help here – by creating the experience of an “average customer” of Amazon’s we can better assess the impact of the outage. This is harder than it sounds because we’ll have to second guess our own biases, but let’s try anyway. Let’s call him Joe.

Given that the outage was simply a “denial-of-service” of sorts, the big variable we must evaluate is time. More specifically for our scenario, we need to answer the question “How timely does Joe’s interaction with Amazon need to be, or, how likely is Joe to wait an hour to complete his purchase?” At the very least, we know Joe is willing to wait two days (maybe more – not sure what the average delivery time is for Amazon) to receive whatever goods he purchases. Throw in what we might assume (my bias) about Amazon’s low prices and the corresponding brand loyalty that comes with it and it seems reasonable to conclude that Joe will wait an hour to make the purchase, and therefore the lost revenue is actually only deferred revenue to be recognized in the future.

But not everyone is average (usually nobody is), and so once we cover a generic case, it is useful to consider the impact of the outliers. Now, we can imagine scenarios where even though a customer can wait for delivery, she can’t wait to place the order – too many other things going on in life. Or even a case where the customer would actually lose a full day due to delivery cutoff times. These are the types of cases that warrant more attention. Certainly it is reasonable to factor these cases into a loss scenario. Let’s say this is true 10% of the time.

The goal here is to be conservative in our estimates (even though it is sometimes beneficial for companies to be liberal after the fact – can hide other problems) so we should remember that these scenarios are typically useful in identifying some sort of discount factor to apply to the initial $5 million estimate. Though it is possible to come up with scenarios where there is a multiplier – maybe holiday seasons – it is less common.

Our lost revenue evaluation has led us to conclude that 90% of purchases will still be made in the future, so the remaining 10% of cases will discount our $5 million loss down to $500,000. Add that to our lost costs and we are back to the initial $5 million estimate, though from a different perspective. While it might be attractive to decide all was for nought, it is worth considering the situations where the costs are much lower, or the revenue is more likely to be lost to see the value in the exercise.

Now, some might suggest (essentially) that the above analysis is really not worth it because a loss is a loss. Not only that, but Amazon’sown numbers have shown (?) that there is no discernible uptick in sales in the period following the outage. As mentioned earlier, it is easier to see how costs are fairly static and therefore turn into losses. On the revenue side, however, it is not clear at all.

In assessing lost revenue in this case, one must do two things: first distinguish between necessity and convenience and second evaluate the impact of buyer’s capacity. The purported lack of a noticeable uptick in sales in the short term could easily be explained if purchases are more oriented around convenience than necessity. Measures associated with shopping carts might be of assistance here (I sometimes leave items in my shopping cart for days if not weeks). Again, this information can be factored into the estimates if need be.

It is uncommon to consider a “buyer’s capacity” but especially with convenience purchases, one might decide that the rate of purchase is a determining factor and even though the shopper returns, she will be buying other items, etc. This justification is easier to believe in cases where capacity is high – that is, the shopper is buying at a rate where fitting in the “lost” purchases is unlikely (and when it happens is noticeable in the numbers). My assessment is that this scenario is unlikely; people are more casual in their shopping experience and will therefore wait to make their purchases. (A similar capacity limit could have an effect on the Amazon side, but that is even more farfetched).

My conclusion is that $5 million is a reasonable loss estimate for Amazon’s outage, but not for the reasons initially believed.

So, for example, using available data they calculated that automatic fire extinguishers in airplane lavatory trash receptacles cost $16,000 per life year saved. (This was in 1993 – maybe smoking was still allowed then?)

Interestingly, these costs ranged from “those that save more resources than they consume to those costing more than 10 billion dollars per year of life saved.” The median cost per life year saved was $42,000. The paper also breaks down amounts by type of intervention, prevention stage, and even provides some data on proposed govt regulations by regulatory agency (FAA median $23,000; EPA median $7,600,000).

As a quick aside, the existence of this data helps one understand that even though circumstances where “success means nothing happened” (in this case, death didn’t happen), there is still plenty of opportunity to assess the benefit of some particular intervention.

These types of “revealed preference” study results can be eye-opening to those that suggest we should spend “whatever it takes” to address some particular concern. In looking at the large variance in costs, perhaps that isn’t the best course of action. It is nice to think we have unlimited resources, but at some point they run out. When they do, not only does that impact overall effectiveness, but opportunity costs come into play.

What does this mean for cybersecurity? Though it is not fair any more to say there is no data available to our profession, it certainly is difficult to leverage the data coming out in ways that are helpful to an organization. However, we can start thinking in terms of estimates and measures that make sense. In particular, we can evaluate and compare costs of various controls to each other and factor in some notion of anticipated risk reduction.

We can learn a lot from studies like these.

The challenge for us when measuring impact of some infosec-related incident is that the systems/assets often keep generating the expected value to an organization. Even DoS events against eCommerce sites – perhaps one of the easiest impacts to measure – should consider loyalty at this stage for people to come back and shop later if a system is out. It is often much more difficult than that – if the formula for Coca-Cola is stolen, it doesn’t impact Coke’s ability to make/distribute the drink; it is more likely to have some impact like lost market share due to black-market knockoffs (not sure if this is a problem in the soft drink world). Even more challenging might be the situations where an illegitimate third party can make even more revenue through stolen IP than the victim – did the victim actually lose that?

Creating estimates with problems like this is really challenging so I think we are much better off starting with the revealed preference thresholds that are out there – trying to assess just how much we spend to pursue/defend IP rights through legal means and using that as a baseline, for example. The logic being, again, that if you spend $5m protecting your IP, that is at least as much as you believe you could lose if you didn’t. I’d take this method over the notions of “brand” and “reputation” that get bandied about (for non-human organizations, it *should* all boil down to lost income and/or increased costs – current or future).

Considering thresholds and revealed preferences, security spending is a great baseline number for estimating risk. That is, if our security spending is $5m then we are betting that $5m is offsetting $5m in potential losses, at minimum (advocates of the GLEIS model might even suggest it is something like 39% of potential losses). This line of reasoning is also useful in helping us develop a “control horizon” – we can draw a line on a risk matrix using this spending (aka minimum estimated risk) as the slope. we can also plot the intersection with IT spending, profit, revenue, or other numbers that might be useful in comparison. As we slice up a risk matrix like this, we can determine whether our judgments about risk are holding up based on where they fit as well.

I consider this approach very ALE-like – certainly aggregated and coarse (though it could be applied to individual scenarios as well) and with the caveat that security spending must be carefully calculated. At the level enterprises are working today, I think this information would be very useful in helping folks understand the risk decisions they are making.

“After multivariate adjustment for major lifestyle and dietary risk factors, the pooled hazard ratio (HR) (95% CI) of total mortality for a 1-serving-per-day increase was 1.13 (1.07-1.20) for unprocessed red meat and 1.20 (1.15-1.24) for processed red meat.”

As with many studies about diet, lifestyle, and death, this one has sparked discussion. The Numbers Guy from the Wall Street Journal, Carl Bialik, wrote two articles on the study itself and the difference between absolute risk and relative risk numbers that often create confusion and annoyance. That article led me to the always excellent Understanding Uncertainty blog post by Dr. David Spiegelhalter’s fuller treatment of exactly what a 13% increased risk of death actually means (dying about a year younger, in case you are wondering). It also provides discussion on correlation/causation caveats and the practical application of the numbers.

All this discussion is interesting and should be useful for any IT risk professional interested in quantitative treatments of risk. But these details are not the reason I am writing this. As I was reviewing the information, it struck me just how difficult this is in the physical world. This quote from Dr. Willett in the L.A. Times article really highlights the problem:

“In principle, the ideal study would take 100,000 people and randomly assign some to eating several servings of red meat a day and randomize the others to not consume red meat and then follow them for several decades. But that study, even with any amount of money, in many instances is simply not possible to do.”

What struck me was not only how hard this is, but also the rigor of the results in the face of the described obstacles. And, even more importantly how much easier this would be for IT risk professionals in the virtual world.

In the virtual world, we actually could design and conduct a study that controlled for almost every variable to quantify risk. We could, for example, deploy 10,000 or 100,000 virtual machine clients around the Internet that were all configured exactly alike with the exception of some specified difference – patched vs. non-patched, different anti-malware solutions and/or signature updates, open vs. closed ports, other configuration changes, etc. About the hardest part would be determining how/where to deploy the VMs and coming up with a “honeymonkey” algorithm to mimic user activity.

Perhaps the biggest challenge would be recognizing and characterizing the intelligent adversary contribution to the variance in the numbers – the popularity of vulnerabilities, exploit techniques, 0days, etc. And that would be the good stuff, as well.

Conducting an experiment like this seems so easy to me that I wonder if somebody is already doing it. I am pretty sure some group (ISC?) used to do some sort of “time-to-compromise” metric for unpatched systems. And I suspect there may be others. Does anyone know of experiments/studies being done similar to this? If so, I’d love to hear about them. If not, why not?

For these others, I will do my best to make them:

It turns out the target this time is “SASO,” a company that must be making headway in driving legislation towards third party code reviews.

“ I’ve opined in previous blogs on the importance of defining what problem you want to solve, specifying what “it” is that you want to legislate, understanding costs – especially those pertaining to unintended consequences – and so on.”

-> Hear, hear (or is that here, here?). In any case, we should all be finding ways to understanding exactly what we want. For example, do you simply want “more secure software” or do you really want “fewer incidents?” I am utterly against legislation because we haven’t defined the problem and more importantly we haven’t scoped out the solution.

This includes legislative mandates on suppliers – who, as we all know – [sarcasm] are busy throwing crappy code over the wall with malice aforethought. Those evil suppliers simply cannot be trusted…[/sarcasm]

-> I had to emphasize the sarcasm here because some security folks actually believe this. However, Mary Ann seems to insinuate that simply because developers are not trying to write write crappy code, they aren’t creating it. And there is good reason to believe (ahem) that software ships with vulnerabilities. The code isn’t necessarily crappy, per se, but it is hard to refute the evidence that it does have vulnerabilities – sometimes many.

Though I understand the frustration and strong emotions here, on both sides, I am not a fan of setting this up as some sort of moral, “good vs. evil” argument. In my experience, most folks really are trying to “do the right thing” even though the approaches conflict.

Having to plow through 1000 alleged vulnerabilities to find the three that are legitimate is way too expensive for any company to contemplate doing it.

-> In my opinion, this is the real problem with this entire space. The tools do not provide high quality, which makes security expensive. And I immediately segue into most vulns aren’t exploited and there are no defined bounds to how long you can look for vulnerabilities.

“creating a market for themselves.”

-> Reference to “demand creation,” one of a handful of conflicts of interest in the security world (really, everywhere). Another is the conflict between security and shipping products.

“ they analyze the binaries to do static analysis”

-> I wonder who this could be. Oh, that’s right, it’s “SASO.”

And thus, suppliers are out of business if they screw it up, because their competitors will be ruthless. Competitors are ruthless.

-> This is standard far “it’s not us, it’s them” mentality that is extremely tricky. Vendors think “competitors” are ruthless which implies they are some sort of exception. Enterprises believe “everyone” is ruthless – there are no “competitors” only prospective suppliers. And again, in a very ambiguous space, it is rare to find a software company that doesn’t have something to say about the quality of their security program.

Of course, of much more importance in all of this – and perhaps Oracle can attest to this as well – is the value the software product provides to the company.

Whom do you think is more trustworthy? Who has a greater incentive to do the job right – someone who builds something, or someone who builds FUD around what others build? Did I mention that most large hardware and software companies run their own businesses on their own products so if there’s a problem, they – or rather, we – are the first ones to suffer? Can SASO say that? I thought not.

-> A strawman that seems a bit of a stretch based on the evidence. I agree wholeheartedly that developers really do try to write secure software, and that companies really do try to ship secure products. Unfortunately, in today’s world there is plenty of evidence that it isn’t good enough. And, to be honest, I think the answer is that SASO has more incentive to do the job “right” – it is their core business.

…why SASO will never darken our code base…

-> I can’t help but think of a CFO asserting that external auditors will never “darken” his financial statements… umm, yeah. Moving on, now.

This next section is the “manifesto” part:

1) We have source code and we do our own static analysis.

-> It is very hard not to be trite here with a “and how is that working out for the industry?” line. This is true of every significant software developer. It seems like vulnerabilities are still missed (ahem).

2) Security != testing

-> Agreed! There is much more to it. But vulnerabilities are where the rubber meets the road. The good news is that the corollary to this statement also refutes Mary Ann’s earlier point that she would never outsource “security.” It really isn’t security, it is testing that is (potentially) being outsourced.

3) Precedent… 4) Fixability…

-> I worry a lot about Precedent. Just not in this case. And fixability is simply a truism that is irrelevant as far as I can tell.

5) Equality as public policy.

-> The more vulnerabilities you fix, the more every customer benefits. I don’t see how this is unfair or unequal.

6) Global practices for global markets.

-> Aha! Finally, we get to the real argument, which is that they already use Common Criteria labs to evaluate security, and Oracle believes it is more comprehensive. A much stronger argument, I believe. Buried.

7) All tools are not created equal.

-> Wow. Lots of nuances to this one. I agree that you shouldn’t mandate a tool, or even approach. That runs the risk of ambiguity and leads to the reason why there shouldn’t be legislation in this regard. But that doesn’t mean there is no value to third-party reviews. The biggest value I see is not independence as if developers are colluding in producing bad code, but independence in that another set of eyes can provide new ways to look at the code and, as has been shown by public disclosures (which I generally don’t support), find more vulnerabilities.

[man, this Oracle post goes on forever!]

[A "cautionary tale"] I told the product group that they absolutely, positively, needed in-house security expertise, that “outsourcing testing” would create an “outsourcing security” mentality that is unacceptable.

-> If I had a dollar for every “cautionary tale” I’ve heard, I would be a rich man. The notion that there aren’t a thousand ways to address “mentality” issues is simply wrong.

By way of contrast, consider another company that does static analysis as a service.

-> So, that contrasting story doesn’t really contrast. It seems to indicate that you *can* outsource security testing, if you do it “ethically,” which I am sure everyone would suggest that is how they work, and, as with the other arguments about what is in the best interests of companies, is certainly in the testing companies’ best interests.

I recently heard that SASO has hired a lobbyist. (I did fact check with them and they stated that, while they had hired a lobbyist, they weren’t “seriously pursuing that angle” – yet.)

-> Ugh. Just ugh.

I have to wonder, what are they going to lobby for?

-> A great, important question that really should be answered by the industry.

In my opinion, neither SASO – nor any other requirement for third party security testing – has any place in a supply chain discussion. If the concern is assurance, the answer is to work within existing international assurance standards, not create a new one. Particularly not a new, US-only requirement to “hand a big fat market win by regulatory fiat to any vendor who lobbied for a provision that expanded their markets.” Ka-ching.

-> Though I think Mary Ann is a bit too confident in her in-house setup, I believe this is a reasonable approach and agree more than I disagree with it. And I find myself agreeing with the rest of the post (minus the book recommendations ).

After reading the entire article (!) I find myself missing something. The assertions about how much security there is seems to conflict with the evidence in the public. I am no fan of public disclosures, but that is the way software security world operates today, and to leave the challenges in the entire software world unacknowledged is missing the point.

The only thing worse than the market is government. So I choose the lesser of two evils, almost every time. But it is worth noting that the amount of “demand creation” in the security space is reprehensible as well. Regardless of that, the software security profession as a whole completely ignores

THE MOST IMPORTANT QUESTION IN SOFTWARE TODAY: For any given application, how many vulnerabilities should be tolerated?

If your answer is none, please follow the yellow brick road to the emerald city. We have to get away from working to perfection and set standards as an industry that define a reasonable level of attention to the vulnerable state of software. This reasonability or vuln tolerance measure could be based on effort, code base churn, size, complexity, age, etc.

Obviously, this is a complex problem with many options, perhaps none of which is perfect. But if we don’t want another “compliance” state regarding software, we really need to address this problem with something other than “we try really hard.”

“Last month’s activity has brought to light some opportunities for improvement. We revisited our policies associated with the 4 million blocked connections and determined that approximately 10,000 (.25 percent) should have been allowed and we made a configuration change to address the issue. In addition, the policy associated with the 1695 initially suspected connections were evaluated and changes to our security posture were made that should reduce these false positives by 50 percent. To address the 5 incidents, we have instituted remedial training for the individuals involved and instrumented the affected systems with new means for intrusion detection.”

Read more in my article at CSOonline.com.