The cool thing about Mary Ann Davidson is she doesn’t mince her words; you know where she stands on every issue and she is willing to own it in the security world. So when I started hearing some buzz about her latest blog post – Those Who Can’t Do, Audit – I expected some sizzle. And got it.

It turns out the target this time is “SASO,” a company that must be making headway in driving legislation towards third party code reviews.

“ I’ve opined in previous blogs on the importance of defining what problem you want to solve, specifying what “it” is that you want to legislate, understanding costs – especially those pertaining to unintended consequences – and so on.”

-> Hear, hear (or is that here, here?). In any case, we should all be finding ways to understanding exactly what we want. For example, do you simply want “more secure software” or do you really want “fewer incidents?” I am utterly against legislation because we haven’t defined the problem and more importantly we haven’t scoped out the solution.

This includes legislative mandates on suppliers – who, as we all know – [sarcasm] are busy throwing crappy code over the wall with malice aforethought. Those evil suppliers simply cannot be trusted…[/sarcasm]

-> I had to emphasize the sarcasm here because some security folks actually believe this. However, Mary Ann seems to insinuate that simply because developers are not trying to write write crappy code, they aren’t creating it. And there is good reason to believe (ahem) that software ships with vulnerabilities. The code isn’t necessarily crappy, per se, but it is hard to refute the evidence that it does have vulnerabilities – sometimes many.

Though I understand the frustration and strong emotions here, on both sides, I am not a fan of setting this up as some sort of moral, “good vs. evil” argument. In my experience, most folks really are trying to “do the right thing” even though the approaches conflict.

Having to plow through 1000 alleged vulnerabilities to find the three that are legitimate is way too expensive for any company to contemplate doing it.

-> In my opinion, this is the real problem with this entire space. The tools do not provide high quality, which makes security expensive. And I immediately segue into most vulns aren’t exploited and there are no defined bounds to how long you can look for vulnerabilities.

“creating a market for themselves.”

-> Reference to “demand creation,” one of a handful of conflicts of interest in the security world (really, everywhere). Another is the conflict between security and shipping products.

“ they analyze the binaries to do static analysis”

-> I wonder who this could be. Oh, that’s right, it’s “SASO.”

And thus, suppliers are out of business if they screw it up, because their competitors will be ruthless. Competitors are ruthless.

-> This is standard far “it’s not us, it’s them” mentality that is extremely tricky. Vendors think “competitors” are ruthless which implies they are some sort of exception. Enterprises believe “everyone” is ruthless – there are no “competitors” only prospective suppliers. And again, in a very ambiguous space, it is rare to find a software company that doesn’t have something to say about the quality of their security program.

Of course, of much more importance in all of this – and perhaps Oracle can attest to this as well – is the value the software product provides to the company.

Whom do you think is more trustworthy? Who has a greater incentive to do the job right – someone who builds something, or someone who builds FUD around what others build? Did I mention that most large hardware and software companies run their own businesses on their own products so if there’s a problem, they – or rather, we – are the first ones to suffer? Can SASO say that? I thought not.

-> A strawman that seems a bit of a stretch based on the evidence. I agree wholeheartedly that developers really do try to write secure software, and that companies really do try to ship secure products. Unfortunately, in today’s world there is plenty of evidence that it isn’t good enough. And, to be honest, I think the answer is that SASO has more incentive to do the job “right” – it is their core business.

…why SASO will never darken our code base…

-> I can’t help but think of a CFO asserting that external auditors will never “darken” his financial statements… umm, yeah. Moving on, now.

This next section is the “manifesto” part:

1) We have source code and we do our own static analysis.

-> It is very hard not to be trite here with a “and how is that working out for the industry?” line. This is true of every significant software developer. It seems like vulnerabilities are still missed (ahem).

2) Security != testing

-> Agreed! There is much more to it. But vulnerabilities are where the rubber meets the road. The good news is that the corollary to this statement also refutes Mary Ann’s earlier point that she would never outsource “security.” It really isn’t security, it is testing that is (potentially) being outsourced.

3) Precedent… 4) Fixability…

-> I worry a lot about Precedent. Just not in this case. And fixability is simply a truism that is irrelevant as far as I can tell.

5) Equality as public policy.

-> The more vulnerabilities you fix, the more every customer benefits. I don’t see how this is unfair or unequal.

6) Global practices for global markets.

-> Aha! Finally, we get to the real argument, which is that they already use Common Criteria labs to evaluate security, and Oracle believes it is more comprehensive. A much stronger argument, I believe. Buried.

7) All tools are not created equal.

-> Wow. Lots of nuances to this one. I agree that you shouldn’t mandate a tool, or even approach. That runs the risk of ambiguity and leads to the reason why there shouldn’t be legislation in this regard. But that doesn’t mean there is no value to third-party reviews. The biggest value I see is not independence as if developers are colluding in producing bad code, but independence in that another set of eyes can provide new ways to look at the code and, as has been shown by public disclosures (which I generally don’t support), find more vulnerabilities.

[man, this Oracle post goes on forever!]

[A "cautionary tale"] I told the product group that they absolutely, positively, needed in-house security expertise, that “outsourcing testing” would create an “outsourcing security” mentality that is unacceptable.

-> If I had a dollar for every “cautionary tale” I’ve heard, I would be a rich man. The notion that there aren’t a thousand ways to address “mentality” issues is simply wrong.

By way of contrast, consider another company that does static analysis as a service.

-> So, that contrasting story doesn’t really contrast. It seems to indicate that you *can* outsource security testing, if you do it “ethically,” which I am sure everyone would suggest that is how they work, and, as with the other arguments about what is in the best interests of companies, is certainly in the testing companies’ best interests.

I recently heard that SASO has hired a lobbyist. (I did fact check with them and they stated that, while they had hired a lobbyist, they weren’t “seriously pursuing that angle” – yet.)

-> Ugh. Just ugh.

I have to wonder, what are they going to lobby for?

-> A great, important question that really should be answered by the industry.

In my opinion, neither SASO – nor any other requirement for third party security testing – has any place in a supply chain discussion. If the concern is assurance, the answer is to work within existing international assurance standards, not create a new one. Particularly not a new, US-only requirement to “hand a big fat market win by regulatory fiat to any vendor who lobbied for a provision that expanded their markets.” Ka-ching.

-> Though I think Mary Ann is a bit too confident in her in-house setup, I believe this is a reasonable approach and agree more than I disagree with it. And I find myself agreeing with the rest of the post (minus the book recommendations ).

After reading the entire article (!) I find myself missing something. The assertions about how much security there is seems to conflict with the evidence in the public. I am no fan of public disclosures, but that is the way software security world operates today, and to leave the challenges in the entire software world unacknowledged is missing the point.

The only thing worse than the market is government. So I choose the lesser of two evils, almost every time. But it is worth noting that the amount of “demand creation” in the security space is reprehensible as well. Regardless of that, the software security profession as a whole completely ignores

THE MOST IMPORTANT QUESTION IN SOFTWARE TODAY: For any given application, how many vulnerabilities should be tolerated?

If your answer is none, please follow the yellow brick road to the emerald city. We have to get away from working to perfection and set standards as an industry that define a reasonable level of attention to the vulnerable state of software. This reasonability or vuln tolerance measure could be based on effort, code base churn, size, complexity, age, etc.

Obviously, this is a complex problem with many options, perhaps none of which is perfect. But if we don’t want another “compliance” state regarding software, we really need to address this problem with something other than “we try really hard.”