I think this question makes an assumption that higher prices lead to better PCI audits (which are supposed to lead to lower likelihood of breach). It is worth keeping in mind that the “unwanted outcome” for PCI is a negative audit that rescinds the ability to process credit cards.

The PCI auditor decision can be framed in the same way we perform any risk assessment – comparing the difference in costs between providers to the anticipated difference in value at risk. So it might be worth it “worth it” to use a low-cost provider if the difference in their costs over another preferred provider is greater than the anticipated increase in risk.

The second story was the Heartland PCI story that took off with Bill Brenner's interview of CEO Robert Carr. My initial take there was that this was another instance where PCI was broken and we should pursue alternative methods of compliance like Andrew Conry-Murray's. In other words, I was making a black and white assumption based on my predisposition to dislike compliance audits.

Then I realized that I was doing the exact same thing I was frustrated at Schneier about – oversimplifying the nature of controls to a binary outcome  of work/don't work.

There is a much bigger problem here than I was exhibiting cofirmation bias and being hypocritical in some respects. That is, we still don't know WHAT WORKS. We don't have a good definition of success nor do we have any evidence or results that demonstrate that controls WORK. This doesn't mean they don't work; it simply means we haven't done a good job proving it. (I had a similar discourse regarding Microsoft's SDL a while back).

My assertion is that locks "work" if (as I described earlier) the risk-adjusted amount of damages (or expected value of losses) is reduced by more than the cost (total cost) of the lock itself. Similarly, PCI "works" if the risk-adjusted amount of damages is reduced by more than the cost of the audit. (Note: In a previous take on whether PCI is working, I focused solely on a risk reduction without accounting for the cost of the control.)

Clearly, my bar is set much lower than complete security, but much more practically when it comes to determining appropriate controls for an enterprise program.

Update: After re-reading my post, I realized I was short on explanation about why the Robert Carr interview showed PCI was broken. I agree with everyone that said PCI isn't about catching that single compromise – that's not the point, the point is "compliance." But when you peel that onion, the real value all of a sudden disappears… compliance really *IS* supposed to reduce risk and perhaps it doesn't… subject to the criteria I laid out in the rest of the post.

In my previous post, I wrote:

We don't have Level 4 Merchant numbers*, which some accounts I've read suggest number in the millions, but it seems unlikely to me that Verizon would be called into shops that small (this is a bigger assumption than I like, but there it is nonetheless). At least we can create a better example with the numbers. So, 17 of the 90 companies (19%) that Verizon worked at claimed to be PCI Compliant. Using the Level 1-3 numbers, that means 17 out of 3,700 PCI compliant companies were compromised, for a success rate of 99.54%.

We can now suggest that PCI is "working" if the success rate for non-PCI-compliant companies is lower than 99.54%. We don't really have a good way for determining a comparable population of companies in this group, but we can find the equivalent population size and infer from there. To have the same success rate, the 73 remaining cases must be part of a comparable group of about 16,000 non-pci-compliant companies.

Here is where it would help to have a better sense for the number of companies at various revenue levels, but I don't have quick access to them, so you have to decide for yourself whether the comparable population is greater than 16,000, in which case PCI is not working, or less than 16,000, in which case PCI may be working.  

Anyone want to offer an opinion?

* There are a lot of caveats here, but this is just a thought exercise anyway.

The data shows that 19% of the breaches were from companies that were PCI-compliant. This seems like a large number and may encourage folks to believe that it provides evidence that PCI doesn't work. The report itself doesn't say this, but it may be inferred from the data.

Once again, we have a base rate problem. The interesting data here would be to compare the ratio of breached/total PCI compliant companies to breached/total non-PCI compliant companies. So, for example, if 19 PCI-compliant companies were breached out of, say 2 million, then that is a pretty good effectiveness ratio. And if we compared it to 81 non-PCI companies out of, say 1 million, then it would be an interesting point in favor of PCI. Of course, since we are working hypotheticals here it is easy to imagine a scenario that is exactly the opposite here.

In any case, the report data does nothing to support nor refute the effectiveness of PCI compliance.

Cialdini's got a new book out called Yes! that discusses 50 persuasion concepts that are backed by empirical research (that's what I like about all of his concepts – they have been demonstrated in the field). There is some repetition with his other books, but these are 3-4 page "chapters" that are very easy to read.

It happens that I was reading the chapter on social norms (or social proof) at the same time as our recent discussion about PCI

What Cialdini et. al. found in a study on household energy conservation was that people increased or decreased their consumption based on the average neighborhood consumption of energy, in a "regression to the mean" style.

This concept of social norm got me thinking about PCI. One of the themes that comes out of compliance vs. security discussions is that compliance is about meeting a minimum standard and people who "really care" about security (whatever that means) would actually do more. I think the principle of social norms is hard at work here, which makes the "goal" of being PCI-compliant the social norm and acts as a deterrent (or creates a 'boomerang effect' according to Cialdini) to folks that want to be more secure. So if making PCI compliance a minimum baseline is an objective of the program, it is likely to backfire and actually make some companies less secure.

There is good news here. The Cialdini study (which, incidentally can be found here) also used incentives – smiley faces on the reports – to reduce this boomerang effect and incent those people who were consuming lower than average to continue their low-consumption ways.

So, what is the PCI "smiley face"? If PCI wants to encourage, but not necessarily regulate, a higher standard, they should come up with a PCI+ program that could be used perhaps for advertising or some financial incentives…

I think this is an option that warrants serious discussion because it focuses on the intended results. This is particularly beneficial when we have no real evidence about what controls are most valuable to reducing risk; we assume quite a bit about "best practices" that may not help.

The idea is not to try to predict in the aggregate how attackers will attack, but simply to let the enterprise determine for itself what its risk level and security posture should be, given the prospects of potential breach.

Andrew doesn't get too much into exactly what the penalty would be if a breach occurs. I think this is fairly simple – organizations already bear the notification costs. We should simply add the requirement to pay for reissuing credit cards and possibly an annual credit monitoring service for the victims.

There are serious cognitive biases at play when everyone gets up in arms about a single incident and condemns a regulation. (Note: I don't necessarily support the regulation, but I certainly don't think that any single incident, or even handful of incidents, is enough evidence to pass judgment.)

PCI is intended to minimize risk given some set of resources. We know risk is non-zero because the attack surface is non-zero and the threat is non-zero. That means we will have incidents over time. And the more organizations that fall under these regulations, the more incidents we'll have.

One way to evaluate the success of PCI is to compare the number of incidents from PCI-certified companies to another set of similar non-PCI-certified companies. We may find out that risk is reduced in some way, or we may find out it doesn't really matter – the risk is the same.

Sure, insurance may come with prerequisites, but I can’t come up with any scenarios strong enough to be a significant driver for security over and above what exists today. Business continuity insurance – companies already have it. Consumer liability insurance – consumed by regulations. Professional insurance – A possibility, but it doesn’t seem like something with enough magnitude. 

(Note: Specialty Insurance Blog weighed in as well with their opinion, which is substantially supporting Perilocity’s. As an aside, I believe their use of the term “risk management” is inaccurate as a replacement for prevention: an organization can manage its risks very closely and yet still be highly risk tolerant and willing to accept them.)