Confirmation Bias at work?

Evan Schuman has an intriguing blog post on the McAfee blog about whether the reduced number of data breach reports at DataLossDB.com are indicative of fewer actual data breaches. His answer is unequivocally “No.” His reasoning is as follows:

  1. Media outlets are less interested in data breaches and therefore not publicizing them as frequently.
  2. Retailers, banks, and hospitals (etc) are getting better at hiding breaches.
  3. Lawyers are getting better at skirting disclosure laws.
  4. Even if the number of breaches are lower, there are (or may be) larger numbers of cards/records being compromised.
  5. Retailers (et.al. I assume) don’t know about some breaches and therefore can’t report them.

I have a hard time understanding how Schuman can be so sure when he offers essentially zero evidence for his assertions. The approach seems ripe for confirmation bias.

Strangely, Schuman then has this to say:

All of those excuses aside, we can now get to the core issue. Are retailers better protected today? Have they learned their lesson and are they actually running more secure networks than before? The answer to that question, as narrowly phrased, is “Yes, absolutely.” But it’s more along the lines of “Retailers were two percent effective against data thieves before and today they’re nine percent effective.” They’re certainly more secure than they were, but they aren’t even close to being sufficiently secure.

This is confusing to me. If I am reading him correctly, Schuman devotes half of his blog making assertions that data breaches are not going down, and then shoots it all down with this paragraph, where he asserts (unknowingly, I suppose) the opposite. I don’t know how he can believe breaches are continuing at the same clip if retailers are also getting better at security.

The only “out” I see is to say that although vulnerability levels are lower, they are more than offset by increasing threat levels.

2 comments for “Confirmation Bias at work?

  1. November 7, 2009 at 1:14 pm

    Thanks for the note linking to this story, but I wanted to try and better articulate what that piece was trying to say. As a practical matter, there is no contradiction between the fact that retailers have gotten better at security compared with five years ago (If you remember what things were like with the major retailers about five years ago, it would have been hard for them to have NOT gotten better) and the fact that breaches haven’t sharply reduced.
    Consider a neighborhood burglary ring. In a hypothetical community, five years ago was a time when neighbors left their doors unlocked all day, deadbolts were all-but-nonexistent and people regularly and publicly discussed when they’d be out of the house and for how long. Today, this hypothetical community locks their doors, keeps their mouth shut and uses high-security deadbolts. Is the security in that neighborhood much better? Sure. Can you tell me from that the burglary rate in that neighborhood has dropped? Not at all. There are many high-crime neighborhoods that routinely lock and deadbolt and they still have lots of burglaries.
    What would more likely happen in that community is the thieves would do an ROI calculation. How valuable are the contents of those houses? If they’re storing lots of expensive pharmaceutical products, gold bars, unopened boxes of high-end electronics and trash bags overflowing with millions of dollars’ worth of unmarked bills, the thieves will figure out ways around those deadbolts.
    In retail today, the payment card data–and, to a lesser extent, CRM data–is worth a huge amount to cyber thieves. Therefore, there’s no contradiction between pointing out that retailers are much more secure but that the thieves are having to work harder to get at the stored goodies.
    You also raised this point: “I have a hard time understanding how Schuman can be so sure when he offers essentially zero evidence for his assertions.” There’s nothing especially mysterious about the points raised. Are you disputing any of them?
    Just working off of your summary of the story. I’ll try and explain each element and why I saw no need for further proof (but I’ll be happy to offer more privately, if you’d like).
    1) “Media outlets are less interested in data breaches and therefore not publicizing them as frequently.”
    We scan tons of media outlets every day as we try and track security issues closely. We have simply seen a marked reduction in how often those stories are covered. There’s nothing surprising there. Data breaches were much more newsworthy a year or two ago. Now the typical small breach is a yawner. Searches on Google, Yahoo, Bing and others will make this abundantly clear.
    2) “Retailers, banks, and hospitals (etc) are getting better at hiding breaches.” In talking with retailer and bank IT managers daily, they have learned a lot from the TJX and related breaches and have poured more resources into this. Do you dispute that?
    3) “Lawyers are getting better at skirting disclosure laws.”
    As more state disclosure laws–often contradictory–get passed, companies are understanding what the exemptions are. The “law enforcement is investigating” exemption is probably the most popular. Again, have you seen that lawyers are getting worse at this? The disclosure laws are relatively new and they clearly are getting better as they learn them.
    4) “Even if the number of breaches are lower, there are (or may be) larger numbers of cards/records being compromised.”
    As Visa and other card brands crack down and are getting better at detecting fraudulent activity early, the cyber thieves need to target larger numbers of cards during any one heist. It’s the only way that they can emerge with enough valid names to make a strong profit.
    5) “Retailers (et.al. I assume) don’t know about some breaches and therefore can’t report them.” As we’ve reported many times, the major breaches are generally discovered first by the card brands and the U.S. Secret Service (or a processor) and the retailer is then given a heads up that they’re the common point of purchase. Are you disputing that? Typically, when someone asks for more proof, it’s usually because they disagree with one or more points. Are you?

  2. admin
    November 9, 2009 at 12:00 am

    @Evan -

    The interesting aspect of your post was not that it is possible to derive a scenario where what you say is not contradictory; it is that it would be much easier to succumb to Occam’s Razor if your analysis points you in a different direction.

    Also, whether or not I dispute some of these points (I do, but more on that later), I was more surprised by your level of certainty with limited evidence. I think some of the points would be very difficult to prove in any case.

    Point-by-point:

    1) Your reduced rate for covering the same incidents is interesting but beside the point. I think that any breach of more than, say, 1,000 records would be reported on if the media has knowledge of it. Heck, they report on obituaries and everybody dies. They also report on vulnerabilities and there are many, many more of them than breaches. Yes, I dispute this one.

    I would be pretty impressed if you could show me a letter to a victim about a breach that wasn’t covered. Clearly, this would be difficult to do.

    2) & 3) It is always in the best interests of retailers, etc. and lawyers to downplay breaches. It is also fairly easy to see where you can legally do this and has been for the life of these regulations. If anything, the growth of regs/laws makes it harder in my opinion to hide, not easier.

    4) This point doesn’t refute whether data breaches are on a downward trend; it is a hedge against the possibility that they are. Regardless, it would be pretty simple to say something like “in 2004, the average number of records compromised was x; in 2009 that number is now y,” where x

    5) This point in particular doesn’t seem to jibe with your claim that retailers are getting better at security. The evidence is good that the duration of a compromise before discovery is months. I don’t see why how you can say this is getting worse – which it would have to in order to contribute to your major assertion.

    I am not really asking for “more” proof because no proof was offered to begin with. As I mentioned, my main point was that you had a level of certainty that was hard for me to see and you seemed to ignore the most obvious/simplest explanation in order to fit into some sort of preconceived notions.

    Pete

Comments are closed.