Part of the point is that the attacks will be “random” or perhaps some other controlled distribution. We don’t need to know whether some configuration could be compromised by some specific attack – we can test that easily. What we need to know is how likely is it in the wild – taking into account volume of benign activity, time, perhaps strategic placement at various points throughout the ‘Net, etc..

So, if we have two groups of 5, 50, 500, 5000 systems strategically placed and they have very specific configurations to test for some single variable or set of variables (e.g. aggregated patches), then we can see how long they last under those circumstances. And we can do it over and over, continuously measuring the circumstances.

This is really just a more formalized way of doing what (I think) the Internet Storm Center has been doing by measuring the time to compromise or whatever it is.

We would certainly need to think through the client vs. server and active vs. passive activity of the groups (e.g. honeymonkeys vs. honeypots).

The use case that goes through my mind the most is for enterprises considering an update (think about your patch timing paper) trying to decide when to patch by measuring the threat component of risk. The idea isn’t about possibility, it is about probability.

Pete

BTW, since my VP (Scott Charney) is in the touted shortlists, I’m not sure if I can endorse anyone. He might take it amiss if I endorse him, and likely would if I endorse someone else.

The point of #2 is to deploy systems live on the ‘Net but allow for control groups and test groups. Since with computers you can manipulate single variables (e.g. config settings) you could evaluate the impact of changing configurations using real data – evidence and outcomes.

A smart man once said the following:

“Another way to put this is if you want to improve something, you have to start by measuring it. Let’s start measuring security outcomes, so we can start assessing the processes, errors or hostile acts that lead to those outcomes.”

That is the goal of the lab – to create experiments where we can measure outcomes. (Perhaps not exactly what that smart man intended, but it fits well, IMO).

Btw, am I hired?

Pete

On #2, how do you ensure that it does better than common criteria labs?