[Pete Lindstrom is VP of Research at Spire Security, LLC and host of the AMP Firehose 1-day Workshop (vendor bakeoff) coming up in Chicago on 10/29. Register at www.regonline.com/AMPFirehoseCHI.]

I believe the folks at Gartner put a lot of research and effort into their Magic Quadrant analysis. That said, I can’t help but conclude that “vision” and “execution” don’t quite do it for me when it comes to identifying appropriate candidate solutions to address a problem. They just seem to be too much about marketing, which is very important to the companies but only ancillary to an enterprise’s needs. Sure, they want a solution that will be viable for the long-term, but other than that it is pretty insignificant.

To address this issue, I have put together a set of questions in 4+1 evaluation categories that I believe provide more insight into the important attributes of a solution. The first round of categories was introduced at AMP NYC a month ago. Here is my second revision. Opinions and advice are welcome.

1. Company/Product Information: What level of confidence does the company information provide that the company and product will remain viable for your organization?

Consider:
• What year was the company founded?
• What is the background of the management team?
• How many employees does the company have?
• What is the funding status/source of finances?
• What is the product name and version?
• How many customers does the company have for the pertinent product?
• What certifications and tests were done on the product?
• What other 3rd party reviews, awards, or other supporting evidence exists about the product?
• What is the pricing model for the solution?

2. Functional Operation: What level of benefit does the functional operation of the product have?

Consider:
• Primary operation – scan memory state, scan configuration/file system/network state, monitor/record system call activity, monitor/record network traffic, isolate memory, isolate system activity, isolate network communications.
• Trigger action – detect “known good” execution, detect “known good” activity, detect “known bad” execution, detect “known bad” behavior, detect anomalous execution, detect anomalous behavior.
• Response options – allow, deny execution, kill process, kill network connection, reroute network communication, log event, notify user, notify admin (alert), other.
• Recovery options (post-infection) – Restore config to known good state, remove bad files/objects, identify similar issues across network, notify/update other control solutions.

3. Architecture & Administration: How well does the product’s architecture fit in with your organization’s existing security processes? How likely is it to provide benefits? What features does it have to support implementation and administration?

Consider:
• Where/how are any product sensors or agents deployed throughout an enterprise (endpoint, network, cloud, other)? How are they protected?
• Where/how does the product admin/management function work? How is it protected? (endpoint, network, cloud, other)
• Where/how does the product log/data/storage function work? How is it protected? (endpoint, network, cloud, other)
• How is information shared a) with the solution components; and b) with others?
• How does the solution get installed/implemented in the environment?
• How customizable is the configuration and interface?

4. Technical Integration: How well does the solution integrate into the IT ecosystem? How easy will it be to implement and maintain?

Consider:
• How does the solution integrate with other products from the same company?
• How does the solution integrate with 3rd party security solutions?
• How does the solution integrate into an IT architecture?
• What are the prerequisites for user directories, management servers, etc?
• What standards, communication protocols, platforms, languages, frameworks, etc. are supported?
• How robust is the API for third party access?

The final category is actually a rollup of the other four, since the differentiators and value come from the previous specifics being identified.

Key Differentiators / Overall Value Proposition
When looking at the complete picture of the solution, how strong are the overall benefits derived from the individual evaluation categories?

I believe these evaluation categories more properly reflect the needs of the enterprise. What do you think?