[Full-disclosure] [Professional IT Security Providers - Exposed] Redspin, Inc. (C+)

[secreview <secreview@xxxxxxxxxxxx>]

We received 22 requests from different people to perform a review of
Redspin! Their website can be found at http://www.redspin.com. We
haven’t done a review of anyone in quite a while, the last review that
we did was for Pivot Point Security who got an A (we still recommend
them). We apologize for this long delay but we have been very busy
traveling (yes we still have jobs doing consulting work sometimes).

As you can see from the comments that we received in other posts we
have a lot of catch up work to do, but to be honest we are not sure
that we will be able to do it. This review might be our final and last
review depending on how much more travel we have. (We have lives, some
of us have families, and we can’t keep doing this for free even though
we feel that this is a great service).

We did a lot of research on Redspin and we managed to get a copy of two
reports that they did for two different customers. We won’t share those
reports with you because that would be unethical, don’t ask.

Redspin claims that it is a “pure penetration testing firm”. What they
mean by “pure penetration testing” is that they do not resell third
party software or hardware. They also say that “don't find problems on
your network so that [they] can make more money; [their] penetration
testing services reveal vulnerabilities, [that] will help you become
more secure.”

We verified their claim with our own research. Redspin will not try to
sell you software or hardware… but they might try to sell you software
as a service. (see their www.jetmetric.com website).

Redspin takes it a step further and is brutally honest about their
methodology for delivering penetration-testing services. They openly
admit that their services rely on automated vulnerability scanners
(Nessus) and are enhanced by manual testing. In fact, Redspin says that
automated scanners “can miss about 40% of the security risk so they
alone do not adequately assess risk. Furthermore, about half of the
findings from a vulnerability scan are false positives”.

Any security company that relies on automated scanners can weed out
false positives, but doing that doesn’t really increase the depth and
accuracy of testing. A false positive, also known as an error of the
first kind, or a Type I Error, is the rejection of a null hypothesis
when it is in fact true. In more simple terms, this is the error of
observing a difference when in fact there isn’t one. Identifying false
positives is fairly easily done, as it only requires inspecting the
results produced by a scanner.

But what about False Negatives? A False Negative, also called a Type II
Error, or an error of the second kind, is the error of failing to
reject a null hypothesis when it is in fact not true. More simply, a
False Negative is the error of failing to observe a difference when in
truth there is one. So, if an automated vulnerability scanner tests a
vulnerable service (a known vulnerability) but the scanner doesn’t
detect the vulnerability then the vulnerability is excluded from the
report. If this is the case then Redspin’s methodology will break down
because there will be no result in the report for Redspin to manually
test. That vulnerability will fly under the Redspin radar but might not
be missed by a hacker. So how many vulnerabilities does Redspin miss?
It’s a question worth asking.

Redspin does say that “vulnerability scanning is not suitable on its
own as a complete or billable service offering, it does provides some
value in the early reconnaissance phase of a more comprehensive
External Network Security Assessment”. They have a typo in that
sentence, but other than that, they are right. Vulnerability scanning
does have a position in the industry and is a huge time saver,
especially when testing large numbers of systems. Just don’t rely on
one vulnerability scanner like Redspin does, use two or more like the
OSSTMM proposes.

Redspin says “manual analysis is at the heart of all of [their]
assessments which not only gives you confidence that you have a
complete view of your security risk, but provides tailored reporting
and recommendations enabling simple work-arounds and cost-effective
mitigation strategies for most security issues.” Based on our research
Redspin’s “manual analysis” isn’t what we expected it to be. It is not
based on vulnerability research and is strictly based on the inspection
and verification of scanner output.

What we can say is that their “manual analysis” doesn’t produce the
highest quality reports that ever we’ve seen, but it does produce
reports that are higher than average quality. The Redspin reports have
very few, if any, False Positives but will contain more False Negatives
than a report that is centered on solid (vulnerability) research.

One thing that Redspin does that we really don’t like is to ask their
customers to lower their defenses before they do testing. That’s right,
they ask their customers to white list their scanner’s IP addresses so
that the customer’s Intrusion Prevention System doesn’t block the
scanner. We verified this during 3 different interviews on three
different dates. We even talked to one Redspin customer to confirm it,
and they did. We think that a security testing company should be able
to test around a customer Intrusion Prevention System. If they can’t
then that really brings their capabilities into question.

We feel this way because Intrusion Prevention Systems are a part of the
networks defenses and they should be tested. Disabling them for a
security test prevents them from being tested. If they aren’t tested
then how does one know how effective they are? It just doesn’t’ make
sense. On top of that, the test won’t properly reflect the actual
security level of the network being tested.

Something that Redspin claims is that they’ve done is “ground breaking
security research”. We’ve searched high and low for this “ground
breaking security research” but haven’t found it anywhere, so we’re not
sure what they are talking about. When looking at the research page on
their website we see white papers that might make good blog entries,
but we don’t see any “ground breaking security research”.

When we’re told that a company does “ground breaking security research”
we expect to see things like them finding security bugs in critical
systems, or publishing professional security advisories, and maybe even
publishing proof of concept code. Redspin doesn’t do any of that. The
only thing that we were able to find was an “Ultr@ VNC 1.0.1 viwer PoC”
(and what’s the point of that?).

In conclusion, Redspin’s services are slightly better than average.
Their manual testing isn’t true manual testing at all; it’s the
inspection of output from scanners and the elimination of false
positives. We don’t like the fact that Redspin asks its customers to
disable their IPS before being tested, and Redspin doesn’t seem to have
any Vulnerability Research capability.

Its not all bad, Redspin is very honest about their methodology, they
are focused on quality, and they are passionate about what they do.
We’d recommend Redspin to people with testing requirements that do not
require extreme depth and that can afford some False Negatives. By no
means is Redspin a company that we’d suggest you stay clear of, but
they’re certainly not the best in the industry.

As normal, if there are any issues with this review and its
truthfulness please let us know and please provide proof. We will make
changes if we need to and we strive to be as honest and fair as we can
be. Thanks for reading!

Score Card (Click to Enlarge)




--
Posted By secreview to Professional IT Security Providers - Exposed at
8/10/2009 08:51:00 PM

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/