I got a call last week from a newspaper reporter. He was wondering
about the status of the iDefense Vulnerability Contributor Program
(VCP: Here and Here)
and other commercial programs of the same ilk that buy vulnerabilities
from independent researchers. He was wondering if there has been any
attitude change in recent years as the security community has matured.
This is a great question. Let me try to answer it.
Paying for bugs used to be very controversial. When iDefense
pioneered the idea many years ago, the security community was aghast
that somebody might actually pay researchers to find weaknesses in
enterprise-level software.
I am reminded of that great movie, Casablanca when Claude Rains announces to the crowd in Humphrey Bogart's café that he is closing it down for the night.
Bogart: How can you close me up? On what grounds?
Rains: I'm shocked, shocked to find that gambling is going on in here!
[a cashier hands Rains a pile of money]
Cashier: Your winnings, sir.
Rains: [sotto voce] Oh, thank you very much.
Rains: [aloud] Everybody out at once!
That's my favorite scene in the entire movie, but I digress.
Today, the practice of buying vulnerabilities is more accepted.
There are many organizations that participate in this activity today.
Responsible Disclosure -- the act of buying and identifying bugs,
informing the responsible vendor, but keeping the information private
until the vendor fixes the issue -- is not perfect by any means. The
process is frustratingly slow in some cases; however, it is the process
that has stabilized over time and all parties benefit. Researchers
pursue their interests and organizations compensate them for their
efforts. Vendors discover new bugs in their products. Responsible
Disclosure participants provide value to their customers in the form of
early warning until the bug is fixed. Finally, the security community
as a whole benefits because bugs are identified and fixed; maybe not as
fast as we would all like, but fixed all the same. Most in the security
community realize this. There are outliers for sure; those who have
proclaimed that the system is too slow and, out of frustration, hurl
zero-day vulnerabilities into the public without warning. This just
introduces chaos into the system and causes everybody to twist
themselves into knots reacting to whatever the impact might be. I like
to think about Responsible Disclosure in similar ways that I think
about a democracy in action. It is painful and slow and inefficient,
but it is far superior to other more destabilizing options.
But then the reporter hit me with an even harder question. He
asked, "How effective is this method, given that the underground market
has developed to the extent that cyber criminals can offer much better
prices for vulnerabilities?"
Let me just say that I reject this notion. The vulnerability market is
a market like any other. The price of a vulnerability reflects whatever
the market will bear. People buy vulnerabilities for different reasons.
At VeriSign / iDefense, we buy them to give our customers early
warning. Other white hat researchers buy them to enhance their security
products. Black hat or grey hat purchasers have their motivations too.
You may not like their motivations and you might not condone them -- I
certainly do not -- but they exist all the same and are a fact of life.
Ultimately, all purchasers make a decision on the worth of a
vulnerability based on the value that the vulnerability may bring to
support their motivations. Is it likely that some buyers may want to
pay a significant amount of money over what other buyers would pay?
Sure. Does that mean that these other buyers have no role? Absolutely
not.
There are many kinds of security researchers: white hat, grey hat and
black hat. They decide which kind of purchasers they want to deal with
and what kind of researchers they want to be. White hat researchers
find bugs in enterprise-level software, help the Internet community
become a little safer and get paid for their efforts. Grey hat and
black hat researchers, in some cases, give their research up to
nefarious organizations whose purpose they may not fully understand.
They might get paid handsomely for their efforts but they are
definitely not making the Internet safer. Unfortunately, vendors do not
fix those kinds of vulnerabilities until the damage is done. The bad
guys who leverage that research may use it to commit cyber crimes, to
conduct cyber espionage or to execute some other nefarious purpose. The
security community does not get to fix that vulnerability until
somebody discovers it and there is no guarantee that we will discover
it either.
As an aside, don't think that a grey hat selling that kind of
research is free from criminal charges either. I have heard some
researchers say that they are just building tools. They are not
culpable since they do not actually pull the trigger to do anything
illegal. At least in the US, this is just not true. Consider the
Gonzalez-TJX case if you don't believe me. Authorities sentenced
Stephen Watt -- the guy that wrote the sniffer software called "Blah
Blah" that the Gonzalez team used to sniff credit numbers from
high-profile retail stores -- to two and a half years in prison for his
efforts even though he was not part of the Gonzalez team that actually
conducted the operation. Two and a half years! That is not
insignificant.
But I digress.
I fundamentally disagree with the notion that since the black hats and
the grey hats can possibly make more money selling vulnerabilities,
then the white hats have no role to play. Clearly that is not true. The
iDefense VCP program is alive and well. It produced more than 50
vulnerabilities last year; at least 30 of which were high-impact
Microsoft vulnerabilities. Other white hat operations had similar
successes.
I believe it is clear that white hat vulnerability research is
here to stay. Now if I can just find that cashier and get my winnings.
Sean
Larsson, one of our Advanced Research Lab (ARL) analysts, and I traveled to
Seattle last week to participate in Microsoft's 9th annual Blue Hat conference;
a cornucopia of outside grey hat security researchers and inside Microsoft
developers all gathering to talk about the security issues of the day. The
Microsoft Office product guys specifically asked Sean to attend because they
have been amazed this past year at the number of high quality Microsoft Office
vulnerabilities he has shared with them. To be truthful, they were a little
perplexed that one guy could be so prolific compared to an entire Microsoft
team dedicated to the same purpose.
The Microsoft system is pretty impressive too. They have built this
distributing fuzzing framework that works sort of like the SETI@Home
(Search for Extraterrestrial Intelligence) project. For those not familiar with
fuzzing or the SETI project, let me elucidate.
According to the Open Web Application Security Project (OWASP),
"Fuzzing is a Black Box software testing technique, which basically consists in
finding implementation bugs using malformed/semi-malformed data injection in an
automated fashion." This technique has been around for years and is one of the
primary methods used by the iDefense ARL team. Three issues with this technique
make it a lot easier to say then to do. First, you have to write a fuzzer for
each application that you want to test. This is not trivial. Second, you have to
run it over and over again with crazy input data looking for the application to
choke (Crash). By over and over again, I mean thousands of times if not
hundreds of thousands of times. Finally, after all of this is done, the real
work begins for the security researcher. He must examine each crash location
within the code to see if there is a way he can leverage the problem that
caused the crash into a consistent and reliable exploit.
Microsoft is using the SETI project idea to help with the second issue. The
problem that the SETI researchers have is that they collect all of this data
from Radio Telescope observations and do not have enough computing power to
process it. Their solution was genius: convince people interested in the
research to use their own computers to chew on the data during downtime; i.e.,
when the computer goes into screen saver mode. They built a distributed
data-crunching network that works when the computer is idle. This is
essentially what Microsoft did for fuzzing.
Microsoft has hundreds of active products. Each development team develops their
own fuzzers for the code they are trying to test. The process management for
dealing with all of the bugs found was a mess. There was no consistency in
reporting, no standard method for collecting information and no coordination
between the teams about any progress made. Enter the Microsoft Distributed
Fuzzing Framework.
The Framework has two cool features. First, it is a web based system that
allows any developer to drop a fuzzer into the framework from any location
within the Microsoft network. Second, the framework keeps track of any
Microsoft computers that have joined the framework to participate in after-hour
fuzzing duties. These computers can come from anywhere: secretary desktops, developer
machines, lab machines, anywhere. All crashes found during fuzzing activity are
reported back to the framework and ultimately to the responsible developer for
action. Since the program started this year, Microsoft has found and fixed
millions of bugs. To be clear, these are not all security issues. In fact, most
of them are not. They are simply bugs in the code. In addition, to be fair,
there is a good chance that many of these bugs would never have seen the light
of day through normal use of the product because they were found with crazy
combinations of input data. But that is exactly where the security researcher
wants to travel; code locations that nobody knows about or understands that he
can leverage to run his exploitation code.
All of this brings us back to Sean. iDefense has discovered 12 Microsoft Office
vulnerabilities in the last year; most of these were Sean's work and most were
high quality discoveries with working "proof of concept" exploit code attached.
The Microsoft product team wanted to talk to Sean at Bluehat because, even with
the millions of bugs fixed in various Microsoft products this year with their
really cool and geeky distributing fuzzing framework, none came close to what
Sean found. They wanted to pick his brain about why he has been so successful.
So, Sean spent an afternoon with the Microsoft Office development team to help
them in their endeavors. After the session, Sean was nonplussed. He said that
the Microsoft team consisted of really smart and capable people, but they are,
for the most part, developers and not security researchers. From my
perspective, they don't have that special "X-Factor" that goes part and parcel
with being a white hat bug hunter; that instinct to go for the kill; that
deviousness to understand where one path is bloody brilliant where another is a
hopeless dead end.
Bluehat was a good conference and the Distributed Fuzzing Framework is a really
cool idea, but since Sean is still out-producing that automatic analysis,
perhaps we should point him in the direction of the SETI project. With his
go-for-the-kill X-Factor, we should find our first signs of Extraterrestrial
Intelligence before the new year.
According
to Guillermo Segovia, a member of my Vulnerability Aggregation Team (VAT), we
went through the "Mother of All Vulnerability Announcements" last
week. Microsoft released 34 patches and Adobe released 28. Oracle was scheduled
to release too, but pushed it to this week because of a conference they were
hosting. But, if Oracle had released, that would have increased the number of
patches by 38 for a grand total of --wait a sec, let me take off my shoes and
socks so that I can see my toes and count that high--100 Vulnerabilities. 100
Hundred Vulnerabilities? Is that insane?
Guillermo was pulling his hair out. Not only did he have to help the VAT team
with the heavy load; but also, he had to write the blog for that day (Too Much,
Too Many, All at Once - Oct 14, 2009). Understandably, he was a little snippy.
In his blog post, he railed against the vulnerability gods-- Microsoft, Adobe,
Oracle, the usual suspects -- and wondered why they couldn't cooperate a little
in terms of the release schedule. Did they all have to release 25+ patches on
the same day? Picture Guillermo on top of the Verisign HQs building, black
clouds in the background, lightening flashing across the sky and Guillermo
pumping his fist into the air. Well, it was sort of like that (Actually, he was
sitting at his desk, picking Chipotle out of his teeth and complaining to a VAT
room that wasn't listening to him because they were all too busy; but I like to
pump his image up).
From our point of view during Patch Tuesday, anything under 10 Vulnerabilities
is a light day. Between 10 and 17 is about normal. Anything above that is a
major effort to get our intelligence out to you guys in a reasonable time.
Actually, announcing the Vulnerabilities is not that hard. The analysis piece
is what takes the time; trying to determine if Microsoft's Exploitability Index
makes any sense or if Adobe's reveal is going to be especially nasty.
I hate to say this but we got what we asked for. The security community has
been saying for years that we want big vendors to get on a schedule so that we
can have some control in our geeky, security lives. Guess what? They did it.
They did what we asked. We even predicted it in last year's trends paper. I
hate it when I get what I asked for. Don't they realize I want them to do what
I mean and not what I say?
Here is my new
wish. Instead of Patch Tuesday, I want a shotgun start; each major vendor takes
a day: Adobe - Monday, MS - Tuesday, CISCO - Wed, Oracle - Thursday. As more
vendors come on board, we could spread this out across an entire month.
Just when I thought we had squeezed
every ounce of efficiency out of the iDefense Vulnerability alerting service,
along comes a question that has no easy answer. This puzzler has to do with our
Vulnerability Contributor Program and the exclusive vulnerabilities that pop
out at the end of that process. The question is this: How do we score an
iDefense exclusive vulnerability using the Common Vulnerability Scoring System
or CVSS?
If you are a vulnerability enthusiast and CVSS geek, you know that you have
several choices in the Temporal Score Metrics box when it comes to adding
knowledge of an Exploit to the score. You can choose one of the following:
Not Defined
Unproven that Exploit Code
Exists
Proof of Concept Code
[Exists]
Functional Exploit Code
Exists
High
For
you kids bouncing in your seats and waving your hands in the air to answer the
question, consider this. Does this choice have to do with publicly known
exploits, meaning that everybody on the Internet knows about them or does it
have to do with exploits that only I know about? That is the royal
"I" in that last sentence meaning all iDefense customers know about it.
Historically, we have annotated exclusives in our alerts as "Proof of
Concept Code [Exists]" or "Functional Exploit Code Exists". In
other words, we have acknowledged the iDefense exploit in the CVSS score.
Understandably, this choice bumps up the CVSS value. One of our customers
raised a red flag last week and said they would prefer us to only use these
values if the vulnerability was known to the general public. You see, they have
built-in processes around the iDefense CVSS score. If it gets above 9.something,
then their escalation processes kick in and they have to do a lot of extra work
quickly to mitigate this new, high-severity threat. They would prefer not to do
that for threats that are not known to the general public.
My razor-like retort to that statement is to remind the customer of the reason
he or she subscribed to the iDefense service in the first place; to get early
warning about threats like these and to mitigate them before they go public. It
sounds to me like the system worked.
But, now that I am home, have had a beer or two, have kicked off my shoes and
reclined in my easy chair, I have the time to be magnanimous.
(I linked "magnanimous" to dictionary.com so that I can come back
later and figure out what the word means.)
You see, we don't mark other security vendor's claims to exclusive
vulnerabilities the same way as our own. If Immunity claims to have a new
exploit, we mark that as "Unproven that Exploit Code Exists" until the
vulnerability gets into the public sphere. Obviously, Immunity must have
something or they would not say anything. So, why count them differently? Well,
because if we have an exclusive, we know we have it. If Immunity has it, we are
not sure; we are sort of sure - in fact mostly sure, but not completely.
All of this nonsense does not really answer the question though.
Deapesh Misra, one of our senior analysts on the Vulnerability Aggregation
Team, said the answer is quite simple. He had to speak slowly and use one
syllable words for me to get it, but he said the answer is in the name of the
scoring system itself. It is the Common Vulnerability Scoring System; emphasis
on the Common. It is not the Rick Howard Gee-Whiz Make-it-up-as-You-Go scoring
system or even the iDefense Exclusive Scoring System. It is the Common
Vulnerability Scoring System. The theory is that if everybody in the world
applied the rules of the system correctly to the same vulnerability, everybody
would calculate the same score. This works most of the time, plus or minus
small discrepancies.
As much as I do not want to admit it, the customer was right. I hate when that
happens. We are changing our process. From now on, we will not consider
iDefense exclusives in the CVSS score unless they are in the public somehow. We
will document the stuffing out of them in our published alert, but the CVSS
score will remain pure.
Now, let me look around for another issue that I can point my razor-like
retorts at. Just remember to speak slowly to me.
I got a call last week from a newspaper reporter. He was wondering
about the status of the iDefense Vulnerability Contributor Program
(VCP: Here and Here)
and other commercial programs of the same ilk that buy vulnerabilities
from independent researchers. He was wondering if there has been any
attitude change in recent years as the security community has matured.
This is a great question. Let me try to answer it.
There are many kinds of security researchers: white hat, grey hat and
black hat. They decide which kind of purchasers they want to deal with
and what kind of researchers they want to be. White hat researchers
find bugs in enterprise-level software, help the Internet community
become a little safer and get paid for their efforts. Grey hat and
black hat researchers, in some cases, give their research up to
nefarious organizations whose purpose they may not fully understand.
They might get paid handsomely for their efforts but they are
definitely not making the Internet safer. Unfortunately, vendors do not
fix those kinds of vulnerabilities until the damage is done. The bad
guys who leverage that research may use it to commit cyber crimes, to
conduct cyber espionage or to execute some other nefarious purpose. The
security community does not get to fix that vulnerability until
somebody discovers it and there is no guarantee that we will discover
it either.
research is free from criminal charges either. I have heard some
researchers say that they are just building tools. They are not
culpable since they do not actually pull the trigger to do anything
illegal. At least in the US, this is just not true. Consider the
Gonzalez-TJX case if you don't believe me. Authorities sentenced
Stephen Watt -- the guy that wrote the sniffer software called "Blah
Blah" that the Gonzalez team used to sniff credit numbers from
high-profile retail stores -- to two and a half years in prison for his
efforts even though he was not part of the Gonzalez team that actually
conducted the operation. Two and a half years! That is not
insignificant. 
