Home > Security > This Was Not What I Had In Mind

This Was Not What I Had In Mind

So, after I completely whiffed on the initial disclosure of the DNS flaw, I wrote the following:

So there’s been some skepticism about the DNS flaw. I want to be clear: It was richly deserved. A “put up or shut up” mentality is critical to the survival of our industry. It’s just too easy to make stuff up, if you can just wave away detractors with “I can’t prove it…it’d be UNSAFE.”

The danger from that statement is very tempting and very real. Our credibility as an industry — ultimately, our ability to get bugs fixed — depends on that statement being called out as the bullsh*t that it is.

We, as an industry, have gone back and forth on full disclosure (i.e. tell everyone) vs. responsible disclosure (i.e. tell everyone after the vendor has had a chance to fix things). Partial disclosure has always been looked down upon, rightfully so, because it’s so amazingly easy to abuse. But if our goal is to protect customers, and one particular bug will affect almost all of them, and a phased disclosure of information will protect the greatest number of customers possible — then perhaps there’s a place for this mode.

It’s certainly not a path you can safely decide to take by yourself, however. That’s what I did, when I refused to tell anyone else in the security industry what the bug was. It’s not something I’d ever do again. It’s not just that you can’t vouch for your own bugs. It’s that, without peer review, you don’t know what bugs people are going to think you’re recapitulating, and you even don’t really understand the severity of your issue.

It’s your blood, sweat, and tears in there. Hard to be impartial.

Tie that in with — it’s not just your credibility you’re betting with, when you go out with partial disclosure, but the credibility of every security researcher working to fix problems — and it’s really not something you can do alone.

That’s what I thought was clear. But what’s happened since my Black Hat talk is something really problematic. Partial Disclosures are occurring, and rather than the press saying “Well, we’ll wait until there’s more information to report on this”, they’re saying, “Wow! This must be like the Kaminsky bug! The Internet’s going to die!”

Awww hell no.

Look. Not every bug is Internet killing. I mean, it’s 2008, if you can’t profit on it nobody’s doing it. And killing the Internet is the first thing you’re told not to do in “Being An Internet Parasite 101”. There’s no profit, so it’s not going to happen — well, barring nation-state level extortion, anyway.

So, what’s been going on with partial disclosures?

First, there’s RSnake and Jeremiah Grossman with Clickjacking. Oh, it’s definitely a bug, unless you think any web site should be able to snap photos from that camera in your laptop. It’s not Internet ending, as RSnake has visibly and repeatedly had to explain to people. But it’s something that needs to get fixed. Why is it a partial disclosure? Because the vendor asked — nicely — for a little more time to complete the responsible disclosure process.

I think everyone can agree that it would have been better to have stayed fully dark until a vendor patch was ready, if waiting for a vendor patch was the path to take.  Partial disclosure is not good here, but positive working relationships with vendors are.  At least we have pre-emptive excision from “Internet Killing” class.

Not so with the newest case.  Now, there’s Robert E. Lee and Jack Louis with their TCP Denial of Service attacks. Now, it’s a bit silly to assume Jack Louis doesn’t know the history of TCP attacks, as it’s silly to assume I don’t know the history of DNS attacks. (You’d be amazed how many people thought I’d just reinvented the birthday attack.) Jack’s written more crazy TCP code than you have, for all values of you including me and possibly Fyodor. Do their attacks work, mostly as they’re saying? Almost certainly. There’s dozens of weird corner cases in TCP where resources and timers are allocated. It’s entirely feasible that at least some of them have nasty effects on the system above and beyond three way handshake flood.

We’ll have to see. But while Robert and Jack appear to have just wanted to have talked about them at T2, something has gone wrong.  People are reporting this as a Internet Killing class bug, because the last time someone wanted a couple more weeks, it appeared to actually be one.

Look.  It’s a DoS, from non-spoofable address space.  We’ve operationally been surviving DoS’s for years and years; if we have a more efficient DoS, that correlates to a smaller botnet. Well, we’ve got freakin’ huge botnets out there, and we’re doing OK. Things go down for a little while, and then lots of IP space gets blocked.

Again, this is Jack Louis we’re talking about, so I’d bet any detractor in the world he has some ridiculously cool resource exhaustion attacks to talk about.  He probably knows more about TCP than you do.  (If you don’t know what Time-Wait Assassination is, he definitely does.)  But the “meta-message” that “I’m not telling you everything, because the Internet will come to an end if I do” almost certainly comes from his desire to finish up some of the cooler attacks, colliding with my reticence to talk this summer because I was trying to get people to patch.

That’s not Jack or Robert or RSnake’s fault. I do think it’s my responsibility to clean up, if not for all the researchers of the world, then for the press.

To all the people in the press: Guys, you’re all awesome, but someone’s going to lie to you. They are going to lie to you, because it will make them money. (See above, it’s 2008.) Consider: A partial disclosure only makes sense, operationally, as a call to arms. It is a disclosure that there is a fault, for which action must be taken. If the action to be taken is to spend a ton of money on a new and fabulous gizmo, then some portion of your readership will go ahead and do just that. And indeed, the longer the period of time between partial disclosure and “full” disclosure, the more gizmos will be sold, and the more money will be made.

To the nth degree, this strategy incentivizes fly-by-night shops to make bugs up, and not even disclose them to the vendor, because that would stop the gravy train as the vendor called foul.

So, press, I must apologize to you. I have put you in the situation where either you acquire complete validation of a flaw from a second source — and thus risk getting scooped! — or you get played. Since I’ve done this to you, I’m going to do something about this. I am going to create something of a community council, that will pre-vet (under legal and strict NDA) any bug that someone claims is so very important that it cannot be disclosed to the point of independent reproduction. Members of this council will have to have publicly presented work in the subject area that is under consideration. I’ve spoken to a decent number of people, and everyone is somewhere between very pissed and legitimately afraid of a flood of unjustified partial disclosures.

Faced with an unending stream of “is the Internet dead yet?” Slashdot posts, everyone I’ve spoken to appears fully on board with providing an honest judgement regarding the legitimacy of findings.

Now, I expect we will reject, out of hand, almost all claims. But we will do so, with the full technical argument brought by the finder, rather than presumptions based on old flaws. Attacking the strawmen implied by partial disclosure is a losing scenario for literally everyone involved.

It stops here. Reporters: There will be a more formal process coming, but please mail or call me if anything more of this nature shows up. Hackers: Would you volunteer to enter an NDA with someone, to help publicly assert whether this sort of behavior is legitimate? Mail me too, especially if you wouldn’t trust me vouching for a given bug.

I think as a community, we’re going to need to determine the limited number of scenarios where the benefits from partial disclosure outweigh the risks. Again, partial disclosure only makes sense as a call to arms. What are users supposed to do, if not install a patch or implement some reasonable operational procedures? A partial disclosure is an ask — what are you asking for?

Vendor intransigence is not an excuse. Patches cannot be reliably synthesized in the time between partial disclosure and when someone figures the attack out. If things are so broken that there must be a release, then the release must be compelling enough that the intransigent vendor cannot deny the validity of the findings.  Such a release is not going to be partial!

(Of course, the existence of a council might actually help with occasional intransigence. Hackers — if you’re sitting on a bug you think is Internet threatening, and you’re not getting any progress with the vendor, mail me too.)

There are of course other factors. Effect on infrastructure, which patches very slowly, is one. The severity of the bug, the difficulty of mitigating attacks using existing operational procedures, and oh yes, the actual novelty of the attack all matter as well (there’s a reason it’s called ‘news’). Put simply, no bug that does not actually threaten the stability of the network, in a way we aren’t already mitigating, should ever follow partial disclosure.

I will say, it’s totally OK to fix old and severe bugs that nobody deigned to repair. But you actually do need to get them fixed 🙂

So, what options do Robert and Jack have with the sockstress bugs? Without full details on the attack, the only detail everybody can judge is that there are no patches for this attack — but the same operational mitigations that we use for botnets, will almost certainly cover this issue as well. Both are dings against this bug getting slotted into the Internet-Threatening Partial Disclosure bucket. Things are complicated, however, by the fact that this has already gone off the rails and into the press and vendor realms.

What exactly they should do, really does depend, and isn’t really my place to say.  After all, it’s their bug, they can do what they want.  But now that this has gotten into vendor hands, I think they need to weigh in, to Robert and Jack, what their perspective on this issue is.  It’s entirely possible that this issue has been wildly overblown — it’s a legitimate class of bug, yes, but nothing to panic over and something that can be discussed without “Killing the Internet”.  In this case, there’s nothing wrong with Robert and Jack releasing one of their many bugs, just so they can stop talking in theory.

If it’s, in fact, a more severe issue — severe enough to pull the talk, at least as per the vendor — I do think the community needs to have its say on the nature of the fault class.  It’d just be too tempting for a fly-by-night operation to threaten vendors into ‘playing along’.  A talk can be pulled to be nice to a vendor, or a talk can be pulled because it’s a massive scale threat, but only the community can judge which is which.

Yes, this is all enormously messy. The next time anyone asks why partial disclosure is not more common — this is why.

Categories: Security
  1. October 3, 2008 at 3:18 pm

    Nice thoughts, however I guess it’s up to anyone to decide which path they go. Basically a lot of peeps knew DNS had far more issues than we knew, the difference is that people now know it has problems. “socketstress”, the same. The TCP/IP stack is vulnerable by design, and probably hard to fix. But it isn’t limited to “socketstress” anyone who pays a little attention knows that a lot of TCP attacks are published that never got mainstream attention. If I understand the “socketstress” issue enough, I think what it might be. After a 3 way handshake, send a ACK>FIN with bogus injected ack_num (ack_num<-1) but with correct seq_nums, this would recompute all checksums and flush all reassembly, with enough (few million) packets this will deplete the timers while keeping an alive state, rejecting further connections. It’s one attack that might work. But again, people knew about TCP/IP issues/attack theorems for a very long time, including a lot of possible DNS issues c.q. DNS attack theorems. The real question for me is: Why didn’t they inform people, or fix it?

    Well I guess my conclusion is:

    1) They knew it’s tough to fix.
    2) They kept it silent, hoping no-one would find out.
    3) They knew, kept silent to work on a fix.
    4) They couldn’t care less.

    Maybe those who knew about the TCP/IP c.q. DNS problems were working on a theoretical fix in all those years, making it a real NDA.

    Food for thought. 😉

  2. Adam Cecchetti
    October 3, 2008 at 3:25 pm

    If you are going to create a console it needs to be vendor independent. It has to have people that are professional auditors code, design, and penetration who find world ending bugs for a living. A console of individuals that are not in the weeds can’t properly serve in this role. One false bell ring ends the credibility of such a group.

    For instance when someone marketing team can’t get their “world ending bug” published. What prevents copy cat groups from appearing as a PR stunt, for street cred, mad loot and women, etc? They just spin up their own “we are the partial disclosure task force”. Our world ending bug really is one now the PDTF say so!

  3. October 3, 2008 at 3:32 pm

    On the other hand -excuse me for the 2nd post- But, I heard that the socketstress issue won’t be detectable, however since ICMP can send issues regarding IP datagram errors, isn’t this then trivial to detect? Since it doesn’t matter if you’ve obtained a 3-way handshake or not. OK it requires a boot to become live again, but I don’t understand that one can’t detect it form happening. That’s what I was wondering about. Ah well, I’m probably running ahead of the issue since I don’t know exactly how it works what Robert and Jack have found.

  4. October 3, 2008 at 4:42 pm

    rvdh–

    To the discloser, goes the spoils. It’s one thing if you knew about something for years, but if you didn’t talk about it, or release any code, it’s totally fair game for other people to discover it. I mean, look at the word: Dis-cover. To remove the covers.

    I learned many, *many* years ago that almost everything I’ve ever done has floated around various spooky places forever. I’m OK with that. Humbled by it, actually.

    Adam–

    Agreed. 100%. Really, you need people who themselves will on the line if they make this call wrong. That’s why you’re invited.

  5. concerned friend
    October 4, 2008 at 2:12 am

    I though you should know that pretty much everyone who’s anyone is laughing at you. I’m not one of them, so let me kindly suggest you read through your post again from the standpoint of someone who’s clear-headed. Please…

  6. October 4, 2008 at 7:23 am

    I certainly agree on that, however if one reports an issue to vendors, where vendors say: It can’t be fixed. What do you do, keep silent or semi-disclose and let other dis-cover it as well. In my opinion it’s pretty useless to semi-disclose it, people will find out if you give them a place to look for. There are a lot of smart people, far smarter then we like to think we are.

    So I’m really not for semi-disclosure, one either disclose or keep it silent and work with vendors to fix it. But I am pretty sure with the TCP/IP implementation in various operating systems, little can be done in such a short time-span.

    But again for me, it depends on it’s impact, I’m not talking about trivial browser bugs or XSS, that’s just common fun. That’s a far different level of responsibility, which I surely never hope to carry.

  7. October 4, 2008 at 11:05 am

    rvdh–

    The point is, I’m not seeing the point of semi-disclosure or partial disclosure, except in pretty rare circumstances. Maybe my case counted, maybe it didn’t, but I certainly wish I had included others in that decision.

    concerned friend–

    Oh, I’m sure some people are laughing. It’s a mess, a sort of mess that I’m sure they all think they predicted. They’re not entirely wrong.

    Most of these same people thought 30 days wouldn’t help get people patched. So they shouldn’t be *too* satisfied with their Miss Cleo powers. But I think we can all agree, some organized way for security reporters (who don’t know) to get a clear answer on the severity of an issue, from those of us in the community — not necessarily me — is, if imperfect, way better than the path we’re on now.

  8. Dan W.
    October 4, 2008 at 8:06 pm

    Okay, so how do you get people to pay attention. I was able to find an error in Internet Explorer 7 which Microsoft has patched in the Internet Explorer 8 beta and is not there in Internet Explorer 6. It took a lot of prodding and bugging of Microsoft and then I decided okay, I need to let US-Cert, The Department of Homeland Security know about this and they emailed me back with a case number on it and asked for any more details I had which I gave them so I think I did the right thing by disclosing the issue to Microsoft first and then US-Cert but it took the federal government and their issuing a case number on the issue to get the problem really addressed and it still has not been patched in Internet Explorer 7. This does give me the satisfaction that I at least let Microsoft know about the problem and then US-Cert so I know at least I did the right thing and I have no big desire for glory or fame and just want to help the industry from the many destructive and evil forces out there.

    Computer user since about 1983 on an IBM PCjr where I learned BASIC programming and enjoyed playing my first computer games on cartridges and my first big adventure game was King’s Quest 1 by Sierra On Line on a 5.25 inch floppy disk. I still have that machine in storage and use it at least every few years and it still works well.

  9. October 6, 2008 at 9:06 am

    Yes, partial disclosures suck. Yes, some reporter will potentially write a story about something that isn’t true – but then again this has happened many times in the past already. I’d certainly be willing to vet vulnerability announcements for reporters.

    The only thing I don’t like about this idea is that it is Kaminsky playing the role of the anti-Kaminsky’s last time around and so it is setting off weird alarm bells in my head.

    Last time:
    Dan: “I have a bug that will break the internet”
    Other researchers “Bullshit, its a birthday attack, if it was real, he’d give more information, blah blah blah”

    This time:
    Lee and Lewis: “We have a bug that can DOS anybody, and hence break the Internet”
    Dan: “Nah, its just a dos, we’re good at defeating dos’s. I wish we had more information to verify it, …”

  10. October 6, 2008 at 10:21 am

    A council of people designed to triage vulns and determine if they are “partial disclosure” worthy is a crazy idea. Having to get hype production approved goes directly against the finder’s economic incentives. Finders practicing partial disclosures before talks are looking to market their talks and information. Everyone “in” the security industry benefits from this hype machine. Conference organizers see increased ticket sales, consultants who understand the issue get client bonus points, vendors get to position their product as ready to fight these unexpected threats, and the finders get a nice notch on their resume.

    Why would a finder go through your council when they are not even sure that it won’t be disclosed anyway? (*cough* DNS *cough*)

    Who can be on this council? As Adam pointed out, it should be vendor independent. Good luck finding people with no incentive to know bugs early and gate their disclosure rate. The people who used to do this as entirely a hobby are pretty much gone.

    We had a model where attackers and patchers got to start at the same point, with the same information, the same tools and race to secure their systems. Now we have a model where patchers have less information, less patching material, and attacker’s have just enough information to go out and rediscover the bug.

  11. Freedom
    October 6, 2008 at 11:40 am

    An open and free market, with minimum regulation and enforcement of law and culture, is best for IT.
    Credibility? You want to try to create trust when everything is gamed on purpose?
    Getting paid requires a lot of people and systems to access benefits and risk. An independent group is only like a ponzi scheme.
    Pogo, we have meet the enemy and they are us. How are you going to solve this?
    Sorry charlie, we live best open, and free from a-z.

  12. October 6, 2008 at 5:12 pm

    Charlie–

    The situation is a little different. I ran a quiet conspiracy for a half a year to synthesize simultaneous patches for all known platforms, and thus had actionable intelligence on day one. We don’t have that in this case. What is the action people are supposed to take here?

    On a larger scale, I didn’t go out with enough validation. I said so, pretty explicitly, a few days after this all hit. I’ve been pretty consistent on that. But I didn’t come out empty handed, either: All the vendors, the inventor of DNS, and CERT were all backing me. The question is, how can we scale that level of backing to future bugs that are found, so that we don’t end up with the situation where the bigger the claim, the less it needs to be proven? More importantly, how do we get to the point where when the issue does become public, it has the necessary gravitas in both the IT and security community to be treated as it deserves?

    This is a process I’d have happily submitted my own work to. More importantly, it’s a process that reporters and editors should be able to demand, before they write breathless pieces that the Internet is seriously threatened.

    Chris–

    You’re thinking too short term. The problem is that within the next year, press people may very well be inundated with claims of Internet killing bugs. They’re going to need a way to filter the crap that I’ve sent their way — and researchers are going to need a better rebuttal to a bug than “I bet it’s X and X is lame”.

    For better or worse, “I bet it’s X and X is lame” is now dead. We need better. “He won’t submit to NDA’d peer review” ain’t bad. Theoretically vendors would be in a position to call BS, but vendors have a much deeper conflict of interest than hackers do.

    Again, this is less about this particular issue — Jack’s got a DoS of some sort, of that I don’t doubt — and more about the complete lack of validation. That’s at least partially a result of me leaving a “template” of “Internet Going Down, News At 11” bugs out there, without some degree of controls over how that’s used.

  13. Codejunky
    October 7, 2008 at 1:56 am

    Hi guys

    I am not a security researcher but I enjoyed the read non the less. I will be posting a link to this page on House Of Hackers as I believe this will hit a chord with some of the members.

    Congrats on the DNS flaw you found and for putting forward such a responsible and probably necessary solution to an old problem. One thing you may also consider advertising to hackers is a disclosure procedure for company systems which have been hacked by grey hats.

    The reason I suggest this is that a school hacker recently did some grey hat work and suffered the concequences of disclosing the attack. He is now out of school which is of no benefit to anybody and facing trial (have posted a link below).

    http://www.securityfocus.com/brief/829

    I believe the facility for responsible disclosure of both flaws and unsecured systems should exist

  14. Dan W.
    October 7, 2008 at 1:39 pm

    How can we get people that are willing to work as a group and see finding the software bugs as part of their hobby and potentially a reward of better and more challenging careers in the future? I can see all the problems with partial disclosure and see how important it is to have closed and responsible disclosure. I appreciate all of the work that CERT has done. Unfortunately, I do get annoyed at Microsoft for insisting that their newest operating systems are best. I do get annoyed at Apple for tying I-tunes, Bonjour and Quick-time as one and not allowing the use of I-tunes without Quick-time and have them as separate programs which I would prefer. The case with Microsoft about the newest being the best is not necessarily the case because the more complex and complicated the operating systems get the easier it will be to find flaws within their software layers. At my work, I have seen script errors and buffer overflow errors and they are unfortunately too common. Anyway, if we are able to have a small group of volunteer professionals here that are willing to test and find errors in software then what safeguards can we have to determine that the person is who they claim to be (things like multiple strong passwords, 256 bit AES encryption, biometrics, etc.) and also what safeguards can we have that will help prevent people from working with this group for a while and then not bolting and disclosing a flaw publicly to hackers for money.

  15. October 7, 2008 at 8:21 pm

    I agree wholeheartedly with this sentiment. This (growing?) trend of ‘dribble disclosure’ has got to stop, it doesn’t help anyone really (arguably other than the folks doing the dribbling and pandering to the media).

    If you’re going to disclose, do it or don’t. Whether you do it before or after contacting vendors etc. and getting the issues fixed is entirely beside the point IMO. Either way, if you’re going to talk to the media about a vulnerability, you need to also have presented wtf the vulnerability is so that:
    some people can be working to fix it
    other people can be figuring out where else it could be used (so that those broken places can also be fixed)
    none of these people will be wasting cycles trying to recreate a problem that’s already been identified by following a broken up set of half eaten bread crumbs

  16. D2
    October 10, 2008 at 1:01 am

    K, am confused. Is this not what CVSS gives us?

    CVSS http://nvd.nist.gov/cvss.cfm

    DNS flaw:
    http://web.nvd.nist.gov/view/vuln/detail?execution=e2s1

    Just give us a CVSS rating pre-release of actual details via the normal channels.

  17. Wyrd
    January 9, 2009 at 5:48 am

    Hi. I just wanted to say that I agree with you that the industry needs to have that “put up or shut up” mentality. I don’t want to be rude, but this should be obvious: that’s what makes it behave like a science rather than just a bunch of people standing around yelling that the sky is falling.

    Thank you for your work on the DNS bug. You’re cool.


    Furry cows moo and decompress.

  1. October 6, 2008 at 8:42 am
  2. October 7, 2008 at 8:13 am
  3. October 7, 2008 at 5:02 pm
  4. October 10, 2008 at 10:52 pm
  5. October 12, 2008 at 11:23 am
  6. October 13, 2008 at 12:28 am
  7. October 19, 2008 at 3:25 am
  8. October 28, 2008 at 11:57 am
  9. November 18, 2008 at 11:21 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: