Home > Security > Ow My Toe

Ow My Toe

So there’s been some skepticism about the DNS flaw.  I want to be clear:  It was richly deserved.  A “put up or shut up” mentality is critical to the survival of our industry.  It’s just too easy to make stuff up, if you can just wave away detractors with “I can’t prove it…it’d be UNSAFE.”

The danger from that statement is very tempting and very real.  Our credibility as an industry — ultimately, our ability to get bugs fixed — depends on that statement being called out as the bullsh*t that it is.

That being said:

It was my belief that this case was an exception to the rules.  Nobody reading this can know if I was right or not, because (almost) nobody knows the bug.  However, so far, things are going well:

1) Patches are out, from almost everybody.
2) People are installing them.
3) Nobody is panicking.
4) Nobody is being exploited.

Where I come from, that’s pretty damn cool.  But that could all have been accomplished…with no actual bug, just vague hand-wavey threats.

And that’s where I went wrong.  The vendors know the story.  The DNS community knows the story.  But I went live with nobody else in the security community having provided peer review.  The whole point of the security community is that vendors and even subject matter experts don’t always know how their systems can fall over — that knowing how to build something does not at all imply you necessarily know how to break it.

I thought, being Dan Kaminsky, with years of DNS experience, that I could vouch for my own bug.  Er, no.  Not even me.  Especially not me, the beneficiary of what can only be described as “redonkulous amounts of press”.

And then, when Thomas Ptacek asked me point blank to tell him what the bug was, I refused.  It’d be UNSAFE.

Right.  We saw where that went. Right where it should have.

And so, on the urging of Rich Mogull, who’s been instrumental at bringing this entire endeavor out from under the shadows (and who was kind enough not to demand the technical details in order to do it), I did what I should have from the start.  I provided technical details of the attack to Thomas Ptacek and Dino Dai Zovi, submitting myself for peer review.

It went well.

So here’s the bottom line.  I think people don’t have enough information right now, to determine whether there indeed exists any context in which a huge press rush should occur with so few deep technical details.  When everything is on the table, I leave it to the community to judge whether we have gained or lost credibility through this effort.

But it’s clear that, in lieu of details, to not even have respected and completely independent members of the community vouching for your work cannot stand, no matter how respected you are in the community, no matter how many vendors are behind you, no matter what.  OK.  So that’s a fairly big lesson learned, in a process I’ve sort of been making up as I’ve gone along.  Thanks to Dino and Thomas for setting me straight.

Now, that all being said, what people should really do is listen to Sarah.

Categories: Security
  1. saso
    July 11, 2008 at 3:43 am

    if you ask me it is actually better that Thomas and Dino step over on your side after the initial first skepticism. if you would have them in from beginning they would probably be just part of that “mysterious” team together with you the vendors.

    PS: i am still waiting for the answer to my question if home routers are the weal link in this because most of them are not getting patched and they have Dnsmasq and stuff like that in there?

  2. John
    July 11, 2008 at 5:42 am

    I checked my DNS yesterday and it said I was vulnerable. I checked again this morning and it says I am not, although I have done nothing to my DNS server (I am the admin). What gives?

  3. Cooper McBean
    July 11, 2008 at 11:51 am

    First, thank you very much for discovering and leading the charge on fixing the DNS flaw. I am not a computer expert so how can I as a non-expert “make sure the ports listed below aren’t following an obvious pattern.
    Fri Jul 11 18:41:29 2008:undefined TXID=undefined TXID=20757 TXID=24746 TXID=55329 TXID=20826 TXID=21237”

  4. July 11, 2008 at 12:28 pm

    I have been in professional IT for 20 years and it always has amazed me how nobody wants to take responsibility to update all the servers running the internet or at least put a router in between them and the internet. While firewalls seem to be great at keeping out new viruses, most computer still get infected with 10 to 15 year old viruses. When I am in Detroit and I am trying to get to a server in California the fact that my signal is hijacked to china and then back to the US is because the DNS needs updating. Routing tables have not been touched in some cases 15 years. Firewall attacks may not get to your computer but when they are attacking at 10 times per second it hard to have good bandwidth. Also, ATT owns most of the hardware for the internet and all the ISPs claim that its not their responsibility. Its good to see somebody finally took this seriously. I seldom see professional, collabrative, progressive solutions for the internet that help everyone. I do not want to go 30 hops instead of 6. Thanks maybe now the internet will work the way geeks think it should. Fast, Efficient and Properly.

  5. L. Taylor
    July 11, 2008 at 1:05 pm

    It is an astonishing collaboration, and also a predictable reaction.

    I code for one of the smaller organizations that was not invited to the meetings. That isn’t a complaint. We know we’re little.

    It has been difficult figuring out what we needed to do to make our DNS server safe from this new exploit.

    I hope I got it right.

  6. Christopher Morrow
    July 11, 2008 at 2:00 pm

    So, patching BIND/MS-DNS/foo-dns for some enterprises isn’t going to fix this issue. Firewalls or other NAT devices may (in fact do in many cases) remove the randomness in the recursive server requests upstream… So, how important is the randomness of the query source port again?

  7. Nicholas Weaver
    July 11, 2008 at 3:45 pm

    Again, however, since you (rightly) don’t want public speculation, could you provide your email address so I can send you my private speculation, given what you’ve publically stated?

    (Namely: Firewall can’t save you, source port randomization done right will, and what little I know already, because thats enough for me to see how to automagically DNS cache poison a huge fraction of the world. If my thought are the same as yours, cool, I’m not stupid. If its different, its more ammo you can use at BlackHat. :).

  8. L. Taylor
    July 11, 2008 at 5:09 pm

    P.S. feel free to remove my comment.

    I’d just like confirmation that randomizing the port (with a good RNG) and randomizing the query, along with careful matching is enough.

  9. Barry G
    July 11, 2008 at 11:51 pm

    For anyone having trouble… Dan’s email is on his main page, on the upper right, with the text (and link):
    “This is the personal blog for security researcher Dan Kaminsky, who can
    be reached at this email address”.

    July 12, 2008 at 12:45 pm

    I see yesterday you blog…
    EXCELANCE! for you.

  11. July 12, 2008 at 2:03 pm

    Nicholas Weaver: Sidebar.

  12. Jay
    July 13, 2008 at 2:14 pm


    You so rawk.

    – Jay

    P.S. DNS over SOCKS5 through SSH over DNS. Way better than DNS alone.

  13. winkyeah
    July 14, 2008 at 7:28 am

    Hi, quick question. Will people who run firewalls at home (from Linksys, Belkin, etc.) need to update their home firewalls or does this only impact the name servers themselves? The reason I ask is this: since the patches have been put on I have to reboot my old Pix 501 every day otherwise ping requests take about 30-40 seconds. Does that make sense?

  14. Mo
    July 14, 2008 at 10:57 am

    I feel for you.

    It is hard to have evidence in hand, then find it almost impossible to find allies to work with you on an abnormal mundane, such as you found.

    Though not really harmful as mentioned, none the less, people still rely on the security of a public system, to be as airtight, as it possible can be.

    Since this is not a physical, but a technical glich. It wouldn’t be all that hard to plug up.

    We live in a close enough is good enough world.

    The children who chide, are exactly that. What they don’t know, they really don’t know, and it shows.

    Thank you for the info, I was glad to be able to investigate your question and found our network is relatively secure, and our IP totally vulnerable.


  15. Andrew Dalgleish
    July 14, 2008 at 7:10 pm

    Why are there so many “unknowns” on the CERT list?

  16. Ryan
    July 15, 2008 at 6:22 am

    Thanks for the posts and info. However, your “Check My DNS” button is returning a 404:

    “Firefox can’t find the server at d914d94fbdaa.toorrr.com.”

  17. Dola
    July 15, 2008 at 8:01 am

    Jimmy Kimmel and his niece, lol. 😉

    Look, I’d like to know something. Your test says “your DNS appears to be safe but make sure it doesn’t follow an obvious pattern”; what is an “obvious pattern” ? If it’s not “1000/1001/1002…” it’s ok ?
    Also, I didn’t install windows patch. If the DNS Client is disabled, there’s no need to install the patch, right ?

  18. Richard Johnson
    July 15, 2008 at 10:04 am

    Thanks for doing the additional peer review. It was seriously helpful for us.

    The original peer review inherent in the initial team was good for helping convince coworkers, as many know Vixie, Mockapetris, etc. personally, and trust their technical judgement. Along those lines, my knowing you personally, and following your previous DNS work (wait, how can anything that’s so much fun be work?), was enough to convince me you’d found a new and wonderful way to fill DNS caches.

    From there, my looking at the cache poisoning possibilities while knowing there was something interesting to be found let me come up with a few hypotheses. What was missing was a clearer sense of how easy it would be to run an attack. Your picking additional security research peer reviewers I know and trust personally (both their technical skills, and their good sense) fixed that pretty much immediately.

    The review tipped the balance towards the more exciting hypotheses. That in turn factored into our judgement about the risk to our users’ bank accounts. So we patched very darned quickly for this organization.

    So thank you again for having the second peer review done. Also, thank you for the timing you came up with, even if it was by accident.

    The original notice got us thinking and paying attention, then you reinforced the message in a call and answer dialogue. Well done, sir.

  19. July 16, 2008 at 3:40 am

    Certainly got me thinking. Obvious pattern…..I guess so.

  20. jpoley
    July 17, 2008 at 9:08 am


    Just heard you on the Infoblox webinar… I just wanted to say thanks for being one of the good guys.


  21. July 19, 2008 at 12:21 am

    Certainly got me thinking. Obvious pattern…..I guess so

  22. RR
    July 21, 2008 at 7:24 am

    So cool this video ! 😀 All the best Dan

  23. August 11, 2008 at 11:37 pm

    girl pinoy

  24. December 7, 2008 at 5:38 am

    Я подписался на вашу RSS ленту, но сообщения почему-то в виде каких-то значков непонятных 😦 Как это исправить?

  1. July 11, 2008 at 7:09 am
  2. July 11, 2008 at 9:17 am
  3. July 11, 2008 at 9:30 am
  4. July 11, 2008 at 10:11 am
  5. July 11, 2008 at 11:22 am
  6. July 11, 2008 at 11:49 am
  7. July 11, 2008 at 11:55 am
  8. July 11, 2008 at 4:17 pm
  9. July 12, 2008 at 10:45 am
  10. July 12, 2008 at 10:55 am
  11. July 12, 2008 at 1:06 pm
  12. July 13, 2008 at 7:10 am
  13. July 13, 2008 at 1:31 pm
  14. July 14, 2008 at 1:50 am
  15. July 14, 2008 at 8:30 am
  16. July 14, 2008 at 9:10 am
  17. July 15, 2008 at 2:57 am
  18. July 15, 2008 at 3:56 am
  19. July 15, 2008 at 3:21 pm
  20. July 16, 2008 at 1:33 am
  21. July 22, 2008 at 10:34 pm
  22. July 23, 2008 at 7:24 pm
  23. July 24, 2008 at 6:25 am
  24. July 25, 2008 at 11:12 am
  25. August 7, 2008 at 10:36 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: