Home > Security > An Astonishing Collaboration

An Astonishing Collaboration

Wow. It’s out. It’s finally, finally out.


So there’s a bug in DNS, the name-to-address mapping system at the core of most Internet services. DNS goes bad, every website goes bad, and every email goes…somewhere. Not where it was supposed to. You may have heard about this — the Wall Street Journal, the BBC, and some particularly important people are reporting on what’s been going on. Specifically:

1) It’s a bug in many platforms

2) It’s the exact same bug in many platforms (design bugs, they are a pain)

3) After an enormous and secret effort, we’ve got fixes for all major platforms, all out on the same day.

4) This has not happened before. Everything is genuinely under control.

I’m pretty proud of what we accomplished here. We got Windows. We got Cisco IOS. We got Nominum. We got BIND 9, and when we couldn’t get BIND 8, we got Yahoo, the biggest BIND 8 deployment we knew of, to publicly commit to abandoning it entirely.

It was a good day.

CERT has details up, and there’s a full-on interview between myself and Rich Mogull up on Securosis.  For the non-geeks in the audience, you might want to tune out here, but this is my personal blog and I do have some stuff to mention to the crew.

There’s something very important about what we accomplished here.

We. Because there’s absolutely no way I could have pulled this off by myself.

Paul Vixie is an institution. Having long maintained the Internet’s most popular DNS server, Paul simply knows everybody. Paul was absolutely instrumental in pulling together the engineers we needed to solve this problem. We needed Florian Weimer there, all the way from Europe. We needed David Dagon, and Jinmei Tatuya, and Wouter Wijngaards. We needed Microsoft, Cisco, Nominum, Neustar, and OpenDNS.

And we really needed CERT.

It was an interesting discussion, with lots of disagreement, but ever-growing consensus. After evaluating several options, one approach was clear — and, I must admit, somewhat embarassing to Paul.

DJB was right. All those years ago, Dan J. Bernstein was right: Source Port Randomization should be standard on every name server in production use.

There is a fantastic quote that guides a lot of the work I do: Luck is the residue of design. Dan Bernstein is a notably lucky programmer, and that’s no accident. The professor lives and breathes systems engineering in a way that my hackish code aspires to one day experience. DJB got “lucky” here — he ended up defending himself against an attack he almost certainly never encountered.

Such is the mark of excellent design. Excellent design protects you against things you don’t have any information about. And so we are deploying this excellent design to provide no information.

To translate the fix strategy into a more familiar domain, imagine large chunks of Windows RPC went from Anonymous to Authenticated User only, or even all the way to Admin Only. Or wait, just remember Windows XPSP2 🙂 This is a sledgehammer, by design. It cuts off attack surface, without necessarily saying why. Astonishingly subtle bugs can be easily hidden, or even rendered irrelevant, by a suitably blunt fix.

Of course, it remains obvious that something new is up, and that something will be found eventually. But there’s a lot of buggy systems out there, vulnerable not just to new bugs but bugs that have been known for years. If all this effort ever accomplished was sweeping old and crusty BIND8 off the Internet, if we could finally fully eliminate Joe Stewart’s (edit: Originally Vagner Sacramento’s, thanks Joe!) Birthday Attacks from 2002, if we started doing something about Amit Klein’s Transaction ID Randomness finds (even the deeply underrated client vulns) from last year, and yes, if the static port assignments DJB warned us about ages ago were finally shut down — then this would still be a huge win.

There are reasons why the new issue is particularly severe, but I think reasonable people can agree that anything that could scrub even the old bugs would be a boon to the security of the Internet. And so, I ask the open research community…assume I found nothing! Assume this is nothing but a stunt, to finally get people to take Joe and Amit and DJB seriously, and to give network scanners a crystal clear fingerprint of what a trustable recursive server looks like.

Joe and Amit and especially DJB have done some incredible work. I’d look terrible at the end of it, but their bugs would finally get fixed, and stay fixed. As for me, I dunno. Go back to graphics 🙂 Mmmm…SIGGRAPH…

For those of you who won’t make that assumption, I have a request. It is very unusual, and maybe unreasonable. But I have to ask.

I want you to explore DNS. I want you to try to build off the same bugs I did to figure out what could possibly go wrong. Maybe I missed something — I want you to help me find out if I did, so we can deal with it now instead of later. I do want all this. But I also want my family to be able to use the Internet in peace. I’m not asking for forever. I am asking about thirty days. I’ve done everything in my power to get the patches available, no matter the platform. But the code doesn’t (always) install itself. While I’m out there, trying to get all these bugs scrubbed — old and new — please, keep the speculation off the @public forums and IRC channels. We’re a curious lot, and we want to know how things break. But the public needs at least a chance to deploy this fix, and from a blatantly selfish perspective, I’d kind of like my thunder not to be completely stolen in Vegas 🙂

Now, if you do figure it out, and tell me privately, you’re coming on stage with me at Defcon. So I can at least offer that 🙂

Categories: Security
  1. July 9, 2008 at 4:14 am

    An amazing find…carry on the good work, you deserve your thunder.

    Speak soon,


  2. Nicholas Weaver
    July 10, 2008 at 8:24 am

    Question: What IS your email/contact info for our random speculation?

  3. Joey Hess
    July 10, 2008 at 9:50 am

    When I try your checker (in FF3; on debian, FWIW), I get a 404 error page in the box, it seems to be failing to load http://853784753500.toorrr.com/printme.html (exact hostname varies)

    Does this mean I’m vulnerable, not vulnerable, or on an unsupported platform? Or something else strange going on.

  4. July 10, 2008 at 9:56 am

    I’m just wondering, so you found all this bug on all these diff platforms/OS’s, i can’t stop wondering… to be able to find all that, how many times did you use the bug in order to gain access on each of those platforms/OS’s to be sure that the bug is real and not just another fake seemingly flaw ? And if so of which side that counts? White Hat or Black Hat ?:) I’m pretty sure there were plenty people whom knew about this bug, found it, use it to gain access and do silly things and that’s how it was spotted.Atleast that’s kinda logical to me.
    But if you’re saying that you changed the course of history by doing this, meaning the Security was first to find the bug and not the actual Hackers, then that’s a First. Correct me if i’m wrong but i never found 1, ONE ! person whom found a bug and didn’t use it. I know back in the old days when a wellknown(these days) person found a major bug, use it to gain access on all possible or imaginable “spots” before he actually reported the bug and provided the fix for it. How are you different from any other ?

  5. July 10, 2008 at 9:56 am

    Thanks for the massive efforts ;>

  6. Laurence
    July 10, 2008 at 1:34 pm

    Thank you.

  7. Phil
    July 10, 2008 at 2:01 pm

    I just hope BIND admins aren’t as clueless as RedHat.

    Their updated BIND and caching-nameserver RPMs still have example (and real) .conf files which contain the following:

    query-source port 53;
    query-source-v6 port 53;

    Even a patched BIND will still use a fixed port if those lines are in the config.


  8. ian
    July 10, 2008 at 3:25 pm

    Tried Security Update for Windows Server 2003 (KB951746) and the DNS vulnerability scanner on doxpara.com says I’m still vulnerable?

  9. Phil
    July 10, 2008 at 3:57 pm

    RedHat have released an updated advisory, fixing the issue I mentioned above, and warning admins to check their configs.


  10. July 10, 2008 at 7:18 pm

    I apologize for the cross-posting, Dan, but can you address this:

    I’m also interested in the comments highlighted by Tom Cross from IBM/ISS X-force reiterating a comment sent to FD regarding DNS and NAT devices. You can find Tom’s post here: http://blogs.iss.net/archive/dnsnat.html

    ” I’m writing this blog post in order to draw more attention to that observation, as I’m not sure that the security industry has realized its full implications. As far as we know, this observation applies to any NAT device, not just the firewall imipack tested, and we think it has significant implications for network architecture.

    When a host on a computer network selects a source port for a UDP request, it selects a port that is unique for that host. When a one-to-many hiding NAT device receives that request and translates it, it has to assign a new source port, because the NAT device has to assign unique ports for all of the hosts on the internal network. X-Force is not aware of any NAT device on the market that randomly selects UDP source ports. Therefore, when patched DNS clients and servers are used behind NAT, they are still vulnerable to attack. The source port entropy introduced by the recent patches is cancelled out by the NAT device.

    It is our opinion that network administrators who have caching internal DNS servers behind NAT should plan to move those servers into a DMZ where they can be directly assigned a unique Internet IP address. Any servers that remain behind NAT devices will remain vulnerable after patches have been applied, and X-Force suspects that given the amount of media attention DNS Cache Poisoning has received this week that an increased frequency of attack is likely in the coming months.”

    What say thee to that? Patching may not be enough if you physically have to move DNS servers to what amounts to DMZ’s and then expose them with 1-to-1 NAT’s instead of hides…oh the fun.


  11. July 10, 2008 at 7:54 pm

    Really DNS, are importants in the Web, than it could provocate many damages to people.

    Specially in my country, DNS are full, because the capacity is for only a amount, but we have exceed it.


  12. urbanriot
    July 10, 2008 at 8:17 pm

    DJB was lucky? I think not… he speculated a future security issue.

  13. Someone who doesn't know much about DNS
    July 10, 2008 at 10:20 pm

    I’ve wondered for a while about something. Can a DNS request request the resolution of more than one domain name? If so, it seems like you could ask remote domain servers to resolve both “www.google.com” and “7fe85813f1d6d7cc.bogus” for you, with the rejection message acting as proof of knowledge of the nonce.

  14. A. K. Basu
    July 11, 2008 at 12:08 am

    How to resolve the infection (Cache Poisoning)?

  15. July 11, 2008 at 12:51 am

    Wow. If that’s truth it’s impressive.

    Random ports? Hmmmmmm…………

  16. July 11, 2008 at 1:05 pm

    Thanks for your dns resolver checker! I’ll note that I do indeed see non-randomness in the source port selections in my current dns resolver that the simple check script didn’t notice, so beware folks.

    But it would be much more useful if we could run it against arbitrary dns servers, and from remote hosts. It uses javascript for both the request and as part of the first response and javascript often isn’t included in text-mode browsers. How about a python or perl version?

    Thanks again, and congrats on such a well-rolled-out announcement.

  17. tonyn
    July 12, 2008 at 5:47 am

    Ian wrote:
    > Tried Security Update for Windows Server 2003
    > (KB951746)


    > and the DNS vulnerability scanner on
    > doxpara.com says I’m still vulnerable?

    This indicates that your ISP still needs to update their systems. (Or another ISP along the chain that your local ISP uses for access.)

    End users are generally not the target of DNS poisoning, except maybe on your local Wi-Fi or college network. The usual target would be the DNS system of a large ISP or corporate.

    Joey Hess wrote:
    > When I try your checker (in FF3; on debian, FWIW), I get a 404
    > error page in the box, it seems to be failing to load
    > http://853784753500.toorrr.com/printme.html (exact hostname
    > varies)

    Something between you and toorrr.com is dropping packets. Possibly accidental or possibly policy from your ISP, (or college/employer if you are not at home).


  18. asd
    July 14, 2008 at 3:46 am


  19. Vincenzo
    July 15, 2008 at 3:30 am

    Tried Security Update for Windows XP SP2 (KB951746) and the DNS vulnerability scanner on doxpara.com says I’m still vulnerable…

  20. July 15, 2008 at 12:13 pm

    DJB’s track record on writing secure, solidly-designed software is the reason that I STILL run qmail today.

  21. July 16, 2008 at 4:05 pm

    What are some of the other types of consequences from the bugs and DNS poisoning to be aware off. Have you personally been infected.


  22. Bill
    July 17, 2008 at 4:36 am

    “Luck is the residue of design” — Branch Rickey.

  23. David R
    July 21, 2008 at 1:24 pm

    Kuddo’s for doing this “right”. We could have ended up like the final act in animal house, but somehow, somewhere the miracle of doing the greatest good triumphed. I believe again. Hope is alive. Good does exist.

  24. evil_tgn
    July 23, 2008 at 2:11 pm

    Nice Shot, thx.

  25. Sadi
    July 27, 2008 at 10:57 am

    i don’t want to think about what if he hadn’t find that bug before “others”…

  26. July 27, 2008 at 1:11 pm

    Hi Dan,

    I am writing from India, where the bandwidth problem is a big issue. I have tried working your vulnerability check here, but the baandwidth does not allow it. It always says that my request has been timed out. So it is it possible for you to develop your solution in a downloadable format so that we can check our systems and sleep peacefully at night.

    BTW your video is pretty cool to make people realize that the problem is beyond comprehension of common people.


  27. me
    July 28, 2008 at 1:39 pm

    Me again.

    Your tester says my company’s DNS is vulnerable (identifies it by an IP number, says it’s always using the same port number)

    The other tester mentioned online
    identifies it by an outside vendor’s site, and says it’s very safe, highly random.


  28. July 30, 2008 at 9:42 am

    We have a nice class C BIND Scanner here:


  29. August 24, 2008 at 7:16 pm

    cheapest domain names

  30. Adrian
    October 5, 2008 at 9:35 pm

    Wonder what the behaviour of an authoritative DNS server for say example.com will be if it is injected with wrong IP address of http://www.example.com. With it reply from the poisoned cache with wrong IP address or the correct IP address from the zone file?

  1. July 9, 2008 at 5:14 am
  2. July 9, 2008 at 5:22 am
  3. July 9, 2008 at 5:24 am
  4. July 9, 2008 at 6:37 am
  5. July 9, 2008 at 8:08 am
  6. July 9, 2008 at 8:20 am
  7. July 9, 2008 at 9:50 am
  8. July 9, 2008 at 10:25 am
  9. July 9, 2008 at 10:43 am
  10. July 9, 2008 at 10:51 am
  11. July 9, 2008 at 11:13 am
  12. July 9, 2008 at 12:00 pm
  13. July 9, 2008 at 12:09 pm
  14. July 9, 2008 at 1:02 pm
  15. July 9, 2008 at 1:03 pm
  16. July 9, 2008 at 1:06 pm
  17. July 9, 2008 at 1:20 pm
  18. July 9, 2008 at 1:29 pm
  19. July 9, 2008 at 4:26 pm
  20. July 9, 2008 at 5:08 pm
  21. July 9, 2008 at 5:53 pm
  22. July 9, 2008 at 8:15 pm
  23. July 9, 2008 at 8:57 pm
  24. July 10, 2008 at 2:35 am
  25. July 10, 2008 at 3:17 am
  26. July 10, 2008 at 8:00 am
  27. July 10, 2008 at 8:12 am
  28. July 10, 2008 at 8:20 am
  29. July 10, 2008 at 8:40 am
  30. July 10, 2008 at 9:16 am
  31. July 10, 2008 at 9:29 am
  32. July 10, 2008 at 9:32 am
  33. July 10, 2008 at 9:58 am
  34. July 10, 2008 at 10:22 am
  35. July 10, 2008 at 12:02 pm
  36. July 10, 2008 at 1:53 pm
  37. July 10, 2008 at 3:56 pm
  38. July 10, 2008 at 9:20 pm
  39. July 10, 2008 at 11:26 pm
  40. July 11, 2008 at 1:48 am
  41. July 11, 2008 at 1:54 am
  42. July 11, 2008 at 4:51 am
  43. July 11, 2008 at 8:06 am
  44. July 11, 2008 at 8:07 am
  45. July 11, 2008 at 11:31 am
  46. July 12, 2008 at 1:50 am
  47. July 12, 2008 at 5:43 am
  48. July 12, 2008 at 9:29 am
  49. July 12, 2008 at 11:12 am
  50. July 18, 2008 at 10:10 pm
  51. July 19, 2008 at 4:36 am
  52. July 20, 2008 at 12:45 am
  53. July 21, 2008 at 8:39 pm
  54. July 22, 2008 at 9:17 am
  55. July 22, 2008 at 10:53 am
  56. July 22, 2008 at 12:00 pm
  57. July 22, 2008 at 2:05 pm
  58. July 22, 2008 at 4:34 pm
  59. July 22, 2008 at 10:25 pm
  60. July 23, 2008 at 5:09 am
  61. July 23, 2008 at 8:14 am
  62. July 23, 2008 at 3:40 pm
  63. July 23, 2008 at 3:57 pm
  64. July 23, 2008 at 5:33 pm
  65. July 23, 2008 at 6:03 pm
  66. July 23, 2008 at 7:22 pm
  67. July 24, 2008 at 3:16 am
  68. July 24, 2008 at 3:38 am
  69. July 25, 2008 at 9:30 am
  70. July 25, 2008 at 12:42 pm
  71. July 25, 2008 at 1:59 pm
  72. July 25, 2008 at 2:08 pm
  73. July 25, 2008 at 10:33 pm
  74. July 27, 2008 at 10:43 pm
  75. July 28, 2008 at 2:45 am
  76. July 30, 2008 at 12:51 pm
  77. July 30, 2008 at 11:07 pm
  78. August 1, 2008 at 10:02 pm
  79. August 2, 2008 at 11:16 am
  80. August 4, 2008 at 8:34 am
  81. August 4, 2008 at 11:23 am
  82. August 7, 2008 at 5:12 am
  83. August 8, 2008 at 4:23 am
  84. August 10, 2008 at 3:49 pm
  85. August 12, 2008 at 4:42 am
  86. August 15, 2008 at 4:12 pm
  87. August 15, 2008 at 8:15 pm
  88. August 22, 2008 at 2:02 pm
  89. August 22, 2008 at 3:53 pm
  90. August 26, 2008 at 1:56 pm
  91. August 28, 2008 at 6:11 am
  92. September 4, 2008 at 12:04 am
  93. September 8, 2008 at 11:33 am
  94. November 14, 2008 at 5:43 pm
  95. May 23, 2009 at 11:16 pm
  96. July 17, 2009 at 5:53 pm
  97. April 26, 2011 at 7:29 pm

Leave a Reply to Adrian Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: