Home > Security > Here Comes The Cavalry

Here Comes The Cavalry

So, all the way back at the beginning of the year, I was astonished once I realized engineers from companies all around the world were flying out — on a few weeks notice, barely — to discuss how to fix my bug.

And now, with the bug finally public…everyone else is joining in.

First off, the ISP’s are on board!.  Andy Greenberg from Forbes went ahead and talked to the likes of AT&T, Time Warner Cable, and Cablevision, and while they’re not all patched up yet, they’re indeed working on it.  Let me take a moment to quote Time Warner here:

Time Warner Cable spokesman Alex Dudley responded, “We do have a solution to the DNS issue that we feel works for us, and we’re working on rolling it out as quickly as possible.” He added, “We have to make sure it works and that it doesn’t impact the system in any way.”

That was, in a nutshell, the very idea of this entire approach — to somehow give people a way to get these fixes out in a controlled manner, without forcing a panic.  If there’s one thing you do not want to be on fire, it’s your DNS server.  Nothing makes a network go wobbly faster.

Now, I’m not saying ISP’s don’t need to get (un)cracking.  The headline of Andy’s article is “Hackable Broadband Left Unpatched”, and with the notable and named exceptions of Comcast and Verizon, this is true.  But seeing Time Warner, and Earthlink, and even AT&T doing the right thing and actively investigating how to spread this fix out across their entire network is fantastic, even if it is not instantaneous.

In a rough spot right now, unfortunately, are some of the Firewall and NAT vendors.  I feel pretty bad about this — I simply underestimated how many DNS infrastructures were behind a NAT, and how many of those NATs were altering port numbers in flight.  Tom Cross from X-Force ISS has done some very good work analyzing the NAT situation, while the various vendors affected are working on a rather compressed schedule to provide mechanisms that either won’t interfere with or will supplement DNS entropy.

In the meantime, there’s some interesting research I’d like to call attention to.  Michael Rash has been looking into IPTables, and has found that Linux’s IPTables can be made to implement strong port randomization independent of the name server itself.  I haven’t tested this approach under load (read: queryperf) but this is very impressive work, that requires no code to be pushed in order to secure a nameserver running on or behind Linux.  (Update:  Same, for PF. OpenBSD FTW.)

Again, I’ve spent most of my career in corporate environments.  I’m well aware it’s sometimes easier to push a config than to push code.  So, I’ve been paying special attention to solutions that allow administrators to do just that.  Michael’s IPTables trick is one approach.  There is another:  Forwarding.  Name servers can be configured to send all requests directly to another host — another host that may itself be a new system, freshly patched, and possibly even outside a NAT’s port-sequentializing boundary.

As a service to the community, I asked Paul Vixie if ISC wouldn’t mind writing a short document about how to forward your DNS traffic from your unpatched server to something more safe(See also, how to configure a forwarder on Windows. EDIT:  You may need to set this registry key to prevent insecure failover.)  Note, if you must forward, it’s most secure to do so to a name server that’s still on your network but happens to be patched — but in a pinch, you’re much better off forwarding to OpenDNS or another free and patched name service provider than going direct (and insecure).

A couple people have been curious if there’s any way to detect that they’ve been attacked.  A few months ago, I was lucky enough to get a chance to work professionally with Jose Avila.  Jose’s done a fairly absurd amount of work on production DNS systems, and I needed someone of his talents.  At some point, we got to talking about an open source cache auditing script.  He actually just released one, under a BSD license:  CacheAudit.  (See also:  README on DailyDave, Slides from DNS OARC).  We potentially have some rough times ahead, and this sort of code is great to have around.

Finally, on a more personal note, you might notice that www.doxpara.com just experienced something of a revamping — and, er, there’s a rather nice looking DNS Test Widget off in the corner of the page.  How did this happen?  I’m many things, a master of Content Management Systems is not among them.  You can blame Crystal Williams, formerly of Warner Brothers’ Technology Department and now starting a consulting practice up here in Seattle.

Why a widget?  Because “It worked for Madonna, maybe it can help secure the Internet” was just too awesome not to try. And, as Crystal says, always err on the side of awesome.  (Though I suppose I do need to make that widget embeddable in other sites, and a little more flexible.  Until then, Duane Wessel’s TXT approach should help.)

So, that’s the latest on the DNS front.  Special thanks incidentally to those who could publicly be speculating on this matter, but are choosing not to.  You’re doing a lot to help keep this ship afloat, and I look forward to thanking all of you in Vegas.

Categories: Security
  1. wally
    July 19, 2008 at 12:36 am

    Hi Dan,

    Let’s do it the zero-knowledge-protocol-approach:
    I read through the RFC, found what you are not going to publically disclose until Blackhat in August and wrote an exploit that, in combination with a birthday attack, is going to need roughly 200.000 udp packets to again have the same result as before the patch. Would you agree that until DNSSEC gets implemented no one will be able to really fix the problem?


  2. Jon Oberheide
    July 19, 2008 at 8:39 pm


    As far as embeddable widgets go, Niels has whipped up an embeddable image for others to post on their sites:


    Jon Oberheide

  3. Jeff
    July 21, 2008 at 11:55 am

    As for using firewalls to protect against this – recent versions of Checkpoint can be made to randomize source ports and XIDs of outgoing DNS traffic. You can find the settings in Smartdefense, under Application Intelligence->Cache Poisoning->Scrambling.

    Cisco ASA’s DNS Inspection allows for scrambling of XIDs, but not source ports currently. Through conversations with Cisco, I’ve found that they’re adding this functionality to a forthcoming ASA build – 8.0(5) – due out any day now.

    I’m not usually a fan of Checkpoint’s AI and SmartDefense, but in this case it can be helpful – you can even randomize client connections to external servers which might help if you don’t have an internal DNS server or are concerned about various internal segments poisoning each other…

  4. July 22, 2008 at 3:09 pm

    I just wonder, are the (small) DNS implementations in modems/routers (e.g. Fritz!Box) vulnerable too? I have not seen patches for modems. There are quite a lot of modems with a small DNS.

  5. January 28, 2009 at 1:33 am

    Check my DNS function is really a helpful tool! Thank you for sharing this!

  1. July 23, 2008 at 2:48 pm
  2. July 23, 2008 at 7:25 pm
  3. July 24, 2008 at 7:07 am
  4. July 25, 2008 at 9:47 am
  5. July 27, 2008 at 5:15 pm
  6. July 31, 2008 at 9:00 am
  7. August 7, 2008 at 10:30 am
  8. December 8, 2008 at 1:13 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: