Dan Kaminsky's Blog

So, all the way back at the beginning of the year, I was astonished once I realized engineers from companies all around the world were flying out — on a few weeks notice, barely — to discuss how to fix my bug.

And now, with the bug finally public…everyone else is joining in.

First off, the ISP’s are on board!.  Andy Greenberg from Forbes went ahead and talked to the likes of AT&T, Time Warner Cable, and Cablevision, and while they’re not all patched up yet, they’re indeed working on it.  Let me take a moment to quote Time Warner here:

Time Warner Cable spokesman Alex Dudley responded, “We do have a solution to the DNS issue that we feel works for us, and we’re working on rolling it out as quickly as possible.” He added, “We have to make sure it works and that it doesn’t impact the system in any way.”

That was, in a nutshell, the very idea of this entire approach — to somehow give people a way to get these fixes out in a controlled manner, without forcing a panic.  If there’s one thing you do not want to be on fire, it’s your DNS server.  Nothing makes a network go wobbly faster.

Now, I’m not saying ISP’s don’t need to get (un)cracking.  The headline of Andy’s article is “Hackable Broadband Left Unpatched”, and with the notable and named exceptions of Comcast and Verizon, this is true.  But seeing Time Warner, and Earthlink, and even AT&T doing the right thing and actively investigating how to spread this fix out across their entire network is fantastic, even if it is not instantaneous.

In a rough spot right now, unfortunately, are some of the Firewall and NAT vendors.  I feel pretty bad about this — I simply underestimated how many DNS infrastructures were behind a NAT, and how many of those NATs were altering port numbers in flight.  Tom Cross from X-Force ISS has done some very good work analyzing the NAT situation, while the various vendors affected are working on a rather compressed schedule to provide mechanisms that either won’t interfere with or will supplement DNS entropy.

In the meantime, there’s some interesting research I’d like to call attention to.  Michael Rash has been looking into IPTables, and has found that Linux’s IPTables can be made to implement strong port randomization independent of the name server itself.  I haven’t tested this approach under load (read: queryperf) but this is very impressive work, that requires no code to be pushed in order to secure a nameserver running on or behind Linux.  (Update:  Same, for PF. OpenBSD FTW.)

Again, I’ve spent most of my career in corporate environments.  I’m well aware it’s sometimes easier to push a config than to push code.  So, I’ve been paying special attention to solutions that allow administrators to do just that.  Michael’s IPTables trick is one approach.  There is another:  Forwarding.  Name servers can be configured to send all requests directly to another host — another host that may itself be a new system, freshly patched, and possibly even outside a NAT’s port-sequentializing boundary.

As a service to the community, I asked Paul Vixie if ISC wouldn’t mind writing a short document about how to forward your DNS traffic from your unpatched server to something more safe.  (See also, how to configure a forwarder on Windows. EDIT:  You may need to set this registry key to prevent insecure failover.)  Note, if you must forward, it’s most secure to do so to a name server that’s still on your network but happens to be patched — but in a pinch, you’re much better off forwarding to OpenDNS or another free and patched name service provider than going direct (and insecure).

A couple people have been curious if there’s any way to detect that they’ve been attacked.  A few months ago, I was lucky enough to get a chance to work professionally with Jose Avila.  Jose’s done a fairly absurd amount of work on production DNS systems, and I needed someone of his talents.  At some point, we got to talking about an open source cache auditing script.  He actually just released one, under a BSD license:  CacheAudit.  (See also:  README on DailyDave, Slides from DNS OARC).  We potentially have some rough times ahead, and this sort of code is great to have around.

Finally, on a more personal note, you might notice that www.doxpara.com just experienced something of a revamping — and, er, there’s a rather nice looking DNS Test Widget off in the corner of the page.  How did this happen?  I’m many things, a master of Content Management Systems is not among them.  You can blame Crystal Williams, formerly of Warner Brothers’ Technology Department and now starting a consulting practice up here in Seattle.

Why a widget?  Because “It worked for Madonna, maybe it can help secure the Internet” was just too awesome not to try. And, as Crystal says, always err on the side of awesome.  (Though I suppose I do need to make that widget embeddable in other sites, and a little more flexible.  Until then, Duane Wessel’s TXT approach should help.)

So, that’s the latest on the DNS front.  Special thanks incidentally to those who could publicly be speculating on this matter, but are choosing not to.  You’re doing a lot to help keep this ship afloat, and I look forward to thanking all of you in Vegas.