Infoblox used to commission an annual survey of the Internet’s DNS namespace by a great little company called The Measurement Factory. In the 2010 DNS Survey, we found that, in a random sampling of millions of subzones of .COM, .NET and .ORG, nearly 20% of zones seemed to have all of their authoritative DNS servers on the same network—in fact, probably on the same subnet (a single IPv4 /24). And, nearly three-quarters of zones had all DNS servers advertised in the same Autonomous System, or AS. One mishap and a single network or AS could disappear temporarily and with it, an organization’s DNS service.
Ten years ago, I said this was an accident waiting to happen. I reminded readers of the embarrassing DNS outage Microsoft suffered years ago when a technician misconfigured a border router in Redmond. Because all of their authoritative name servers were on a single network behind that router, users couldn’t resolve most Microsoft domain names for an extended period.
The solution to this problem is straightforward: Use a diverse set of authoritative DNS servers for your zones. Make sure they’re on different networks and advertised in different Autonomous Systems. If you’re thinking, “Wait, my company only has a single AS, and a single netblock,” don’t worry-they don’t all have to be your DNS servers, running on your networks:
- There are lots of companies now in the business of providing cloud-based DNS services. You could pay one of them to run secondary DNS servers for your zones. These service providers generally run DNS servers in points of presence around the Internet and use anycast to help resist distributed denial of service attacks.
- You could run your own secondary DNS servers, but outside your own network, in a public cloud such as AWS or Azure. With Infoblox’s NIOS product, those cloud-based secondary DNS servers could still be part of your Grid.
- If you need diversity cheaply, you could arrange to “trade” secondary DNS service with another organization. Back when I was at Berkeley, we “swapped” secondary service with Columbia University. They configured one of their DNS servers as secondary for some of our zones and we returned the favor. As far as I know, no money changed hands. If your organization has a peer/sister/subsidiary with their own network, that might be a good choice.
I stopped talking and writing as much about “authoritative diversity” years ago because I thought it wasn’t necessary. I figured I’d just be boring readers who lumped “authoritative diversity” into “widely accepted good ideas” along with “drinking lots of water” and “maxing out your pre-tax retirement contributions.”
And along came the attack on Dyn.
This was October 2016. Some Bad Guys used the Mirai botnet to attack Dyn, a provider of authoritative DNS services. Mirai pumped a massive volume of traffic at Dyn’s DNS servers, resulting in three successive outages, each lasting hours. To users, it looked like much of the Internet was simply down: Dyn hosted the zones of many, many popular web sites and other Internet destinations.
Yet no particular DNS magic was needed to withstand the attack on Dyn, just authoritative diversity. I dutifully put all those slides back into my standard presentation, dusted off my DNS clerical collar and started preaching about the wisdom of “not putting all your [DNS] eggs in one basket.”
At one Infoblox users group meeting in London, the month after the attack, a young man caught me during the break that followed my presentation. The company he worked for, he said, had exactly the setup I recommended: They were a Dyn customer but they also ran a few of their own DNS servers on Infoblox appliances in their DMZs. (If I recall correctly, he also admitted, modestly and somewhat sheepishly, that they had this configuration only because they’d been migrating from running their own DNS servers to Dyn and hadn’t completed the migration.) His employer was a financial services firm, and they took the availability of their Internet services extremely seriously, so much so that they paid a third party to monitor the availability of those services around the clock. And through the whole attack on Dyn, they’d been unreachable to the monitoring service for only a handful of minutes.
Fast forward to 2020—a year rife with challenges—and Cloudflare has an hours-long outage. Their services, including DNS, aren’t available. Lots of organizations disappear from the Internet.
[Taps mic.] Is this thing on?
Don’t make me bust out those slides again. They’ll bore you to tears, I swear.
