Horse Sense #142

How the Internet Broke on Friday 10/21/2016

Twitter, Netflix, and a number of other popular web sites could not be reached for several hours last Friday. The wires were fine. The servers were fine. So what was the problem?

"Jenny I've got your number" --Tommy Tutone

A key building block of the Internet is called the Domain Name System (DNS). DNS phone book servers translate names like into numbers a computer can understand: an Internet Protocol (IP) address like On Friday, lots of Internet of Things (IoT) devices, like cameras and digital video recorders, were hijacked and asked phone book servers run by the company Dyn to look up IP addresses for them. Dyn servers had so many requests, they simply could not respond in a timely matter to anyone. When computers could not get the IP address they needed, they could not find their way on the Internet. Web sites became unreachable. All the connections and web servers were fine. If you typed the IP address instead of the name into the browser, the web servers would have responded. Unfortunately, unless you already knew the IP address, you would need to use a DNS phone book server to find it.

Distributed Denial of Service (DDoS) attacks like the one on Friday are pretty hard to stop if you are the target. If an army of machines sends gibberish to a target, that target's Internet link or server will be overwhelmed and disappear from the Internet. These brute force attacks can even overwhelm the target's Internet Service Provider's (ISP's) ability to service other clients by flooding their lines. ISPs will commonly block attacking traffic, but that means the target still is disconnected from the Internet.

Making Dyn the target of the DDoS attack knocked out thousands of web servers at once and millions of people experienced the effects.

The attack was much more successful than it should have been. This is because many companies, like Twitter, have ignored best practices and specified only one company, Dyn, to hold their phone book information. Had they specified an additional phone book provider, and there are millions, to hold phone book information, they would not have gone dark. Making such a change is trivial and DNS phone book services are extremely inexpensive.

Twitter and many of the other companies made their DNS problems worse than they might have been. When you get phone book information from a DNS server, it tells you how long that information should be considered good. Once you cross that time threshold, the information is not to be trusted, so it is thrown away. This allows people to move web and e mail servers around on the Internet from one IP to another. You use the same name, but the number changes. Setting an expiration time also allows you to do some load balancing between multiple servers on the Internet as the initial response may be to use IP address A and the next response is to use IP address B. But.... If the expiration time is too long and you move a server from one IP to another, people who have the old IP address information will keep trying to go to that address and get no response. If the time is too short, you will needlessly ask phone book servers over and over for the same IP address. This puts additional load on DNS phone book servers and increases the likelihood that you might have an issue getting a response. Those companies who had short expiration times in the Dyn phone book had more problems than they might have had otherwise. Companies that used longer expiration times experienced fewer problems because the information people needed could have already been learned before the attack and marked as still valid, so no look up was needed. Until that information expired, these customers could reach the web sites they wanted.

Many companies use an additional trick that can cause problems as well, redirection. When you look up, the answer comes back that there has been a name change, so look up instead. Now you have to do *two* DNS lookups. Not only can this cause delays in displaying web sites, if either lookup fails, you cannot get the IP address you need. For example, if the first DNS server responded with a new name that only Dyn knew about, you would be out of luck.

Think that this could not happen to you? These are major corporations and they had (and still have) DNS configuration issues. Many federal government sites do as well. 95% of the time I see something in an organization's DNS that can be improved so that their mail is delivered more reliably and with less chance of being spoofed, so they can stay "on the air" more reliably, etc. Years ago, I tried to get the FBI to fix DNS and e mail issues. They thought they were fine. Six months later, massive numbers of scam e mails went out that looked like they came from the FBI. It turned into a denial of service attack that basically cut their Internet connection.

If you want to know more about DNS, I recommend the very readable "DNS and BIND" book. Cricket Liu and Paul Albitz are now in the 5th edition of this book and it is the key reference work for anyone on the subject of DNS.

Forget Me Not

Anything that connects to a network is likely to have some ability to communicate as well. That means it is a "computer," whether it is a camera, network switch, a DVR, a refrigerator, a sprinkler system, or a printer. Collectively people tend to call these, Internet of Things (IoT) devices. Do IoT devices need to talk to the entire Internet? No. But, they are normally set up without restrictions. Furthermore, firewalls could keep these devices from going where they should not, but almost no one configures a firewall to control traffic going out of the network, they just try to control what comes in. I would argue that the reverse thinking is more appropriate. You care more about what leaves your network than about what enters it.

There is no reason that your refrigerator should be asking a DNS phone book server where Netflix is. Nor should it try to send mail to 10,000 people. But, on 99% of the networks I see, they can do exactly that.

IoT devices are a great place to mount attacks. Typically, no one is watching them. They just sit on the network and people forget about them. This means these computers can be used to mount attacks on your network or to attack someone else, like Dyn. Very little attention has been paid to these devices. They are literally out of sight and out of mind. This makes them a perfect back door for nefarious characters. We might notice and think it is funny when a refrigerator or TV in a big box store starts spontaneously showing porn. We think it will not happen to us. This attack on Dyn shows it can and will.

Bigger Problems Ahead

DNS is critical to connecting from one device to another, even on local networks. But, as people start moving more and more of what used to be local resources onto the Internet (into the cloud), a disruption in DNS becomes a very big deal. If your e mail goes down for a while, that could be a problem. But, what if your phones and accounting system do as well?

We cannot protect against every type of attack in all circumstances, but we can do a lot to protect ourselves and others against DDoS attacks beginning with better DNS and firewall configurations.

If you want to talk to someone about your DNS or your security, Iron Horse can help.

©2016 Tony Stirk, Iron Horse