(by Paul Vixie)
Operating one’s own local DNS resolution servers is one of the simplest and lowest-cost things an IT administrator can do to monitor and protect applications, services, and users from potential risks.
A non-networked computer is very rare now because most useful work involves data and services that are distributed across some kind of “internet” — that is, any network that speaks IP, public or private. Notably, most non-internet networks have failed — we won’t be building or connecting to AppleTalk or DECnet or SNA or XNS networks ever again. Consider, then, that most useful work relies on TCP/IP, which itself relies on the Domain Name System (DNS). We should know as much as possible about our use of the DNS, but most IT administrators are much too busy learning their specialized craft to also become experts on every enabling technology.
In this article, I will describe the use of DNS by users and applications, which means DNS resolution. There’s much more than this to say about DNS — for example, DNS content management and provision. But while providing DNS content is a specialized task, accessing that DNS content through resolution is a universal activity — no work of any kind can begin in TCP/IP networks until DNS has been accessed at least once and possibly many times. Importantly, for this discussion, among the kinds of “work” I refer to are online crime, network abuse, and surveillance.
In the 1980s, when TCP/IP’s design was mostly finalized by the research and education communities, it was common to have only one or a few computers on a campus because computers were very large and expensive to acquire, operate, and maintain. This concentration caused many services and applications to share a single computer, which users would access using “dumb” terminals. Thus, when the DNS was created, the resolution service was installed on at least one computer on every campus. We learned to expect rapid answers to most DNS questions, usually receiving an answer within a millisecond or so. Networked applications such as e-mail or file service were operated under the assumptions that at least one DNS access would be necessary for every “transaction” but that DNS access would be fast enough that this would not reduce the performance of applications and services that depended on it.
Before the commercialization and privatization of the Internet in the 1990s, there was no online crime or abuse. When only scientists and engineers could access the Internet, and the goals were academic rather than commercial, we had safety by accident. Much of today’s Internet crime and abuse is made possible by the total absence of security consideration in some of the Internet’s oldest core protocols and services — and it has proved very difficult to add security considerations decades later to a network that was designed to accommodate only trusted users and trusted applications. One of the most serious security problems is that malicious actors can create DNS content to facilitate their online crimes, and they will receive the same excellent service for this malicious content as we get for non-malicious content.
Continuous consolidation among Internet companies has created some dominant players, such as Google, Cisco, IBM, and Cloudflare, each of which offers a global DNS resolution service identical to the resolution service every network had to run for itself back in the early days. This is in some ways a natural progression because these external DNS resolution services (such as 220.127.116.11, 18.104.22.168, or 22.214.171.124) are free, convenient, and, perhaps unknown to most users, also a vital source of network intelligence for the companies that operate these resolution services. Many IT administrators working today were not working in the field when DNS resolution service was always provided on-campus, under local policy, by local staff. As a result of this trend and that generation gap, there are added costs and subtracted benefits, or losses, from the use of external DNS resolution servers, either the global kind (126.96.36.199, et al.) or the regional kind (as provided by one’s own ISP).
Accounting of Losses
The first loss we experienced from externalizing the once-local DNS resolution service was privacy. Our external DNS transactions are rarely protected from surveillance, and while such protection is now being developed, that protection will come in the form of added complexity. The best way to avoid having one’s DNS transactions observed, tracked, or analyzed by third parties is to not externalize those transactions in the first place. Most of us need not fear surveillance of traffic inside our own networks, which makes those “owned networks” a great place to keep data we don’t want published.
The second loss from externalizing DNS resolution is performance. No matter how many DNS resolution servers are externally constructed, none will be reachable by our users and our applications in less than 1 millisecond. Thus, the number of transactions we can process per unit of time will be reduced by the need to wait for speed-of-light transmission delays across a distance greater than our campus. Most web browsers now contain an internal DNS “cache” to offset this penalty. Most other networked applications lack this cache. Again, I question the need for application-level caching when a simple local DNS resolution server would offer the same benefits to all networked applications across an organization.
The third loss we experience as a result of using external DNS resolution servers instead of operating such servers locally on our own campus is in monitoring. Most malware and most botnets make use of the DNS to reach their command and control operators, and some use the DNS as a stenographic exfiltration channel for the victim’s personal information. Every modern DNS resolution service has some way to monitor “outbound” traffic to detect insider threats. However, to get the benefit of such monitoring, it’s necessary to see your own traffic. If every application running inside the network has a direct relationship with an external DNS resolution service, then only the operator of that service — not the organization’s IT administrator himself — will be able to monitor that traffic for threats. Some of these resolution service operators offer monitoring to their users, but most do not, and those that do sometimes monetize their observations of that traffic. (This is known as “surveillance capitalism.” For example, the DNS queries coming from the customer of an ISP might be data mined for keywords that are then used to send focused advertising content to that customer.)
The fourth loss we can experience from using an external rather than a local DNS resolution service is control. Modern DNS technology includes the capability of policy enforcement, whereby malicious DNS patterns are rejected by the resolution server based on policy settings (from the local security operations center) and policy subscriptions (from external security information providers). Policy of this kind can reject resolutions based on known-bad domain names, name server names, name server addresses, or distant mail/web/content server addresses. However, this incredibly powerful capability is available only to operators of local DNS resolution servers. Some external DNS resolution providers offer this kind of policy filtering but without the fine-grained control that a locally operated server can provide.
Despite this growing shift toward external DNS resolution services, it’s not necessary to completely choose only one way that DNS resolution is provided to all local users and servers. A mixture of internal and external DNS resolution consumers is not only possible but quite common. The DNS resolution service to be used by a server is usually set in its control panel or in the private cloud’s control panel, whereas for a desktop or laptop or mobile device, it is set in the control panel of the DHCP server. So, it’s possible to experiment and to experience a very gradual and surprise-free transition from use of external DNS resolution servers to internal ones.
Every open source server platform, such as Linux or BSD, offers many free implementations of the DNS resolution service. The oldest of these is called BIND, but newer implementations such as PowerDNS, Unbound, and Knot are also well-trusted, production-ready software packages. Most will offer some kind of template configuration that includes local DNS resolution. While you certainly will tune and enhance that configuration over time, the “jumping in” cost is extremely low. Commercial products are also available for DNS resolution, from vendors such as Microsoft, Infoblox, Nominum, BlueCat, and others.
ADVERTISEMENT. CLICK FOR SOUND.
Diversity is essential. Every campus needs at least two independent DNS resolution servers, ideally situated on different LAN segments with different power sources. Safe configuration is also essential — operators must ensure that no query coming from outside the network will be answered because this is a well-known and quite popular method for attackers to amplify their distributed denial-of-service capacity. It’s worth spending half a day studying documentation, HOWTO files, and forums before turning up any new service — but that’s especially true for a DNS resolution service.
The general outline of activities related to establishing and operating a local DNS resolution service on your campus is as follows:
- Research and experiment to find a solution and configuration you can live with initially.
- Deploy at least two diversified internal servers, which are probably virtual servers.
- Configure management to cause software, configurations, and updates to be automated.
- Conduct testing, from both on-campus and off-campus, to ensure correct (and onlycorrect) operation.
- Monitor, including both operational events like reachability, but also transaction logging.
- Reconfigure a small set of test subjects, including some servers and some end users.
- Maintain a watchful settling-in period to gain measurable confidence about the new methodology.
- Reconfigure a larger set of test subjects or, perhaps, the remainder of the campus.
- Deploy a testing server for experimenting with automated logging and policy filtering.
- Research and experiment on different alternatives for transaction monitoring.
- Research and experiment on different alternatives for resolution policy filtering.
- Perform a gradual rollout into production of new solutions that have good results from experimentation.
Operating one’s own local DNS resolution servers is one of the simplest and lowest-cost things an IT administrator can do to monitor and protect their applications, services, and users from potential risks. These risks — including surveillance capitalism, unmanageable external dependencies, attacks carried via DNS, and attacks that could be detected via DNS — have a much higher potential cost than the mitigation strategy outlined here. Additionally, the DNS resolution service is so central to every other IT-related activity that any and all IT administrators who take the time to investigate and master this technology will amplify their effectiveness and the value they bring to their enterprise.
This article was initially published HERE.