The internal service works for only part of the team: how to fix private DNS without a risky cutover

When “reconnect the VPN” works only by accident

The workday starts with a familiar complaint: an internal application will not open. It works for some colleagues but times out for others. Reconnecting the VPN sometimes fixes it, yet nobody can explain why.

The investigation soon turns into an exhausting search for someone to blame across the network, VPN, cloud, and application teams. The service itself may be healthy. The inconsistency often appears earlier: clients send the same DNS query through different paths and receive different addresses.

Do not begin by flushing every cache or replacing every resolver. Trace one failed query first. That will show where the expected answer changed, disappeared, or reached the wrong group of users.

Trace one query from start to finish

Choose the hostname of the failing service and two devices: one where it works and one where it fails. For each device, record the network, VPN state, configured recursive resolver, returned answer, TTL, and test time.

Next, determine where the resolver sent the query. It might reach a private authoritative zone in the office, a separate cloud zone, or public DNS. If the same name returns different addresses, establish whether this is intentional split-horizon DNS or an accidental result of two independent configurations.

A simple map is useful:

client → office or VPN → recursive resolver → routing rule → authoritative zone → answer

Add cloud networks, backup resolvers, and public DNS. The map should show real query paths rather than product names.

Find conflicts and hidden rules

Create an inventory of private and public zones. For each zone, record its owner, editing location, publication servers, and intended client groups. Then compare records for identical names.

Three situations deserve particular attention:

• the same zone is maintained manually in both the office and the cloud;
• an old resolver contains an undocumented rule used only by VPN clients;
• a private name is missing from the internal zone, so the query unexpectedly reaches public DNS.

A DNS view should be an explicit choice, not a side effect of network topology. Office devices and managed VPN clients might receive a private address, while external clients receive a public one. Document the signal that selects each DNS view and the priority used when more than one rule matches.

Create one write path for DNS changes

The main cause of configuration drift is not the number of DNS servers. It is the number of places where people independently edit the same record. The target design needs one controlled source for changes. Records may be published from it to several zones or environments, but parallel manual editing should stop.

Cloudflare’s Internal DNS announcement describes one current implementation: separate recursive and authoritative roles, internal zones, DNS views, and resolver routing policies. It is an example, not a required platform. Every implementation must answer the same vendor-neutral questions: who receives the query, which zone answers it, which rule selects the response, and what happens when no internal match exists?

Public DNS fallback should not be enabled unconditionally. Decide for each private zone whether a public answer is acceptable. A typo in a private hostname should not silently become an external query.

Migrate in waves instead of making one large cutover

Start by collecting baseline measurements: successful response rate, latency, common NXDOMAIN responses, and answers for known test names. Lower the TTL of records that will change well before the cutover. Reducing TTL at the moment of migration does not remove answers already cached under the old value.

Move a small pilot group first. Include clients with different access paths, such as office, VPN, and remote networks. Compare actual DNS answers, not just whether the application opens. Define promotion criteria in advance: the expected address for every group, acceptable latency, no leakage of private names, and a stable successful-query rate.

Keep the previous route available for rollback, but do not continue editing both the old and new zones. After a successful pilot, expand coverage in separate waves. If the measurements deteriorate, point clients back to the previous resolvers and remember that caches may retain the new answers temporarily.

Test failures before users encounter them

Run at least the following checks for every DNS view:

• an existing private name returns the correct address;
• a nonexistent private name returns the expected NXDOMAIN and does not leak outside;
• a public name continues to resolve normally;
• an unavailable authoritative server produces a predictable result;
• a stale cache updates after its TTL expires;
• a device outside the VPN does not receive a private answer;
• a client matching several rules reaches the correct DNS view.

Do not treat NXDOMAIN, a timeout, and an unavailable-server error as the same condition. They may look similar to the user, but they point to different faults and require different fixes.

Anti-patterns that bring drift back

Do not perform a large cutover without a pilot group. Do not use cache flushing as the main migration plan. Do not leave DNS view selection rules only in administrators’ memories. Do not maintain the same zone manually in several systems. Do not assume that automatic public DNS fallback is safe for every private name.

A migration is not complete merely because the new resolver answered one test query. It is complete when the team knows the source of every record, can see the query path, can verify the rules, and has tested the rollback.

A short migration-plan template

Record the current route, conflicting zones, target DNS views, single write location, test names, and baseline measurements. For each migration wave, specify the client group, start time, success criteria, decision owner, and exact rollback action. Separately document when the old route can be retired and how long caches and errors must be monitored afterward.

Sources

• Cloudflare, “Internal DNS is now generally available”: https://blog.cloudflare.com/internal-dns/