The Site Works for Some Users but Not Others: Finding a DNSSEC Key Rollover Failure

DNSSECincident responsenetwork infrastructure

An on-call engineer's desk with two keys and a broken chain representing a DNSSEC incident

The Site Works — but Not for Everyone

Imagine the night shift after a planned DNSSEC key rollover. The site opens from the office. It fails over a mobile connection. One DNS server returns an address, while another returns SERVFAIL. Application monitoring is green and the server is reachable, yet some users still cannot get in.

The website itself may be fine. The failure can happen earlier, while a DNS server is deciding whether it can trust the answer. The address is available, but DNSSEC refuses to treat it as authentic.

Cloudflare’s report on the .AL incident describes this class of failure: a new DNSKEY was published in the zone, but the parent DS did not point to the expected key. Some resolvers still had old data in cache while others saw the new state. The result looked random to users even though it had a precise cause.

What Actually Broke

Think of DNSSEC as checking a key against a trusted key list. The domain publishes a public DNSKEY. Its parent zone publishes a DS record that tells resolvers which key to trust.

Those two pieces must agree after a rollover. If the zone publishes only the new DNSKEY while the DS still identifies the old one, validation fails. The authoritative DNS server can return the records, but a recursive server that checks signatures rejects them and answers SERVFAIL.

The data exists and the network works, but the user still cannot receive an authenticated answer.

Compare Before You Restart Anything

Do not flush caches or disable DNSSEC first. Preserve evidence: record the time, full response, and server address for every check. Then compare at least two independent recursive DNS servers with an authoritative source:

dig example.ua NS
dig +dnssec example.ua DS
dig +dnssec @<authoritative-IP> example.ua DNSKEY
dig +dnssec @<resolver-IP> example.ua A
dig +dnssec +cdflag @<resolver-IP> example.ua A
delv example.ua A

Focus on three clues:

  • If a normal query returns SERVFAIL but +cdflag still receives signed data, signature validation is the likely failure point.
  • If the authoritative server answers while recursive servers disagree, compare DS, DNSKEY, and cached values.
  • If the published records are now correct but failures continue, check the TTL. Old data may still be sitting in caches.

No single clue proves the cause. Match the DS to the intended DNSKEY, then check algorithms, signature validity periods, and key flags.

Wait, Fix, or Roll Back

Choose based on the state the internet can see, not simply on which action feels fastest.

Wait when DS and DNSKEY already agree and the remaining errors fit a known TTL. Set the next check time immediately.

Fix the current state when the new key is correct but the parent DS has not been updated. This requires the registrar or registry; restarting the web server cannot solve it.

Roll back when the old key is still available, the DS still trusts it, and restoring it will not create another signature failure.

An NTA Is an Emergency Bridge, Not a Repair

If the proper repair will take time, the operator of a recursive DNS service may apply an NTA: a temporary exception that skips DNSSEC validation for one zone. This can restore availability, but it also removes the cryptographic authenticity guarantee while the exception is active.

Use an NTA only when the impact is serious, the authoritative data has been verified independently, a quick zone repair is not possible, and the exception has an owner and exact automatic expiration time. A site owner cannot force external DNS services to apply one.

EDE 33 can make this workaround visible. A DNS server may attach the code so logs and monitoring show that it returned data only because a temporary exception was active. A missing EDE 33 does not prove that no NTA exists; not every server and client supports the code.

How to Know the Incident Is Over

Retrieve DS and DNSKEY again through independent servers, verify signatures, and wait for the required TTL. Remove the NTA, then repeat the queries without +cdflag through several DNS servers.

Do not extend the exception for “just one more hour” without a new decision. The action plan needs an owner, end time, rollback criteria, registrar or registry contacts, and a clear recovery check. Keep emergency contacts outside the affected DNS zone, or they may disappear along with the site.

What Not to Do

Do not disable DNSSEC globally because one zone is broken. Do not treat an authoritative answer as proof that the full chain of trust is healthy. Do not remove the old DNSKEY before the DS and caches have moved to the new state. Most importantly, do not confuse two different goals: bringing the site back online and restoring authenticated DNS data.

Sources

Quick checklist

  • compare answers from at least two independent DNS servers
  • match the parent-zone DS with the current DNSKEY
  • record test times, TTL values, and signature validity periods
  • name the owner of the repair or temporary exception
  • limit any exception to one zone and a precise duration
  • repeat the checks and remove the NTA after recovery

Create an Action Plan for a DNSSEC Incident

Help an on-call team investigate a DNSSEC incident and create a short action plan. First ask for: the domain or zone; the time errors began in UTC; authoritative and recursive DNS server addresses; known DS and DNSKEY records; the latest key rollover plan; service impact; available registrar or registry contacts; whether an NTA can be configured; and the maximum acceptable exception duration. Do not assume that SERVFAIL automatically means a DNSSEC problem. Explain each conclusion in plain language: what we observed, what it may mean, and what remains unknown. Do not recommend disabling validation globally. Return the result in this format: 1. What is known and what information is missing. 2. Verification commands with plain-language explanations. 3. The most likely location of the failure. 4. A recommendation: wait, fix, roll back, or apply an NTA. 5. If an NTA is needed, its scope, owner, and automatic expiration time. 6. How to confirm recovery and remove the exception safely. 7. A short update for the team.