Roll back a bad DNS change safely

A DNS rollback is another DNS change. It can restore the authoritative source of truth, but it cannot erase values already cached by recursive resolvers, clients, applications, or negative caches. Recover by stopping new edits, preserving evidence, identifying the failed control plane, and restoring the complete approved prior state once.

Blast radius and owners

Control or service	Responsible owner
Incident command and stop decision	Incident owner
Authoritative zone and provider state	DNS-zone owner
Parent NS, glue, DS, and registrar state	Registrar and delegation owner
DNSSEC keys and signatures	DNSSEC owner
Recursive and client cache observations	DNS operations verifier
Web, TLS, redirects, CDN, and origin	Web application owners
MX, SMTP, sender policy, and queues	Messaging owners
Evidence, communication, and closure	Incident recorder and service owner

Rollback authority must be explicit. More editors do not make a recovery faster.

Prerequisites

• a named incident owner with authority to stop unrelated changes;
• exact approved change and deployment timestamps;
• captured prior RRsets, TTLs, delegation, DS, and provider-specific state;
• access to the authoritative source of truth and account recovery;
• identified affected names, record types, services, and user journeys;
• known old and new destinations kept available where possible;
• observers for authorities, named resolvers, and application layers;
• a DNSSEC owner when DS, DNSKEY, signing, or delegation may be involved; and
• an evidence channel that does not expose credentials or private data.

When the prior state is not known, pause. Guessing a historical value is a new forward change, not rollback.

Inventory and export

Before restoration, capture the failed state:

• change ticket, actor, source of truth, exact old and new values, and times;
• zone export or revision identifiers before and after the change;
• parent NS, glue, and DS observations;
• child SOA, NS, affected RRsets, DNSKEY, and signatures from every authority;
• TTLs learned before the change, current TTLs, and negative-cache parameters;
• named recursive answers with observer and timestamp;
• web addresses, certificate result, HTTP status, redirects, headers, CDN, and origin behavior;
• MX, sender-policy records, SMTP results, queues, and provider logs;
• monitoring alerts, geographic or network scope, and first failure time; and
• health and retention status of the prior destination.

Use the DNS Lookup pathway for named recursive observations. A recursive answer is evidence about that resolver at that time, not proof of the authoritative source or global state.

Preflight

1. Freeze unrelated DNS, registrar, CDN, certificate, and application edits.
2. Confirm whether the harmful state is at the parent, child authority, recursive cache, or application layer.
3. Query every intended authority directly and compare serials, values, and TTLs.
4. Confirm the captured prior state was healthy and is still safe to restore.
5. Confirm the prior destination still accepts the intended hostname, certificate, web traffic, mail, or API requests.
6. Inspect parent DS and child DNSKEY before any DNSSEC action.
7. Estimate overlap using the TTLs served before and after the failed change, plus negative caching where names or types were absent.
8. Define the one restoration action, verifier, success conditions, maximum wait, and escalation threshold.
9. Tell application owners that mixed cached answers are expected during the overlap window.

DNS caching is part of normal resolution architecture [1]. A cache is not necessarily malfunctioning because it still holds the previous or failed value.

Stop conditions

Stop before restoring when:

• the exact prior state or authoritative source is unknown;
• the prior endpoint, certificate, mailbox, or service is no longer available;
• authorities already disagree and the write target is unclear;
• the failure is only at the application layer and DNS remains correct;
• parent DS and child signing state are not understood;
• another responder is editing the same control plane;
• rollback would overwrite a newer approved emergency fix;
• the affected record is shared by services absent from the incident scope; or
• no verifier can observe authority and application health after the change.

During recovery, stop oscillating values after the first restoration. Escalate an unexplained result instead of alternating old and new state.

Ordered steps

1. Declare the incident scope, change freeze, owner, verifier, and communication channel.
2. Preserve failed authoritative, recursive, and application evidence.
3. Select the captured complete prior RRset or provider revision. Restore an RRset atomically where the provider supports it; do not rebuild one value at a time from memory.
4. Submit the restoration once through the authoritative source of truth.
5. Confirm every authoritative server publishes the restored value and an explainable SOA serial.
6. If the bad change was parent delegation, restore the complete approved NS set and required glue once, while keeping both authority sets online.
7. If DNSSEC is implicated, execute the state-specific parent-and-child recovery plan. DNSSEC authenticates DNS data and can turn a mismatch into a validating failure [3].
8. Keep old and new application destinations able to serve cached traffic through the overlap window.
9. Track named resolvers and user journeys without flushing caches you do not own or promising global propagation.
10. Make application-layer corrections only when evidence identifies a separate failure with its own owner and rollback.

Do not lower TTL after the incident and expect existing caches to shorten their stored lifetime.

Verification

Verify from authority outward:

1. every intended authority returns the restored complete RRset and TTL;
2. parent delegation, glue, and DS match the recovery plan;
3. child DNSKEY and signatures match any still-published DS;
4. named recursive resolvers move through explainable old, failed, and restored states over time;
5. negative answers expire according to the zone’s negative-cache behavior;
6. TLS covers the hostname through the SSL Checker pathway;
7. status and redirects match the approved behavior through the Redirect Checker pathway and HTTPClarity pathway;
8. email routing and policy match the approved state through the Email Security Checker pathway plus provider logs; and
9. monitoring and representative user journeys recover without hidden errors.

RFC 2308 defines negative caching, including how an earlier absence can remain cached after a record is restored [2]. Some resolvers can also serve stale data under defined conditions, so an expired-looking answer still needs observer-specific interpretation [4].

Rollback

This section handles failure of the attempted rollback.

• Do not automatically reapply the known-bad change.
• Preserve the restored authoritative state if it is internally correct, then determine whether remaining failure is cache, DNSSEC, TLS, origin, mail, or another application layer.
• If the prior endpoint is unhealthy, treat the next destination as a new forward fix with explicit approval, evidence, TTL analysis, and rollback.
• If authorities disagree after restoration, stop resolver testing and repair provider publication consistency first.
• If a cached DS conflicts with restored child signing, keep the matching key and signatures available or use the reviewed emergency parent process.
• If neither old nor new service is healthy, escalate to the service owners and choose the smallest coherent emergency state; do not keep editing DNS to mask an unknown application failure.
• Verify the final authoritative, recursive, DNSSEC, and application states, then record the remaining cache window.

Next step

After service stabilizes, preserve the timeline and compare expected with observed cache behavior. If resolvers still disagree, use Diagnose different answers from different resolvers before changing state again.