AWS Account Purchase: Amazon Cloud DNS Resolution Exception Troubleshooting and Resolution Depth Guide!
In the cloud computing architecture, DNS (Domain Name System) is like the navigator of the digital world. Once DNS resolution on AWS is abnormal, enterprise services may instantly fall into a "lost connection" state: EC2 instance cannot access external API, internal microservice call timeout, user encounter
NXDOMAIN
or
SERVFAIL
Error reported.
Because AWS's network architecture integrates VPC (virtual private cloud), Route 53, VPC Resolver(.2 resolver) and possible local hybrid cloud architecture, DNS troubleshooting often has certain complexity. This article will be from
Public Network Resolution
,
VPC internal resolution
,
Hybrid Cloud Architecture
and also
performance bottleneck
Four dimensions, for you to disassemble the AWS DNS resolution exception troubleshooting and resolution process.
1. core troubleshooting toolbox: the first "stethoscope"
Before digging into the architecture, when encountering a parsing exception, first use the following native tools on the affected EC2 instance or local endpoint to obtain the underlying error status:
Use dig yourdomain.com (Linux/macOS) or nslookup (Windows) to observe the returned Status. Common states include: NOERROR: the resolution was successful, and the problem may be that the application layer cache or port is not available. NXDOMAIN: The domain name does not exist, usually the record is mismatched or has not been synchronized. SERVFAIL: The DNS server fails to process the data. This error occurs when the DNSSEC key is incorrectly configured or the upstream forwarding fails.
The specified DNS server test forces the authoritative name server of AWS Route 53 to query to determine whether it is a global recursive resolution problem or an AWS side problem: Bashdig @ ns-xxxx.awsdns-xx.com yourdomain.com
2. public network environment: Route 53 public hosted zone resolution exception
If you are responsible for multi-account management in enterprise cloud operation and maintenance, you may need to frequently adjust domain name assets. Before the formal investigation, ensure that your account has sufficient management rights. If it is an enterprise-level large-scale deployment, ensure that it is carried out through a compliant and stable channel.
AWS account purchase
Maintenance to obtain an enterprise-level support plan (Enterprise Support), which is critical when encountering an underlying failure of a non-configuration class.
The public network resolution fails, which is usually caused by the following three reasons:
1. The Registrar's Name Servers (NS) is inconsistent with Route 53
This is the most common human error. When you create a new public hosted zone on Route 53, a
WS randomly assigns 4 name servers.
Troubleshooting method: check the root domain name authorization chain through dig trace yourdomain.com, or use the Whois tool to view the NS record filled in by the registrar.
Solution: Log in to your domain registrar backend and modify the NS record completely and accurately to the 4 AWS name servers currently displayed in the Route 53 hosted zone.
2. DNSSEC signature configuration broken chain
For security, many companies have started DNSSEC. If the DS(Delegation Signer) record is not updated at the registrar when the domain name is migrated or the hosted area is changed, the global recursive resolver will return directly because it cannot verify the signature.
SERVFAIL
.
Solution: If not, close DNSSEC synchronously on the registrar and Route 53 side. If necessary, export the KSK (Key Signing Key) information of Route 53 and go to the registrar background to regenerate the matching DS record.
3. Geographical location/delay routing and health check linkage lead to "manslaughter"
Route 53 supports advanced routing policies. If Assess target health (Evaluate Target Health) is configured, Route 53 will no longer return the IP if the bound backend (such as ALB or EC2) health check fails.
Solution: Go to the Route 53 console to check the health checks status of the association. If the backend service is normal but the check determines that it is dead, adjust the health threshold or timeout period of the health check.
3. VPC internal environment: AmazonProvidedDNS exception
In the VPC, EC2 uses the network segment base and
2
The IP address of (e. g.
10.0.0.2
) as a DNS server, that is
AmazonProvidedDNS
(or Route 53 Resolver).
1. VPC attribute is not enabled
If you find that the EC2 instance cannot resolve the AWS internal service domain name at all (such
* .amazonaws.com
) or private hosted zone (Private Hosted Zone) domain name, check the VPC configuration first.
Troubleshooting and resolution: Open the Amazon VPC console. Select your VPC and click Actions (Actions)-> Edit VPC settings. Ensure that DNS resolution is enabled (Enable DNS resolution) and DNS hostnames are enabled (Enabl
e DNS hostnames) These two options have been checked.
2. DHCP Options Sets tampered
If an enterprise uses a self-built Active Directory (AD) or BIND server internally, the DHCP options set of the VPC may be modified. If custom DNS cannot resolve the public or AWS internal domain name, the resolution will be interrupted.
Solution: Check the domain-name-servers in the DHCP options set associated with the VPC. If custom DNS is configured, you must ensure that the custom DNS server is configured with a conditional forwarder (Conditional Forwarder) that forwards requests for AWS internal and external domain names back to the. 2 server in the VPC.
4. hybrid cloud and cross-account environments: Resolver endpoint failures
In a hybrid cloud scenario where an on-premises data center and an AWS VPC are connected via AWS Direct Connect (DX) or VPN,
Route 53 Resolver endpoints (Endpoints)
.
[On-premises Data Center DNS ] ─ ─(VPN/DX)─> [Inbound Endpoint (.2 Resolver)] ─> [ VPC Private Hosted Zone]
When hybrid cloud resolution is interrupted, follow these steps to check the network and permissions:
1. Verify network connectivity: Confirm that the physical/network layer is unobstructed.
On the local DNS server, use
telnet <Inbound-Endpoint-IP> 53
or
nc -puv 53 <Inbound-Endpoint-IP>
Test port 53 for TCP and UDP separately. If it times out, the underlying traffic has not arrived.
2. Check the security group and network ACL: Release traffic.
Inbound endpoints (Inbound Endpoint) consume ENIs (elastic network cards) within the subnet. Check the security group (Security group) bound to the ENI and the network ACL(NACL) of the subnet to ensure that they allow
UDP/TCP Port 53
Inbound traffic, while temporary port (1024-65535) traffic for the backhaul is released.
3. Troubleshoot routing tables and NAT gateways: Check the outbound path.
If the outbound endpoint (Outbound Endpoint) cannot resolve the local domain name, you need to confirm: Outbou
Whether the route table of the subnet to which the nd Endpoint belongs has a route pointing to the local CIDR block (through the virtual private gateway VGW or the Transit Gateway). If the outbound destination is a public network public DNS (such as 8.8.8.8), the subnet must be a private subnet and the route table must be configured with a NAT gateway.
4. Cross-account private hosting zone association: Check IAM and sharing permissions.
In a multi-account architecture, if the domain name is in the private hosting zone of account A and EC2 is in the VPC of account B, it must be executed through AWS CLI or SDK
associate-vpc-with-hosted-zone
Cross-account association is performed, otherwise the VPC of account B cannot sense the existence of the domain name even if the network is smooth.
5. stealth killer: VPC DNS per second per limit (Throttling)
Sometimes, DNS resolution exceptions appear
Intermittent packet loss, timeout, or high latency
This is especially true during peak periods. This most likely triggered AWS's packet per second (PPS) limit.
⚠Critical Quota Limit: AWS stipulates that DNS queries initiated by each network interface (ENI level for each EC2 instance at the network architecture level) to AmazonProvidedDNS(.2 server) are limited to 1024 PPS. DNS requests that exceed this limit will be discarded by the system. This is a hard limit and cannot be raised by submitting a ticket.
How to diagnose DNS rate limiting?
Since the rejected traffic is dropped at the network interface layer, they
I won’t.
in the query log for Route 53.
Diagnosis method: Use tcpdump to capture port 53 traffic on the affected Linux EC2 instance, or use the traffic mirroring (Traffic Mirroring) function of the AWS Nitro architecture instance to import the traffic into the Wireshark for analysis. If you find that DNS requests are dense and there is a large area of Unanswered Ports, you can confirm the diagnosis of Throttling.
Solution:
Enable local DNS caching: Deploy nscd, dnsmasq, or local systemd-resolved on a Linux instance. The vast majority of duplicate DNS requests are digested locally in the operating system and are no longer submitted to AWS.
Optimize the application layer connection pool: Some development languages (such as Java) have a short default DNS cache TTL (time to live) setting, or high concurrent short connections frequently trigger DNS resolution. It is recommended to optimize the long connection architecture or extend the DNS cache expiration time of the application.
Summary and recommendations
Purchase an AWS account
In the face of AWS DNS resolution exceptions, do not blindly modify the configuration. It is recommended to follow the principle of "from inside to outside, from shallow to deep": first look at the local cache and tool feedback, then look at the basic attributes of VPC DNS, then check the security group and hybrid cloud routing, and finally consider 1024 the PPS quota limit in large traffic scenarios. At the same time, the establishment of good infrastructure is the code (IaC, such as Terraform) management habits, standardize the enterprise multi-account management flow, can help you from the source to prevent the vast majority of DNS disasters caused by misoperation.