Alibaba Cloud Recharge Channel: Common Causes and Solutions for SLB Health Check Failed
In the daily work of enterprise-level website operation and maintenance and SEO optimization, the most headache for the team is "the website cannot be opened occasionally", "the access delay suddenly soars" or "the feedback connection of users in some areas is reset".
As an SEO website optimizer, I am well aware of the fatal impact of website usability (Availability) and stability (Stability) on search engine rankings. If the search engine Spider (such as Googlebot or Baidubot) frequently encounters 502 Bad Gateway or 504 Gateway Timeout when crawling your website, the search engine will quickly determine that your website is "unreliable", directly reduce the index crawling frequency, and even significantly reduce the keyword ranking.
In Aliyun's cloud computing architecture, such alarms often originate from the same behind-the-scenes hand--
Alibaba Cloud SLB, ALB, and NLB health checks frequently fail.
.
When the health check fails, SLB considers a back-end ECS instance as "dead" and stops distributing traffic to it. However, if the health check frequently jumps between "success" and "failure", traffic will be cut off and cut back continuously, and front-end users and search engines will encounter a large number of abnormal errors when crawling spiders.
Today, I will analyze the common causes of frequent failure of Alibaba Cloud SLB health check from architecture troubleshooting, network transmission, backend configuration to cloud resource maintenance, and provide a set of effective solutions.
1. why SLB health checks are important for SEO and business?
Before in-depth technical investigation, we first clarify the operation logic of SLB health check.
Alibaba Cloud SLB periodically sends probe requests (such as HTTP GET and TCP handshake) to backend ECS instances to evaluate the health status of the backend.
Normal state: SLB evenly distributes front-end requests to all back-end servers.
Health Check Failed: SLB determines that an ECS instance is abnormal, automatically isolates it, and does not send requests to it.
If the health check is not configured properly or the back-end server has hidden dangers, it will appear.
Frequent health check failures/alternate jumps
. This will not only lead to a sudden increase in the load of a single server and slow response of the whole station (lowering the TTFB index in Core Web Vitals), but also cause intermittent inaccessibility of website pages.
In addition, when performing large-scale cloud resource deployment and business expansion, the operation and maintenance team often focuses on SLB configuration itself, but ignores the management of cloud account infrastructure. In the early stage of the project or in the operation and maintenance cycle, it is important to do it well in advance.
Alibaba Cloud account top-up
and budget planning to ensure that the account is adequately funded, avoid
Due to overdue payment, SLB monitoring rules are invalid, public network bandwidth is limited, or cloud monitoring alarms are suspended, thus concealing the real health check failure.
Six Common Causes and Solutions for Frequent Failure of 2. SLB Health Checks
Based on my many years of site tuning and troubleshooting experience, SLB health check failures can usually be attributed to the following six core reasons:
1. The security group/firewall has blocked the SLB IP address.
This is the novice operation and maintenance and webmaster most easily step on the pit.
Principle and symptoms:
SLB internally passes through a specific private network IP segment (such
100.64.0.0/10
etc. Aliyun reserved network segment) to initiate a health check request to the back-end ECS. If your ECS instance is enabled
iptables
,
ufw
,
firewalld
, or configured in the Alibaba Cloud console
Security Group Rules
If these private network IP addresses are intercepted by mistake, SLB cannot receive a normal response from the backend.
Solution:
Log on to the Alibaba Cloud ECS console and check the security group to which the instance belongs.
In the inbound rules, ensure that SLB's health check IP network segment is allowed to access the backend port (such as HTTP 80, 443, or TCP custom port).
Log in to the ECS system, check the local firewall settings, and add the detection network segment of SLB to the white list: Bash# Take iptables as an example, allow intranet network segment to access iptables -A INPUT -s 100.64.0.0/10 -p tcp -- dport 80 -j ACCEPT
2. The back-end Nginx/Web service configuration or path (Path) returns a non-2xx/3xx response.
Principle and symptoms:
For HTTP/HTTPS listeners, SLB sends the "health check path" set to the backend (the default is usually
/
or
/check.html
) to send a request. By default, SLB only returns
HTTP 2xx or 3xx
The status code is considered successful.
If your backend is configured with forced pseudo-static redirection, unauthorized access blocking (401/403), or the default home page report 404,SLB will determine that the health check fails.
Solution:
Set up a health check page: Do not use the homepage of the website as the health check path. It is recommended to create a lightweight static file (for example,/healthcheck.html) under the root directory of the Web service, and write the content to OK.
Test backend return: Use the curl command on the ECS to test the path: curl -I ht tp:// 127.0.0.1:80/healthcheck.html
Ensure that the returned HTTP response header is HTTP/1.1
200 OK.
Adjust the domain name header (Host Header): If your Nginx is configured with a multi-site virtual host (Virtual Host) and bound to the specified server_name, the default request initiated by SLB may be matched to the default_server by Nginx and return 403 or 404 because it does not have the correct Host header. In this case, you need to explicitly enter the health check domain name in the SLB health check advanced configuration.
3. Back-end ECS system resources are exhausted (CPU/memory/IO soars)
Principles and Symptoms:
If the website suffers from sudden traffic, CC attacks, slow queries, or memory leaks, resulting in the CPU usage of ECS reaching 100 percent or memory exhaustion (OOM), the Web server (Nginx, PHP-FPM, or Java) will not be able to respond to SLB probe requests in a timely manner, resulting in health check timeout.
Solution:
Check the CPU, memory, system load, and disk I/O curves of Alibaba Cloud Cloud Monitoring (CloudMonitor).
Log on to the ECS terminal and use top or htop to view the process that uses the most resources.
If the resources are insufficient due to normal business growth, you should upgrade the ECS specifications or add back-end nodes in a timely manner. At the same time, ensure the stability of the account capital chain and complete the Alibaba Cloud account recharge in a timely manner to avoid the forced shutdown of the node due to the failure of temporary pay-as-you-go instance deduction.
4. Timeout time (Timeout) and health check interval settings are unreasonable.
Principles and Symptoms:
SLB allows you to customize the Response Timeout, Health Check Interval, Health Threshold, and Unhealthy Threshold ".
If the back-end response time occasionally takes 2 seconds due to heavy business logic, and you set the "response timeout time" of SLB to 1 second and the "unhealthy threshold" to 2 times, then as long as two consecutive detections are slightly stuck, SLB will immediately determine that the node is faulty, causing frequent health check failures.
Solution:
Reasonably optimize the SLB health check parameters, and it is recommended to use a relatively smooth parameter combination:
Response timeout: It is recommended to set it to 3 to 5 seconds (leave a certain buffer time for the backend).
Health Check Interval: 2 to 5 seconds is recommended.
Unhealthy threshold: set to 3 times (that is, 3 consecutive failures are completely isolated to prevent accidental jitter and misjudgment of the network).
Health threshold: set to 2~3 times.
5. The number of concurrent back-end connections reaches the upper limit or Keep-Alive problem
Principles and Symptoms:
The health check of the HTTP protocol frequently establishes and disconnects TCP connections. If the back-end web server (such as Nginx or Apache) is set up.
max_clients
If the maximum number of concurrent connections is too small, or there are too many sockets in the TIME_WAIT state, the backend TCP queue overflows and rejects new connection requests from SLB.
Solution:
Optimize Linux kernel network parameters (/etc/sysctl.conf):Ini, TOMLnet.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 30 net.core.somaxconn = 1024
Adjust the Nginx high concurrency parameter: Adjust the worker_connections and keepalive_timeout in nginx.conf to ensure that there are still enough Worker processes to handle SLB probe requests under high concurrency.
6. TCP health check misjudgment in long connection/Websocket scenarios
Principles and Symptoms:
For TCP snooping, the SLB establishes a connection via a three-way handshake (SYN -> SYN-ACK -> ACK) by default, and then immediately sends an RST to disconnect to determine health. Some back-end applications or firewalls will judge this behavior of "only handshaking without data transmission and frequent RST" as illegal scanning, and then actively block the detection IP address of SLB, causing health check failure and jump.
Solution:
Exclude abnormal handshake detection on SLB private CIDR blocks in backend applications or firewalls.
If it is an HTTP application, try to change the SLB monitoring mode to HTTP/HTTPS monitoring for more accurate HTTP status code detection.
3. the "four-step" workflow for troubleshooting SLB health checks
When you see the red "Health Check Failed" alarm on the console, do not panic. It is recommended to perform quick diagnosis in the following order:
[Step 1: ECS Local Test]
Use curl to test the local service port and health check URL to confirm that the backend service is normal.
Down
[Step 2: Network and Security Group Troubleshooting]
Check whether the security group and the local firewall (iptables) have allowed the CIDR block 100.64.0.0/10.
Down
[Step 3: Grab Package and Log Analysis]
Run the tcpdump on the ECS to capture the packet and analyze whether the SLB probe request and the specific HTTP return code are received.
Down
[Step 4: Cloud Monitoring and Resource Troubleshooting]
Check the system CPU, memory, disk, and bandwidth usage, and check whether the Alibaba Cloud account status and resource deduction are normal.
Among them,
Grab Package Command
Very useful. You can run directly inside ECS:
Bash
# Grab from SLB
Port 80 traffic of the private network segment
tcpdump -i any src net 100.64.0.0/10 and dst port 80 -nn
By observing whether there is
SYN
Packet entry and response
HTTP status
, you can locate whether the problem occurs in the "network connection phase" or the "Web application response phase" in seconds ".
4. SEO perspective summary: stability is the best SEO optimization
As a SEO website optimizer, I think the underlying logic of website optimization is never just to write articles and do outside the chain,
Infrastructure stability is the cornerstone of SEO
.
Avoid search engine demerit: Frequent failure of SLB health check will cause intermittent 502/504 errors on the front end. Search engine spiders encounter such errors many times, will quickly determine the instability of the web server, resulting in stagnation, ranking decline.
Ensure user experience and conversion rate: High-availability, low-latency websites can significantly reduce the bounce rate (Bounce Rate) and improve the residence time of the page. These user behavior data are also important indicators for search engines to evaluate page quality.
Pay attention to O & M details and resource management: Ensure high availability of infrastructure not only in code and architecture, but also in daily enterprise cloud resource management. Maintaining sufficient fund management and timely recharging of Alibaba Cloud accounts can ensure the continuous and efficient operation of components such as SLB, CDN, cloud security protection (WAF), and cloud monitoring, and prevent problems before they occur.
By deeply understanding the SLB health check mechanism, configuring reasonable detection rules, releasing security groups, and constantly monitoring the performance of back-end servers, you can completely solve the persistent problem of frequent health check failures and create a rock-solid high-availability architecture for your website!