Emergency Response Guide for Alibaba Cloud Servers under DDoS / CC Attacks
Received a text message from Aliyun: "Your ECS instance has been attacked by heavy traffic and has been sent to black hole cleaning......"
I believe that any webmaster or operation and maintenance to see this news, blood pressure will soar instantly. The website can't be opened, users are frantically complaining in the group, and the boss is staring at you behind your back. At this time, your palms are sweating and you are extremely prone to illness and rush to the doctor.
Calm down. In the face of an attack, panic will not solve any problem. Today's tutorial does not have any theoretical nonsense, directly give you a set.
Emergency Response Guide for Production Environment Against DDoS / CC Attacks
. Put this article in your bookmark, the key moment can save life.
Core concept: first find out what kind of beating you got.
Only when you know your enemy and yourself can you see the trick. There are two main types of attacks, with completely different coping strategies:
DDoS attack (traffic type): Hackers control thousands of broilers and use massive garbage traffic (UDP, TCP Flood) to directly squeeze your server bandwidth. It's like tens of thousands of people squeezing a subway entrance at the same time, and normal people can't get in.
CC attack (application type): The hacker's traffic is not so large, but he uses proxy IP to simulate real users and frantically refresh the most resource-consuming pages on your website (such as search, database query, dynamic interface). The bandwidth is not full, but your server CPU is 100 percent instantly and is directly paralyzed.
The first stage: emergency self-rescue -5 minutes emergency flow
When you find that the website cannot be opened, and the monitoring shows that the traffic is soaring or the CPU is red, immediately start the following emergency steps.
Step 1: Determine whether you have entered a "black hole"
If Aliyun directly "black holes" your server, it means that the attack traffic has exceeded the default defense threshold provided by Aliyun to ordinary ECS (usually about 5G).
Phenomenon: Not only can the website not be opened, but you can't connect the server completely through SSH, and even ping the public network IP of the server is all overtime.
Solution: Log in to the Alibaba Cloud console, search for DDoS protection, and check the status of your instance in the basic protection list. If it is "in a black hole", it usually takes 30 minutes to several hours to automatically unseal.
💡How to save yourself during the black hole period? Don't wait foolishly. If the business needs to be restored, immediately go to the console to bind a new "elastic public network IP(EIP)" to this ECS ". As long as the hacker hasn't found your new IP, the website can be revived for a short time.
Step 2: If SSH can still be connected, urgently troubleshoot CC attacks.
If the server is not in the black hole, but it is stuck to death, connect it with VNC or SSH and type in
top
. If you see
NGINX
or
PHP-FPM
/
Java
When the CPU is full, 90% of them are attacked by CC.
Immediately look through Nginx's access log (Access Log) and catch the features that are frantically brushing the screen:
Bas
h
# View the current most visited potential lead IP (top 20)
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -n 20
Observe these high-frequency access requests and look for what they have in common:
Are they all frantically requesting the same dynamic interface (such as/api/v1/search)?
Is their User-Agent (browser logo) wonderful or exactly the same (for example, they are all old IE6.0 or empty)?
Are they all from the same IP segment?
Step 3: Immediately "close the door and beat the dog" locally"
After catching the feature, make an emergency ban in Nginx.
1. If a specific IP is at fault, directly seal the IP segment:
Open your Nginx website configuration file, in
server
Add directly to the block:
Nginx
deny 123.45.67.89;# Block a single malicious IP address
deny 220.181.0.0/16;# Block the entire malicious C or B segment IP
Then run
nginx -s reload
.
2. If the request characteristics are obvious (such as a specific UA or interface), return the 403 directly:
Nginx
if ($http_user_agent ~* "Scrapy|HttpClient|Java") {
return 403;
}
The second stage: architecture reconstruction-cut off the line of sight of hackers
The above emergency self-rescue can only block low-level attacks. If the hacker changes the proxy IP and continues to brush, or directly uses large traffic DDoS, it is absolutely impossible to rely solely on a single ECS to carry the if.
You need to adjust the structure immediately,
Hide the server behind the scenes
.
Core Defense Architecture Design
Plaintext
[Malicious attack traffic/normal user]]
│
▼
[cloud security defense layer (high-security IP/CDN)]-(cleaning garbage traffic)-> intercept and discard
│
(clean flow)
│
▼
[Source ECS hidden in VPC]]
Step 4: Connect to CDN or WAF (completely hide the IP address of the origin site)
The hacker can hit you because he knows the real public network IP of your server. We need to immediately add a "firewall" between the domain name and the server ".
Access to Aliyun full-site accelerated DCDN or secure CDN: DNS resolution of domain name is directly pointed to server IP,
Instead, point to the CNAME address provided by the CDN. In this way, how hackers check the domain name, only the edge node IP of CDN is obtained. CDN has a natural large flow cleaning ability, can help you block the vast majority of CC attacks.
Turn on "five-second shield" (JavaScript challenge): in the console of CDN or cloud shield WAF, set the protection level to "emergency" or turn on "CC security protection". At this point, anyone visiting your website will first pop up a 5-second wait page with "Detecting browser security. The broiler script does not have a real browser rendering engine and will be completely stuck by this level.
Step 5: Cut off all direct access to the server's old IP (critical)
Many people find that the server is still stuck after receiving the CDN. Why? Because the hacker has already recorded the real IP of your previous server, he directly bypasses the domain name and takes the IP to the death.
The only solution: Go to the Alibaba Cloud ECS console and enter the Security Group ".
Modify the inbound rule: Delete the rule that allows all IP addresses (0.0.0.0/0) to access ports 80 and 443.
Change to: Only the public IP segment of the CDN or WAF you purchased is allowed to access 80 and 443.
In this way, in addition to the clean traffic forwarded through CDN, anyone outside who directly knocks on your server IP will prompt the connection failure. Your server is completely invisible on the Internet.
The third stage: long-term stability -- nip in the bud
After a beating, we must make up for the short board in daily operation and maintenance:
Code-level self-redemption: For high-risk dynamic interfaces such as login, registration, search, and SMS verification codes, graphical verification codes or behavior verification (such as slider verification) must be added. Introduce Redis counters in the code logic to limit the access frequency for a single user or IP address (for example, access more than 5 times in 1 second is directly locked for one hour).
Prepare "money bags" and high-security programs: if you are doing e-commerce, games, or online marketing activities, it is easy to attract professional hackers hired by your peers. Ordinary ECS cannot solve high-traffic DDoS. Once the attack traffic exceeds 50G, honestly purchase Aliyun's DDoS high protection (new BGP). It has T-level cleaning bandwidth, which is really a line of defense stacked with pure hardware and top bandwidth.
It is not terrible that the website is attacked. What is terrible is that the hacker is beating you, but you sit in front of the computer and do nothing but restart the server. Put this set of "hidden source station cleaning traffic application flow limit" method into your mind, even if the hacker comes again, you can still be calm and calm.