Google Cloud Account Recharge: Frequent Recycling of GCP Preemptible VMs (Spot)? Highly Available Alternatives and Practical Guidelines for Preventing Flow Disconnection
On Google Cloud Platform (GCP), preemptible VMs (now collectively known as Spot VMs) are attracting a large number of developers and enterprises with super-high discounts of up to 90%. However,
Spot VMs can be forcibly reclaimed by GCP at any time (Preemption) with only a 30-second graceful shutdown window
.
For Web services, real-time APIs, data stream processing, or node-sensitive tasks, if you rely entirely on a single Spot VM, you often face pain points such as network outages, service outages, and connection outages.
How to solve the problem of "flow interruption" caused by frequent recycling while maintaining low cost? This article will deeply analyze the alternative architecture and configuration scheme of preventing flow interruption.
1. why is Spot VM always recycled?
GCP's Spot VM uses the idle computing power of Google's data centers. When a higher-paying standard (On-Demand) user requests computing power, or a zone (Zone) resource is tight, the system will preferentially withdraw the computing power of Spot VM.
The main factors for frequent recycling include:
Popular zones and models: For example, us-central1-a n2-standard are extremely popular, with very few idle resources.
Peak demand for computing power: Data center computing power usage is generally higher during the daytime on weekdays than at night and on weekends.
Lack of elastic guarantee mechanism: automatic replacement and load balancing are not set, resulting in no new nodes taking over traffic after being recycled.
High-availability alternatives and hybrid solutions 2. GCP preemptive VMs
If your business cannot tolerate frequent drop of Spot VM, it is recommended to abandon the extensive deployment method of "stand-alone Spot" and adopt the following four alternatives and optimization schemes:
Scenario 1: Hybrid managed instance group (Hybrid MIG)
In GCP, instead of using Spot VM instance groups alone, create a
Hybrid architecture combining On-Demand and Spot Instances
.
Implementation principle: Deploy a standard on-demand VM as a "baseline Capacity" to handle the core and guaranteed business traffic. On this basis, the extended cluster uses Spot VM to support peak traffic.
Advantages: Even if the Spot node is reclaimed by 100, the on-demand node at the bottom can still ensure the continuous flow of basic services and only reduce part of the concurrent carrying capacity.
Solution 2: Multi-Zone and Multi-Machine policies
Don't put all your eggs in the same Availability Zone or on the same model.
How to do it: Create a regional managed instance group (Regional MIG) and distribute Spot Instances across more than 3 Availability Zones (such
us-central1-a/B/f).
Decentralized quota: When different models are used together (for example, e2-standard-4 and n2-standard-4 are allowed at the same time), the idle rate of each model is different, and the probability of large areas being recycled at the same time is exponentially reduced.
Scenario 3: Migrate to GKE(Kubernetes)+ Autopilot / Spot Node Pool
If you are running a containerized application, migrating to the GKE container service is a better alternative.
Elastic scheduling: GKE supports Spot Node Pool. When the Spot node receives the reclaim notification, GKE automatically triggers the drains operation to gracefully migrate (Evict) the pod to other available nodes.
Hybrid policy: Set pod affinity (Affinity) and tolerance, deploy the core control plane in the on-demand node pool, and deploy scalable work pods in the spot node pool.
Scenario 4: Purchase Commitment Discounts (reserved discount) instead of Spot
If your business requires 24/7 uninterrupted and stable operation and cannot be transformed into a stateless architecture, it is recommended to abandon Spot VM and switch
CUD(Commitment-Based Discounts Committed Usage Discount)
.
Effect: Promise to use for 1 or 3 years, on-demand VM can get a deep discount ranging from 37% to 57%, which saves money and will not be recycled by 100.
3. Core Configuration Practice Guide to Prevent Service "Discontinuity"
If your business must continue to use Spot VM to reduce costs, you can establish a complete "anti-flow" barrier through the following four steps:
1. Configure the 30 second shutdown script (Shutdown Script) to capture signals
When GCP decides to recycle a Spot node, it issues a
ACPI G2 Soft Off
shutdown signal and retain a buffer time of up to 30 seconds. Must use those 30 seconds for a graceful exit.
Set in metadata (Metadata)
shutdown-script
:
Bash
#! /bin/bash
#1. Send a health check failure signal to the load balancer/gateway to stop new traffic
echo "Draining connection..." > /var/www/html/healthcheck.html
#2. Notify internal service to smoothly cut off long connection (e. g. WebSocket, TCP state)
#3. Disk local unsynchronized data to Cloud Storage or database
gsutil c
p /tmp/cache_state.json gs://my-bucket/backups/
#4. Exit the main process
systemctl stop my-app-service
2. Front Cloud Load Balancing Elegant Connection Unplug (Connection Draining)
If your Spot VM is running behind the Cloud Load Balancer(CLB), you must turn on
Connection Draining (connection removal)
:
Effective mechanism: When the load balancer senses that the node is recycled or the health check fails, it will immediately refuse to allocate new traffic to the VM, but will leave a certain amount of time for the existing TCP connection that has been established (for example, set 15-30 seconds).
Configuration parameters: It is recommended to set the draining-timeout to 20s (which must be less than GCP's 30-second recycling limit) to avoid 502/504 errors caused by user requests being forcibly cut off when the handshake is halfway through.
3. Configure MIG Health Check with Autohealing (Automatic Healing)
Bind the HTTP/TCP health check in the managed instance group (MIG) and adjust the automatic healing (Autohealing) policy to the fastest response:
Check Interval: 5 seconds is recommended.
Unhealthy Threshold (Unhealthy Threshold): Set to 2 times.
Effect: As soon as the node is recycled, the health check is declared abnormal, the load balancer quickly cuts off the traffic, and MIG automatically initiates a new node to supplement capacity in the background.
4. Data decoupling and state externalization (Stateless Design)
The most fundamental principle of flow prevention is
Achieve Statelessness
:
Never store user Session, file upload temporary cache on Spot VM's local disk.
Migrate Session storage to Cloud Memorystore (Redis), connect the database to Cloud SQL, and store files in Cloud Storage. In this way, even if the Spot VM is suddenly shut down, the user only needs to refresh the webpage and reconnect to another node, and the business status is not affected at all.
Tips for 4. Operation and Maintenance and Cloud Resource Settlement Optimization
When planning the operation and maintenance of GCP computing resources, in addition to fault tolerance optimization in architecture design, you also need to pay attention to the continuity of the underlying account and bill.
Many small and medium-sized enterprises or development teams often encounter quota restrictions (Quota) or items caused by abnormal credit card charges when expanding clusters and pulling Spot nodes in batches.
risk of downtime. In order to ensure the smooth operation of cloud infrastructure, many teams will choose to conduct through professional channels.
Google Cloud Account Recharge
With quota generation services, avoid unexpected service interruptions due to bill arrears or settlement channel stuck during peak business hours. The combination of flexible billing management and high availability architecture design is the ultimate solution to achieve GCP cost reduction and efficiency.
Summary
The frequent recycling of GCP Spot VMs is an inherent attribute behind their "ultra-low price. If you want to make good use of this "cost-reducing tool", the key lies in
Change from passive recovery to active prevention
:
Schema layer: Use On-Demand Spot to mix managed groups or migrate to a GKE schema.
Traffic layer: Enable the Connection Draining and rapid health detection of Cloud LB.
Application layer: Capture 30-second shutdown script signals and completely transform the service into a stateless architecture.
To achieve the above points, even if the node is recycled several times a day, front-end users can still enjoy a smooth experience without feeling in milliseconds.
