How can GCP databases migrate smoothly? Using Database Migration Service (DMS) for non-stop migration
In the world of cloud computing, there is one thing that can keep all architects and operations (SRE) working all night long and on tenterhooks, and that is
Database migration
.
When many people hear "moving the library", the picture that comes to mind is: pick a new year's eve or three o'clock in the morning, hang up the home page announcement "system maintenance", and then hurriedly export SQL, transfer files, import new library, modify the configuration... The whole process is like walking a tightrope at high altitude. Once the middle link is stuck or the data is not correct, the business suspension time will be infinitely longer.
But in fact, the current cloud manufacturers have long been out of fashion this hard-core moving method of "killing 1,000 enemies and damaging 800 themselves. Today we're going to talk about an artifact provided by Google Cloud (GCP)--
Database Migration Service (DMS)
.
As a person who has been covered in mud in the production environment, I will teach you how to use DMS to realize "zero downtime (Minimal Downtime)" smooth moving in the most easy-to-understand language today, regardless of the drowsy terms in official documents.
1. core concept: what is DMS "zero downtime move"?
Before we do it, let's take a minute to figure out its underlying logic. The core reason why DMS can achieve "zero downtime" is that it does not throw data in the past at one time, but is divided into two stages:
Google Cloud Account Purchase
Full initialization (stock data replication):DMS first copies all existing historical data in the source database to the target GCP database (such as Cloud SQL). In this process, your online business will run as usual, and normal reading and writing will not be affected at all.
Incremental synchronization (real-time data tracking): After the historical data is transmitted, DMS reads the logs of the source database (such as MySQL's Binlog or PostgreSQL's WAL) to continuously "synchronize" the newly generated business data during the full amount of relocation to the new database. At this time, the data of the old and new libraries is almost completely synchronized in real time.
Since the old and new library data are consistent, you can change the application's database connection string (Connection String) to the address of the new library in broad daylight and at the most leisurely time of business.
The service interruption time is only tens of seconds or even a few seconds for restarting the application and switching the configuration.
2. move before the "three security checks" (preparatory work)
Moving with DMS is like flying. When the security check is done, the flight is half successful. The following three points directly determine whether the migration will fail:
1. Open the log of the source library (the most critical step)
DMS must be able to read the real-time logs of your source database before incremental synchronization can be performed.
If the source database is MySQL: you must ensure that binlog is enabled, and the binlog_format must be set to ROW,binlog
_row_image set to FULL.
If the source library is PostgreSQL: you need to set the wal_level to logical and configure the replication slot (Replication Slots).
2. Think about how to get through the network (connectivity scheme)
It must be safe to talk between the source library and GCP. DMS provides several connection methods, in order of security and recommendation:
VPC peering(VPC peering): If your source database is already in another VPC of GCP or a computer room connected through a dedicated line (Interconnect), this is the first choice, fast and safe.
Reverse SSH Tunnel: The most common civilian scenario. Open a small virtual machine in the network of the source library as a springboard machine, and let GCP connect through an encrypted tunnel.
IP Allow List (Public IP): The simplest but least recommended. Directly expose the public IP of the source database to the IP range of DMS. If only testing can be done, the production environment should be used with caution.
3. Create a migration account
Google Cloud Account Purchase
Do not use directly
root
or
admin
This supreme account to run migration. In the source repository, create a temporary
dms_user
, only give it the necessary read and copy permissions (e. g. MySQL's
REPLICATION CLIENT
,
REPLICATION SLAVE
,
SELECT
etc.). After the move, delete this account directly, which is safe and elegant.
3. actual combat: DMS configuration four steps
After the preparation work is finished, we go directly to the GCP console and search
Database Migration
.
Step 1: Create a Migration Job"
After clicking Create, you need to fill in the basic information of the job:
Source database type: for example, a self-built MySQL database, AWS RDS database, or other cloud databases.
Target database type: Typically GCP-hosted Cloud SQL or AlloyDB.
Migration type: Do not hesitate to select Persistent Synchronization (Continuous) ". Only select this is zero downtime move; If you select "One-time", then the data transmission is over, excluding subsequent incremental synchronization.
Step 2: Configure Source Connection (Connection Profile)
Here is to fill in the "security information" we have just prepared:
Enter the IP address, port number, and dms_user account and password of the source database.
Choose the network connection method you have set (such as configuring Rev
erse SSH tunnel).
Click Test Connection and make sure the progress bar turns green. If an error is reported, usually the firewall or security group does not release it, go back and check the network.
Step 3: Configure the target library (Cloud SQL)
If the target Cloud SQL has not been created, DMS allows you to "create a new one" directly in the interface ".
You can choose the same configuration (CPU, memory, storage) as the source library.
It is strongly recommended that you check the high availability (HA) and automatic backup of the target library here in one step.
Step 4: Start Verification and Run (Start Job)
Before you click Run, DMS has a very powerful "Pre-flight check" function. It will automatically check for you: Is the log format of the source library correct? Is the permission sufficient? Is the version and incompatibility?
If there is a red error, go to the source library to change the configuration according to the prompt. If all the green ticks are checked, click directly
"Start"
.
4. cutover (Cutover): How to perfectly fire the "last shot"
When the job is running, you will see the status become
Full dump in progress
(in full export), then becomes
CDC in progress
(Incremental synchronization).
When the state enters
CDC in progress
Moreover
Synchronization Delay (Lag) approaches 0
At the same time, it means that the data of the new library and the old library have been completely synchronized. At this time, we can prepare for the real "cut.
This is the essence of determining the success or failure of zero downtime. Please strictly follow the following steps:
1. Business tangential "read-only"
In order to prevent new data from being written to the wrong place within a few seconds after switching the connection, first set the service to "read-only mode" on the application layer or the source database ".
2. Wait for the last wave of DMS data to level
Observe the DMS interface to ensure that the delay (Lag) becomes completely zero. This means that the last few pieces of data generated by the source library before it was read-only are lying firmly in the new GCP library.
3. Click "Done (Promote)" on the DMS"
In the GCP console, click
“Promote”
Button. This action will do two things:
Break the synchronization relationship between the new library and the old library.
Purchase a Google Cloud account
Promote the target Cloud SQL from "read-only receiver" to "fully independent read-write production master library".
4. Modify the application configuration and go online in full quantity
Change the database connection configuration of your application to the IP address of the new Cloud SQL, cancel the read-only mode of the application, and restart the service.
Check the business to see if the front desk order and background login are normal. If all goes well, congratulations, this thrilling database move is in use.
Without any perception, the household was successfully completed!
5. Novice to Avoid Pit and Talk about Experience
Although DMS is very useful, as a former person, there are several invisible pits that must remind you:
Large tables lack primary keys (Primary keys): If your source database has some very large tables with no primary keys, DMS will perform extremely poorly during incremental synchronization (CDC) and may even cause synchronization jam. Before migration, be sure to ask the development students to fill the tables without primary keys with primary keys.
Don't save network bandwidth: the full amount of moving phase is very consumption of network bandwidth. If your source database is in the local computer room, the dedicated line or public network bandwidth is too small, not only will the move take days and nights, but also the normal business bandwidth of your computer room may be exploded. Remember to start the full amount of moving during the low peak period of business, or set traffic restrictions in DMS.
Note the time zone (Timezone) issue: Cloud SQL defaults to the UTC time zone. If your source database is Asia/Shanghai (East Zone 8), the application search time after migration may be inexplicably less than 8 hours. Remember to adjust the time zone parameter (default_timezone) to be exactly the same as the source library when creating the target library.
📝Summary
GCP's Database Migration Service(DMS) is essentially a
Free Serverless (Serverless) Migration Tool
(You only need to pay for the hardware and network costs of the target library during the migration process, and DMS itself does not charge software fees).
It turns the once high-threshold, high-risk "manual human flesh relocation library" into a standardized and visual mouse.
The last sentence concludes:
Don't use the old thinking of downtime maintenance to toss the production environment. Using DMS to do continuous incremental synchronization, holding the initiative in your own hands, and drinking coffee during the day to move the library, this is the elegant posture that modern cloud native operation and maintenance should have!