Aliyun Account Recharge Channel: Applicable Scenarios for Big Data Instances and SEO Webmaster Architecture Guide
As a cloud computing architect who has been crawling in the cloud for many years, and also an SEO long blogger who stares at traffic, weight and crawler logs every day, I often find that many technical peers fall into a misunderstanding when facing massive data:
"If there is a large amount of data, it is not enough to directly stack the highest computing power (computing type) or the most expensive cloud disk (ESSD)?"
In small websites or basic businesses, this "big miracle" approach may work. However, when your business evolves to TB-level or even PB-level distributed computing, the traditional "computing and storage separation" architecture (I. e. ECS-mounted cloud disks) will encounter serious problems.
Network I/O bottlenecks and high budget deficits
.
In order to solve the "data handling" problem under this distributed architecture, Aliyun launched
Big data type instance (D series, such as d3s, d3c, d2s, etc.)
.
In this article, from the dual perspectives of architects and webmasters, I will deeply disassemble the underlying technology dividends and core application scenarios of Alibaba Cloud big data instances, and share how to combine them.
Alibaba Cloud account
The ultimate FinOps (financial governance) and SEO log architecture optimization.
1. what is an alibaba cloud big data instance and where is its underlying "hard core?
Traditional ECS enterprise-level instances (such as C- series computing and G-series general-purpose instances) usually adopt the "storage and computing separation" architecture. Data is stored on remote block storage (cloud disk), and each read and write must be transmitted through the intranet network.
while big data instance (big data Instance family, referred to as d series)
is completely different. It adopted
"Local high-throughput large-capacity HDD storage enterprise-class computing power strong intranet bandwidth" gold combination. Its core technical characteristics can be summarized in three key words:
1. Extreme local throughput (Data Locality)
Big Data instances are directly equipped with up
12 TB (single disk) high-capacity, high-throughput SATA HDD local disk
. Because data is directly stored on the local physical hard disk in the same rack as the CPU, the overhead of cloud disk network transmission is completely eliminated, the throughput (Throughput) of massive sequential read and write is extremely high, and supports
Online hot plug bad disk repair
.
2. 1:4 gold deposit ratio
The processor-to-memory ratio is typically $1:4 $. This means that while providing sufficient computing cores, large data frameworks (such as Spark and Flink) are also given sufficient memory buffers to prevent OOM (memory overflow) when performing complex stream calculations or memory aggregation.
3. Ultra-high intranet bandwidth (up to 64 Gbit/s)
When a distributed big data cluster performs large-scale computing, nodes need to exchange data frequently. The big data instance provides a single-instance network bandwidth of up to 64 Gbit/s, which perfectly solves the communication of the cluster during network alignment and shuffling (Shuffle).
Lag.
Four Core Application Scenarios for 2. Alibaba Cloud Big Data Instances
Big data instances are not omnipotent. Their local disks do not support snapshots or online configuration changes. This doomed it to be
Distributed, highly available, and fault-tolerant big data components
Tailored. The following are the four most classic application scenarios in the production environment:
1. Distributed big data computing and storage (Hadoop/Hive/HBase)
This is the "main battlefield" of the d-series instance ". When building an open source Hadoop ecosystem or using Alibaba Cloud E-MapReduce(EMR), data nodes (DataNode) require huge storage space and high throughput.
Architecture pain point: When traditional cloud disks face massive Hive partition table association queries (Join), cloud disk channels are often capped by throttling.
D-series advantages: With the help of local large-capacity SATA HDD disks, the natural multi-copy mechanism of HDFS(Hadoop Distributed File System) perfectly compensates for the shortcomings of local disks without snapshots. The high network bandwidth greatly accelerates the Shuffle process under the MapReduce computing framework, and the overall cluster running efficiency can be increased by more than $30\%$.
2. Distributed Search and Log Analysis Cluster (Elasticsearch)
For Elasticsearch(ES) or ClickHouse clusters that carry billions of logs and need to provide full-text retrieval in seconds, I/O capabilities determine life and death.
Architecture pain points: ES is prone to I/O suspension during write peak periods (such as double 11, big promotion, or when the website is attacked by large-scale crawlers or CC), resulting in split clusters or denial of service.
D-series advantages: The local market of big data instances can provide extremely high sequential write throughput. With the shard and copy Replica mechanisms, not only the upper limit of index storage is greatly expanded, but also the storage unit price of massive cold and hot logs is significantly reduced.
3. High-throughput message queue buffer center (Apache Kafka)
Kafka as a modern distributed architecture "flow reservoir", the core requirements are.
Extreme sequential disk writing capability and huge network packet throughput
.
D-series advantages: Kafka's PageCache mechanism is combined with the high-throughput SATA HDD local disk of D-series instances, which can maximize the physical sequential read and write performance of the disk. At the same time, the network bandwidth of tens of gigabytes on a single machine ensures that hundreds of Consumer (consumers) and Producer (producers) do not lose or delay packets under high concurrent throughput.
4. Big data hot and cold tiered storage (EMR JindoFS + OSS)
Under the general trend of cost reduction and efficiency increase, more and more enterprises are turning to "hybrid big data architecture".
D series advantages: in d3c, d2c and other specification families, enterprises can use EMR J
The indoFS cache mechanism uses the local disk of the big data instance as the "hot data cache layer", and stores large amounts of historical cold data persistently in Alibaba Cloud Object Storage Service (OSS). This architecture not only enjoys the extreme read and write speed of local disks, but also has unlimited flexibility and ultra-low cost of object storage.
3. from Technology to Finance: FinOps with "Aliyun Account"
Big data clusters are usually "throughput monsters" composed of dozens or even hundreds of instances. If they are not configured properly, the monthly bill expenses will be amazing. As an architect, you must immediately switch back to your financial perspective and log in to your
Alibaba Cloud account
Precise cost control.
1. Log in to your Alibaba Cloud account for unified resource planning
At the beginning of the project, the person in charge of operation and maintenance shall, through the business owner.
Alibaba Cloud account
Go to the console and use the official "Cost and Cost" center for cluster budget planning. Since big data instances (d series) do not support seamless change of subscription year-to-month direct transfer to pay-as-you-go
Alibaba Cloud account
It is important to measure the "basic resident power" and "elastic tidal power" of the business in advance.
2. Skillfully use "preemptible instances" to cooperate with Aliyun account saving plan
The node types of big data clusters determine their billing models to play more tricks:
Management node (Master Node): The host NameNode and other core services must be 100% stable. It is recommended to select "Package Year and Month" or bind "Save Plan" to lock in the low price for a long time in the Aliyun account.
Core/Task Node: Because distributed big data has good fault tolerance and replica migration mechanisms, architects can purchase preemptible instances (Spot Instance) from their Alibaba Cloud accounts to host task nodes. When the market price fluctuates, even if individual nodes are released, the cluster can automatically rebalance the data (Rebalance). The perfect combination of technology and finance can save up to more than $70 \%$ in computing costs for business bills.
4. SEO Blogger's Private Words: The Alternative Use of Big Data Examples in Webmaster Operation and Maintenance
As a webmaster proficient in SEO, you may ask:
"What is the relationship between big data examples and our daily website building and SEO?"
The relationship is too big. When your website matrix is bigger, or you are running high-traffic vertical search engines, aggregation independent stations, and multilingual e-commerce, you will produce several G's or even dozens of G's every day.
Nginx access log
.
[Nginx access log structure and analysis flow via Big Data instance]
These logs hide all the tracks of search engine crawlers such as Googlebot and Bingbot.
Webmaster pain points: the traditional server simply can't run massive
Log analysis. Once grep or awk is used to forcibly analyze hundreds of GB of compressed logs in the production environment, the CPU will be directly full, the website will not be opened instantly, and SEO ranking will be seriously damaged.
Architect's solution:
You can use the Logstash to synchronize the Nginx logs of all sub-stations and independent stations to an Alibaba Cloud big data instance in real time.
ELK(Elasticsearch Logstash Kibana)
Or
ClickHouse
in the temporary analysis cluster.
With the huge local disk and powerful disk throughput of big data instances, you can easily perform the following SEO in-depth analysis:
Crawler frequency and time period analysis: Accurately monitor the time period in which Google crawlers crawl most frequently, so as to reasonably avoid the peak period of user access.
Status Code Depth Analysis: Quickly retrieve whether there are hidden 404 errors or 500 pages in the whole station to prevent weight loss.
Crawl Budget optimization: Analyze which low-value invalid URLs waste a lot of crawler energy, and then block them accurately through robots.txt or canonical tags.
5. summary and selection suggestions
Alibaba Cloud big data instances (D series) are not prepared for ordinary stand-alone Web applications, but are "heavy weapons" specially used to subvert the torrent of distributed data in the cloud computing era ". Through the combination of high-throughput local disks and high intranet bandwidth, it perfectly cuts the underlying pain points of distributed components such as Hadoop, Spark, Kafka, and Elasticsearch.
If you are facing TB/PB data processing pressure, or are worried about expensive block storage bills, you may as well log in to your
Alibaba Cloud account
, locate "big data type" on the ECS creation page, and evaluate the d3s or d3c specification family based on the data volume of the business. Let professional specifications do professional things, and you will find that not only will the calculation speed increase exponentially, but the overall IT architecture cost will also usher in a historic structural decline.
