From Data Access to Intelligent Decision-making: A Guide to Building a Panoramic View of Alibaba Cloud's Big Data Product Chain
In the era of digital economy, data has become the core production factor of enterprises. How to transform the massive data scattered in various business systems into efficient business decisions is a common challenge for enterprise architects and data engineers.
Alibaba Cloud is built around the full life cycle of data, covering
Data access, storage and computing, real-time analysis, AI modeling and visual decision making
One-stop big data product matrix. This tutorial provides an in-depth analysis of the underlying architecture logic of the Alibaba Cloud big data product chain and helps you build a highly available, low-latency, and scalable modern data platform (Data Lakehouse / Real-time Data Platform).
1. Alibaba Cloud Big Data Panoramic Architecture Blueprint
Before building specific products, we need to clarify the logical hierarchy of the entire data platform. A mature enterprise-level big data platform is usually divided into four core levels:
-----------------------------------------------------------------------
| Intelligent Decision and Application Layer (Quick BI / PAI) |
-----------------------------------------------------------------------
▲
│
-----------------------------------------------------------------------
| Data Service and Acceleration Layer (Hologres / AnalyticDB) |
-----------------------------------------------------------------------
▲
│
-----------------------------------------------------------------------
| Offline vs. real-time computing layer (MaxCompute / Realti
me Compute) |
-----------------------------------------------------------------------
▲
│
-----------------------------------------------------------------------
| Unified Data Storage and Lake Warehouse Layer (OSS-HDFS / DLF) |
-----------------------------------------------------------------------
▲
│
-----------------------------------------------------------------------
| Data Access and Transport Layer (SLS / DataHub / DTS) |
-----------------------------------------------------------------------
Core product function quick look-up table
Hierarchy
Representative product
Core positioning and technical advantages
Access layer
SLS / DataHub / DTS
Log collection, real-time streaming data flow, database CDC real-time synchronization
storage layer
OSS-HDFS / DLF
Object storage cost reduction and efficiency enhancement, data lake construction and metadata unified management
computing layer
MaxCompute / Flink
Enterprise-level offline hyperscale computing (EB-level) and second-level real-time streaming computing
Service Layer
Hologres
Sub-second interactive analytics, HSAP (Integrated Service Analytics) architecture
Decision-making level
Quick BI / PAI
Advanced intelligent reporting, ML and big model AI decisions
2. Phase 1: High-throughput data access and integration (Data Ingestion)
Data access is the entrance to the platform and needs to be addressed at the same time.
Offline batch
Synchronization
with
real-time streaming collection
Two typical scenarios.
1. Incremental synchronization of business database CDC: DTS
For business databases such as MySQL, PostgreSQL, and PolarDB, use
DTS (Data Transmission Service)
Real-time change data capture (CDC) at the Binlog level:
Configuration points: Create a DTS real-time synchronization task with the database as the source and the DataHub or Hologres as the target.
Advantages: Zero intrusion into the business master database and support automatic synchronization of DDL changes.
2. Log and buried point data collection: SLS and DataHub
SLS (Log Service): applies to server logs, Nginx access logs, and app client tracking points. Deploy a Logtail client on a node to collect tens of millions of logs in seconds through simple configuration.
DataHub: As an alternative to Alibaba Cloud's native Kafka, it assumes the role of a buffer for highly concurrent streaming data and continuously supplies water to downstream Flink or MaxCompute.
3. Phase II: Lake Warehouse Integrated Calculation and Storage (Compute & Storage)
After a large amount of data is deposited, you need to build a dual-track parallel computing engine of "offline batch processing" and "real-time stream processing", and combine the data lake to achieve elastic cost reduction.
1. Off-line number warehouse and number warehouse modeling: MaxCompute DataWorks
MaxCompute (formerly ODPS)
is a fully managed Serverless cloud data warehouse of alibaba cloud. Integration
DataWorks
It can realize standard hierarchical modeling of data warehouse (ODS -> DWD -> DWS -> ADS).
Standardized data warehouse modeling: ODS layer (source layer): through DataWorks data integration, data captured by DTS/SLS is regularly written to the MaxCompute by day/hour partition. DWD layer (detail layer): Run SQL to clean, desensitize, and join dimension tables. DWS layer (summary layer): Periodically aggregates by user, product, and granularity to generate a wide table.
2. Second-level real-time stream computing: Realtime Compute for Flink
For real-time wind control and real-time large-screen scenarios,
Alibaba Cloud Flink Full Managed Edition
.
Typical stream processing link: source table (Source): declares the DataHub or topic in Kafka. Dimension table (Lookup): Associate the user profile or configuration table in the Hologres. Result Table (Sink): Write the calculated window metrics into the Hologres in real time.
4. Phase III: Integration of Service Analytics (Serving & Analytics)
In the traditional architecture, "analysis (OLAP)" and "online service (KV query)" are often separated, resulting in data duplication and redundancy. Alibaba Cloud through
Hologres
realized
HSAP(Hybrid Serving/Analysis Processing)
Concept.
Hologres Architecture Benefits
Seamless connection with MaxCompute depth: supports direct foreign table query on MaxCompute offline tables without complex ETL data reflow.
High concurrent point search and complex OLAP have both: it can not only support primary key query of hundreds of thousands of QPS per second for front-end API, but also support multi-dimensional cross analysis at the second level for BI tools.
Architecture suggestion: Synchronize the ADS layer final result table generated by offline computing to the Hologres, receive data written by Flink in real time, and use it as the data base for upper-layer BI and API.
5. Stage 4: From Data to Intelligent Decisions (AI & Decision)
The ultimate value of data is to guide business action. Through
Quick BI
with
PAI Platform
, you can turn the precipitated data assets into automated decision-making power.
1. Agile Visualization and Business Analysis: Quick BI
Quick BI is an intelligent BI product created by Alibaba Cloud for enterprise decision makers and business operators:
Data Connect: Add Hologres or MaxCompute directly as a data source.
Ad hoc analysis and dashboard: drag-and-drop KPI large screen and mobile report.
AI intelligent exploration: with the built-in NL2SQL (natural language to SQL) function, business personnel only need to ask "what are the top ten commodities in East China last month?", the system can automatically generate charts.
Predictive AI Decisions: PAI (Platform for AI)
When the underlying report cannot meet complex forecasting needs, access.
PAI (ML Platform)
Create intelligent closed loop:
Data preparation: Directly read the feature data from the MaxCompute/OSS.
Model training: Use PAI-Designer or PAI-DSW for churn rate prediction, collaborative filtering recommendations, and credit scoring model development.
Model deployment: Deploy the trained model as a PAI-EAS (highly elastic and concurrent inference service API) with one click for real-time calls by the business side.
6. Scene Actual Combat: Building E-commerce "Real-Time Wind Control and Recommendation" Full Link
To help better understand product chain collaboration, we take a
Real-time wind control and personalized recommendation system
For example, sort out the complete data flow path:
Acquisition (S
LS/DTS): The user's click log is collected in real time through SLS, and the order transaction event is captured from the PolarDB in real time through DTS.
Transmission (DataHub): Logs and CDC data are pushed to the DataHub through a unified scheduled pipeline.
Compute (Flink + MaxCompute): Real-time link (Flink): reads the DataHub and combines the blacklist dimension table in the Hologres to determine whether the behavior is malicious. If yes, the alert API is triggered in real time. Offline link (MaxCompute): Summarize historical full-volume behavior data offline every night and recalculate user preference tags.
Model (PAI): The PAI ML engine reads the MaxCompute tags at night to retrain the recommendation model and writes the prediction weights back to the Hologres.
Decision and presentation (Quick BI/business system): business API second-level query Hologres, real-time recommendation of products to the App front end. Quick BI dynamically displays the risk control interception rate and recommended conversion rate on a large screen in real time.
7. Performance Tuning and Cost Governance Best Practices
When building a large data platform, resource costs and operational efficiency must be balanced.
1. Hierarchical storage of hot and cold data (Tiered Storage)
Hot data (second-level response): It is stored in the Hologres or Realtime Compute memory and maintains fine-grained data of the last 7 to 30 days.
Warm data/cold data: Data over 30 days is automatically archived to MaxCompute or OSS low-frequency/archive storage, reducing storage costs by more than 60%.
2. On-demand scheduling of computing resources
MaxCompute hybrid billing mode: for offline ETL tasks that are scheduled on a daily basis, the subscription (CU reservation) mode is adopted to ensure baseline efficiency; for sudden and temporary large-scale analysis, pay-as-you-go (Pay-as-you-go) elastic expansion is enabled.
Flink Elastic Auto-scaling: Enable Flink to automatically expand and shrink capacity to cope with traffic shocks during peak business periods such as e-commerce promotions and discounts.
8. Summary
Building the Alibaba Cloud big data product chain is not simply "stacking tools", but should follow
"Unified Collection-> Lake Warehouse Building-> Real-time Computing-> Speed Service-> AI Enabling"
The closed-loop system.
Through
SLS/DataHub + DataWorks/MaxCompute + Flink + Hologres + QuickBI/PAI
The golden combination of the enterprise can get rid of the quagmire of the traditional chimney architecture, not only to achieve the full amount of data assets "visible, fast search", but also with the help of artificial intelligence.
It can realize "accurate decision-making and fast response" and truly give full play to the maximum value of digital productivity.
