Tengxun Cloud Data Warehouse ClickHouse Evaluation: Is the "Super Run" in the Big Data Era OK?
If you are a back-end developer, DBA, or data analyst who is struggling with massive amounts of data, you must have heard of it.
ClickHouse
The big name.
In the field of open source big data, this thing is simply fabulous: single-machine performance crush traditional databases dozens of times, and tens of billions of data analysis seconds response. People who have used it all say that watching it run and inquire is like watching it run and blow up the streets.
However, the "difficult service" of open source ClickHouse is also famous in the industry: operation and maintenance are extremely complicated, there are many configuration parameters, and distributed cluster expansion will collapse if it is careless. This also discourages many small and medium-sized enterprises.
In order to solve this pain point, Tencent Cloud launched
Cloud Database ClickHouse(CDCH)
. To put it bluntly, Tencent's experts helped us do all the dirty work at the bottom of the open source ClickHouse and package it into an out-of-the-box cloud service.
Today, we stand in the perspective of real developers, the ClickHouse of Tencent cloud for a comprehensive in-depth evaluation. Don't make a list of official instructions, only talk about dry goods, pain points and vernacular.
Why is 1. ClickHouse so fast?
Before evaluating Tengxunyun's products, let's spend a minute talking. What is the underlying logic of ClickHouse running fast?
Traditional relational databases (such as MySQL) are
row storage
. If you want to check the average age of all users, MySQL must read the entire row of data (name, password, address, age...) of each user from the hard disk, and then pick out the age. This is like having to go through the whole vegetable market in order to buy a Chinese cabbage. IO (Hard Disk Read and Write) goes directly to off the charts.
ClickHouse is typical.
columnar storage
.
It stores the columns "name" and "age" separately. You want to calculate the average age? Well, it goes straight to the "age" column and doesn't even touch the other columns.
Plus it puts the CPU's
SIMD (Single Instruction Multiple Data Stream)
The instruction set is squeezed to the extreme, enabling parallel computing at the physical level. This architecture was born
OLAP (Online Analytical Processing)
, massive log analysis and BI reports.
2. Tengxun Cloud ClickHouse Evaluation: What Does It Help Us?
Since the open source is already very strong, why use Tencent Cloud? We launched a cluster in the console and experienced it in depth. The performance of the following dimensions is the most impressive:
1. Operation and maintenance difficulty: from "hell mode" to "one-button fool"
Anyone who has played open source ClickHouse knows that its distributed cluster (Distributed Table) relies heavily on ZooKeeper for metadata synchronization and consistency collaboration. When the amount of data
When it is extremely large, the ZooKeeper often falls off the chain. Once it gets stuck, the entire ClickHouse cluster will be paralyzed.
Tencent Cloud's solution: Tencent Cloud provides a fully managed architecture with deep optimization and isolation of the ZooKeeper at the bottom.
Practical experience: to create a cluster in the console, you only need to select the configuration (several cores, several g, several nodes), and the entire distributed cluster will be built in a few minutes. For complex underlying configurations such as replica synchronization and sharding rules, Tencent Cloud has done a good job of best practices (Best Practice) for you during initialization. You don't need to look at those hundreds of lines of XML configuration files anymore, saving countless hairs from your operation and maintenance classmates.
2. Expansion and Flexibility: Finally, there is no need to stay up late to move data.
The biggest historical burden of open source ClickHouse is
does not support the real shrink
. Because it is a "computing and storage coupling" architecture, once the hard disk is full to add a machine, you need to manually change the configuration file, but also write a script to migrate the physical data fragments (Parts) on the old machine to the new machine, the process is comparable to changing the engine in the air, a little careless data will be lost.
Tencent Cloud's solution: Tencent Cloud realizes the separation of elastic computing and computing storage (some versions support).
Actual experience: When our test data volume increases sharply, click "Change Configuration" in the console to directly add nodes or expand cloud disks online. In the whole process, the rebalancing (Rebalance) of data is automatically scheduled by Tengxun cloud background, and the query at the business level is hardly affected. This alone is worth the fare back.
3. Console and visualization: finally have a decent "dashboard"
Open source ClickHouse have only one cold command-line client (clickhouse-client) by default. Want to see how much CPU is running in the cluster now? Which query is bursting the memory? Sorry, I have to check the system table myself.
system.processes
Or build your own Prometheus Grafana.
Tengxunyun's solution: Tengxunyun comes with a very complete monitoring market and data management service DMC.
Actual experience: Log on to the console, the throughput, read/write latency, and disk usage of the cluster are clear at a glance. The best thing is its slow query analysis function. If a SQL runs for 10 seconds without a result, the console will directly capture it and show a detailed execution plan to tell you which Join is stuck on. This is an artifact for developers to tune SQL.
3. actual combat scene: what is the most suitable for tengxunyun ClickHouse?
In our actual business evaluation, the following three scenarios ClickHouse showed overwhelming advantages:
Scenario 1: massive log and audit analysis (kill ELK)
I used to do log analysis before, and everyone is used to it.
With ELK(Elasticsearch Logstash Kibana). However, Elasticsearch is extremely memory-eating and has a high expansion rate (100G of original log may become 200G).
ClickHouse record: Pouring the same billions of user behavior logs into ClickHouse, combined with its ultra-high data compression ratio of 1:5 or even 1:10, takes up less than 1/3 of ES's hard disk space. In addition, checking a wide range of aggregated data (such as counting the error reporting trend of an interface last month) is ClickHouse several times faster than ES.
Scene 2: Advertising and Fine Operation (Crowd Selection)
Operation students often have to ask for information: "Help me circle Beijing users who have logged into App, recharged more than 100 yuan and are between 18 and 25 years old in the past 7 days."
ClickHouse results: This label-based (Bitmap) multi-dimensional funnel analysis is ClickHouse's specialty. Using its built-in advanced functions such as bitmapAnd and bitmapOr, tens of billions of people can circle the results in a few seconds, and the operation students no longer have to wait for the next day to get the data after raising the demand.
4. Tengxunyun ClickHouse's "Back of Coins": A Pit That Beginners Must Avoid
Although tengxunyun encapsulates it very well, ClickHouse is ClickHouse after all, and its underlying "physical characteristics" determine that it is not omnipotent. Novice in use, do not use it as MySQL to use, the following minefields must be bypassed:
Never do high concurrent small throughput writing: ClickHouse like "mass overeating" and don't like "eating less and eating more". If you write 1000 times per second and only write one piece of data at a time, ClickHouse the background will merge data shards (Merge) crazily and will soon report a fatal error of Too many parts causing the cluster to hang up. Real people suggest that you must make a local cache (Buffer) in the business layer, or save batches through Kafka, and write at least 10000 or more pieces in each batch as a whole.
It's not good at high-concurrency high-definition point queries: ClickHouse is a beast of a partial family. You let it calculate the sum of 1 billion pieces of data, it 0.5 seconds to you. However, if you want to build an App with tens of thousands of concurrent amounts and let it check the basic information of a specific user according to the user ID (SELECT * FROM table WHERE id = 123), it will fill up the CPU instead. Real people suggest: this kind of high concurrency point check business, honestly use Redis or MySQL.
extremely limited concurrent query capabilities (Max
Concurrent Queries): The ClickHouse default limit for concurrent queries is 100. Because a complex SQL will mobilize all the CPU cores behind it to full capacity. If 100 people submit complex report queries at the same time, the cluster will be stuck directly. It is suitable for data analysts, operations and internal billboards, and is not suitable for large concurrent calls directly to millions of active users on the C side.
5. summary and selection suggestions
After some in-depth evaluation, the overall feeling of Tencent Cloud ClickHouse(CDCH) is:
The flaw is not hidden, the flaw is in the open source gene, and Yu is processed in Tencent.
It inherits the adrenaline-pumping extreme query performance of open source ClickHouse. At the same time, it completely removed the three most criticized mountains of "difficult operation and maintenance, difficult expansion and difficult monitoring" through cloud-native hosting.
Finally, give a word selection suggestion:
If your business data volume has exceeded 10 million or even 100 million levels, the traditional MySQL run a report to turn a few minutes, and you do not have the extra budget to raise a dedicated big data operations team, then,
Direct access to Tencent Cloud ClickHouse
. It can use extremely low hardware and labor costs to bring your enterprise to experience the "push back feeling" of the big data era in advance ".
