Merge pull request #4162 from ClickHouse/ks/kafka-clickpipe-benchmarks

ClickPipes Benchmarks, Vertical Scaling FAQs, and additional pricing information

Author: tpanetti

docs/cloud/manage/billing.md

@@ -458,12 +458,12 @@ This section outlines the pricing model of ClickPipes for streaming and object s
 
 #### What does the ClickPipes pricing structure look like? {#what-does-the-clickpipes-pricing-structure-look-like}
 
-It consists of two dimensions
+It consists of two dimensions:
 
-- **Compute**: Price per unit per hour
+- **Compute**: Price **per unit per hour**.
   Compute represents the cost of running the ClickPipes replica pods whether they actively ingest data or not.
   It applies to all ClickPipes types.
-- **Ingested data**: per GB pricing
+- **Ingested data**: Price **per GB**.
   The ingested data rate applies to all streaming ClickPipes
   (Kafka, Confluent, Amazon MSK, Amazon Kinesis, Redpanda, WarpStream, Azure Event Hubs)
   for the data transferred via the replica pods. The ingested data size (GB) is charged based on bytes received from the source (uncompressed or compressed).
@@ -476,17 +476,27 @@ For this reason, it uses dedicated compute replicas.
 
 #### What is the default number of replicas and their size? {#what-is-the-default-number-of-replicas-and-their-size}
 
-Each ClickPipe defaults to 1 replica that is provided with 2 GiB of RAM and 0.5 vCPU.
-This corresponds to **0.25** ClickHouse compute units (1 unit = 8 GiB RAM, 2 vCPUs).
+Each ClickPipe defaults to 1 replica that is provided with 512 MiB of RAM and 0.125 vCPU (XS).
+This corresponds to **0.0625** ClickHouse compute units (1 unit = 8 GiB RAM, 2 vCPUs).
 
 #### What are the ClickPipes public prices? {#what-are-the-clickpipes-public-prices}
 
-- Compute: \$0.20 per unit per hour (\$0.05 per replica per hour)
+- Compute: \$0.20 per unit per hour (\$0.0125 per replica per hour for the default replica size)
 - Ingested data: \$0.04 per GB
 
+The price for the Compute dimension depends on the **number** and **size** of replica(s) in a ClickPipe. The default replica size can be adjusted using vertical scaling, and each replica size is priced as follows:
+
+| Replica Size               | Compute Units | RAM     | vCPU   | Price per Hour |
+|----------------------------|---------------|---------|--------|----------------|
+| Extra Small (XS) (default) | 0.0625        | 512 MiB | 0.125. | $0.0125        |
+| Small (S)                  | 0.125         | 1 GiB   | 0.25   | $0.025         |
+| Medium (M)                 | 0.25          | 2 GiB   | 0.5    | $0.05          |
+| Large (L)                  | 0.5           | 4 GiB   | 1.0    | $0.10          |
+| Extra Large (XL)           | 1.0           | 8 GiB   | 2.0    | $0.20          |
+
 #### How does it look in an illustrative example? {#how-does-it-look-in-an-illustrative-example}
 
-The following examples assume a single replica unless explicitly mentioned.
+The following examples assume a single M-sized replica, unless explicitly mentioned.
 
 <table><thead>
   <tr>

docs/cloud/manage/jan2025_faq/_snippets/_clickpipes_faq.md

@@ -52,9 +52,9 @@ This corresponds to **0.25** ClickHouse compute units (1 unit = 8 GiB RAM, 2 vCP
 
 <summary>Can ClickPipes replicas be scaled?</summary>
 
-ClickPipes for streaming can be scaled horizontally
-by adding more replicas each with a base unit of **0.25** ClickHouse compute units.
-Vertical scaling is also available on demand for specific use cases (adding more CPU and RAM per replica).
+Yes, ClickPipes for streaming can be scaled both horizontally and vertically.
+Horizontal scaling adds more replicas to increase throughput, while vertical scaling increases the resources (CPU and RAM) allocated to each replica to handle more intensive workloads.
+This can be configured during ClickPipe creation, or at any other point under **Settings** -> **Advanced Settings** -> **Scaling**.
 
 </details>
 
@@ -142,4 +142,4 @@ The philosophy behind ClickPipes pricing is
 to cover the operating costs of the platform while offering an easy and reliable way to move data to ClickHouse Cloud.
 From that angle, our market analysis revealed that we are positioned competitively.
 
-</details>
+</details>

docs/integrations/data-ingestion/clickpipes/kafka/04_best_practices.md

@@ -121,12 +121,28 @@ ClickPipes does not provide any guarantees concerning latency. If you have speci
 
 ### Scaling {#scaling}
 
-ClickPipes for Kafka is designed to scale horizontally. By default, we create a consumer group with one consumer.
-This can be changed with the scaling controls in the ClickPipe details view.
+ClickPipes for Kafka is designed to scale horizontally and vertically. By default, we create a consumer group with one consumer. This can be configured during ClickPipe creation, or at any other point under **Settings** -> **Advanced Settings** -> **Scaling**.
 
 ClickPipes provides a high-availability with an availability zone distributed architecture.
 This requires scaling to at least two consumers.
 
 Regardless number of running consumers, fault tolerance is available by design.
 If a consumer or its underlying infrastructure fails,
 the ClickPipe will automatically restart the consumer and continue processing messages.
+
+### Benchmarks {#benchmarks}
+
+Below are some informal benchmarks for ClickPipes for Kafka that can be used to get a general idea of the baseline performance. It's important to know that many factors can impact performance, including message size, data types, and data format. Your mileage may vary, and what we show here is not a guarantee of actual performance.
+
+Benchmark details:
+
+- We used production ClickHouse Cloud services with enough resources to ensure that throughput was not bottlenecked by the insert processing on the ClickHouse side.
+- The ClickHouse Cloud service, the Kafka cluster (Confluent Cloud), and the ClickPipe were all running in the same region (`us-east-2`).
+- The ClickPipe was configured with a single L-sized replica (4 GiB of RAM and 1 vCPU).
+- The sample data included nested data with a mix of `UUID`, `String`, and `Int` datatypes. Other datatypes, such as `Float`, `Decimal`, and `DateTime`, may be less performant.
+- There was no appreciable difference in performance using compressed and uncompressed data.
+
+| Replica Size  | Message Size | Data Format | Throughput |
+|---------------|--------------|-------------|------------|
+| Large (L)     | 1.6kb        |   JSON      | 63mb/s     |
+| Large (L)     | 1.6kb        |   Avro      | 99mb/s     |

docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md

@@ -54,6 +54,15 @@ No, the ClickPipes for Kafka is designed for reading data from Kafka topics, not
 Yes, if the brokers are part of the same quorum they can be configured together delimited with `,`.
 </details>
 
+<details>
+
+<summary>Can ClickPipes replicas be scaled?</summary>
+
+Yes, ClickPipes for streaming can be scaled both horizontally and vertically.
+Horizontal scaling adds more replicas to increase throughput, while vertical scaling increases the resources (CPU and RAM) allocated to each replica to handle more intensive workloads.
+This can be configured during ClickPipe creation, or at any other point under **Settings** -> **Advanced Settings** -> **Scaling**.
+</details>
+
 ### Upstash {#upstash}
 
 <details>

docs/integrations/data-ingestion/clickpipes/kinesis.md

@@ -159,10 +159,9 @@ If you have specific low-latency requirements, please [contact us](https://click
 
 ### Scaling {#scaling}
 
-ClickPipes for Kinesis is designed to scale horizontally. By default, we create a consumer group with one consumer.
-This can be changed with the scaling controls in the ClickPipe details view.
+ClickPipes for Kinesis is designed to scale both horizontally and vertically. By default, we create a consumer group with one consumer. This can be configured during ClickPipe creation, or at any other point under **Settings** -> **Advanced Settings** -> **Scaling**.
 
-ClickPipes provides a high-availability with an availability zone distributed architecture.
+ClickPipes provides high-availability with an availability zone distributed architecture.
 This requires scaling to at least two consumers.
 
 Regardless number of running consumers, fault tolerance is available by design.