Skip to content

Commit 07e3af1

Browse files
javiergoodrootemrberk
authored
QuestDB Architecture Guide (#146)
Co-authored-by: goodroot <9484709+goodroot@users.noreply.github.com> Co-authored-by: emrberk <emreberk5699@gmail.com> Co-authored-by: Emre Berk Kaya <75899391+emrberk@users.noreply.github.com>
1 parent 7620e4c commit 07e3af1

37 files changed

+793
-41
lines changed

‎documentation/concept/interval-scan.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ returned from a sub-query is not guaranteed to be in timestamp order.
4242
<Screenshot
4343
alt="Interval scan."
4444
height={433}
45-
src="images/blog/2023-04-25/interval_scan.svg"
45+
src="images/blog/2023-04-25/intervalScan.webp"
4646
width={650}
4747
/>
4848

‎documentation/concept/partitions.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ import Screenshot from "@theme/Screenshot"
1818
<Screenshot
1919
alt="Diagram of data column files and how they are partitioned to form a table"
2020
height={373}
21-
src="images/docs/concepts/partitionModel.svg"
22-
width={745}
21+
src="images/docs/concepts/partitionModel.webp"
22+
width={700}
2323
forceTheme="dark"
2424
/>
2525

‎documentation/concept/storage-model.md‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ import Screenshot from "@theme/Screenshot"
2525
<Screenshot
2626
alt="Architecture of the file storing a column"
2727
height={435}
28-
src="images/docs/concepts/columnUpdate.svg"
28+
src="images/docs/concepts/columnUpdate.webp"
2929
width={745}
3030
/>
3131

@@ -39,7 +39,7 @@ mapped memory page, where the required value is read from.
3939
<Screenshot
4040
alt="Diagram showing how the data from a column file is mapped to the memory"
4141
height={447}
42-
src="images/docs/concepts/columnRead.svg"
42+
src="images/docs/concepts/columnRead.webp"
4343
width={745}
4444
/>
4545

@@ -96,10 +96,7 @@ either randomly (via queries) or incrementally (as a data queue). QuestDB
9696
provides a variety of reader implementations.
9797

9898
<Screenshot
99-
alt="Architecture of the storage model with column files, readers/writers and the mapped memory"
100-
height={596}
101-
src="images/docs/concepts/storageSummarized.svg"
102-
width={745}
99+
src="images/docs/concepts/storageSummary.webp"
103100
/>
104101

105102
<span />

‎documentation/guides/active-directory-pingfederate.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Active Directory (PingFederate) guide
2+
title: Active Directory (PingFederate)
33
description: ""
44
---
55

‎documentation/guides/architecture/data-ingestion-engine.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
---
2+
title: Data Ingestion Engine
3+
slug: data-ingestion
4+
description: The QuestDB Data Ingestion Engine supports bulk and streaming ingestion. It writes data to a row-based write-ahead log (WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments ship to object storage for replication.
5+
---
6+
7+
8+
## Data ingestion & write path
9+
10+
The QuestDB Data Ingestion Engine supports bulk and streaming ingestion. It writes data to a row-based write-ahead log
11+
(WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments ship to object storage for replication.
12+
13+
### Bulk ingestion
14+
15+
- **CSV ingestion:**
16+
QuestDB offers a CSV ingestion endpoint via the [REST API](/docs/reference/api/rest/) and web console.
17+
A specialized COPY command uses [io_uring](/blog/2022/09/12/importing-300k-rows-with-io-uring) on
18+
fast drives to speed up ingestion.
19+
20+
### Real-time streaming
21+
22+
- **High-frequency writes:**
23+
The streaming ingestion path handles millions of rows per second with non-blocking I/O.
24+
25+
- **Durability:**
26+
As seen above, the system writes data to a row-based write-ahead log (WAL) and then converts it
27+
into column-based files for efficient reads. At the expense of performance, `sync` mode can be
28+
enabled on commit for extra durability.
29+
30+
- **Concurrent writes:**
31+
Multiple connections writing to the same table create parallel WAL files that the engine
32+
later consolidates into columnar storage.
33+
34+
```text
35+
36+
Contents of the `db` folder, showing multiple pending WAL files,
37+
and the binary columnar data.
38+
39+
├── db
40+
│ ├── Table
41+
│ │ │
42+
│ │ ├── Partition 1
43+
│ │ │ ├── _archive
44+
│ │ │ ├── column1.d
45+
│ │ │ ├── column2.d
46+
│ │ │ ├── column2.k
47+
│ │ │ └── ...
48+
│ │ ├── Partition 2
49+
│ │ │ ├── _archive
50+
│ │ │ ├── column1.d
51+
│ │ │ ├── column2.d
52+
│ │ │ ├── column2.k
53+
│ │ │ └── ...
54+
│ │ ├── txn_seq
55+
│ │ │ ├── _meta
56+
│ │ │ ├── _txnlog
57+
│ │ │ └── _wal_index.d
58+
│ │ ├── wal1
59+
│ │ │ └── 0
60+
│ │ │ ├── _meta
61+
│ │ │ ├── _event
62+
│ │ │ ├── column1.d
63+
│ │ │ ├── column2.d
64+
│ │ │ └── ...
65+
│ │ ├── wal2
66+
│ │ │ └── 0
67+
│ │ │ │ ├── _meta
68+
│ │ │ │ ├── _event
69+
│ │ │ │ ├── column1.d
70+
│ │ │ │ ├── column2.d
71+
│ │ │ │ └── ...
72+
│ │ │ └── 1
73+
│ │ │ ├── _meta
74+
│ │ │ ├── _event
75+
│ │ │ ├── column1.d
76+
│ │ │ ├── column2.d
77+
│ │ │ └── ...
78+
│ │ │
79+
│ │ ├── _meta
80+
│ │ ├── _txn
81+
│ │ └── _cv
82+
83+
```
84+
85+
### Ingestion via ILP protocol
86+
87+
- **Native ILP integration:**
88+
QuestDB supports the [Influx Line Protocol](/docs/reference/api/ilp/overview/)
89+
(ILP) for high-speed data ingestion.
90+
91+
- **Extensions to ILP:**
92+
QuestDB extends ILP to support different timestamp units and the array data type.
93+
94+
- **Minimal parsing overhead:**
95+
The ILP parser quickly maps incoming data to internal structures.
96+
97+
- **Parallel ingestion:**
98+
The ILP path uses off-heap buffers and direct memory management to bypass JVM heap allocation.
99+
100+
- **Protocol versatility:**
101+
In addition to ILP, QuestDB also supports ingestion via [REST](/docs/reference/sql/overview/#rest-http-api)
102+
and [PostgreSQL wire](/docs/reference/sql/overview/#postgresql) protocols.
103+
104+
105+
## Next Steps
106+
107+
- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
108+
- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
109+
- [QuestDB Documentation](/docs)
110+

‎documentation/guides/architecture/memory-management.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
title: Memory Management
3+
slug: memory-management
4+
description: QuestDB leverages both memory mapping and explicit memory management techniques, and integrates native code for performance-critical tasks.
5+
---
6+
7+
8+
## Memory management and native integration
9+
10+
QuestDB leverages both memory mapping and explicit memory management techniques, and integrates native code for performance-critical tasks.
11+
12+
### Memory-mapped files
13+
14+
- **Direct OS integration:**
15+
Memory-mapped files let QuestDB use the operating system's page cache. This reduces explicit
16+
I/O calls and speeds up sequential reads.
17+
18+
- **Sequential access:**
19+
When data partitions by incremental timestamp, memory mapping ensures that reads are
20+
sequential and efficient.
21+
22+
### Direct memory management and native integration
23+
24+
- **Off-heap memory usage:**
25+
QuestDB allocates direct memory via memory mapping and low-level APIs (such as Unsafe) to
26+
bypass the JVM garbage collector. This reduces latency spikes and garbage collection delays.
27+
28+
- **Hotpath efficiency:**
29+
The system pre-allocates and reuses memory in critical code paths, avoiding dynamic allocation
30+
on the hotpath.
31+
32+
- **Native code integration:**
33+
QuestDB uses native libraries written in C++ and Rust for performance-critical tasks. These
34+
native components share off-heap buffers with Java via JNI.
35+
- **Zero-copy interoperability:**
36+
Sharing memory between Java and native code minimizes data copying and reduces latency.
37+
- **Hybrid architecture:**
38+
This integration lets QuestDB use Java for rapid development and C++/Rust for low-level,
39+
high-performance routines.
40+
41+
## Next Steps
42+
43+
- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
44+
- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
45+
- [QuestDB Documentation](/docs)
46+
47+
48+

‎documentation/guides/architecture/networking-layer.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
title: Networking Layer
3+
slug: networking-layer
4+
description: The system exposes RESTful APIs and implements ILP and PostgreSQL wire protocols so that existing tools and drivers work out-of-the-box. It also offers a health and metrics endpoint.
5+
---
6+
7+
8+
## Networking layer
9+
10+
QuestDB exposes several network interfaces and protocols to allow different client applications to interact with the database
11+
12+
### InfluxDB Line protocol (ILP) over HTTP or TCP
13+
14+
The [Influx Line Protocol](/docs/reference/api/ilp/overview/) allows for very high throughput of incoming data. It supports
15+
both HTTP (recommended) or TCP. QuestDB provides official clients in seven different programming languages, as well as
16+
integrations with third-party tools like Apache Kafka, Apache Flink, or Telegraf. Any ILP-compatible library can be used
17+
for ingesting data into QuestDB over HTTP.
18+
19+
The default port number for ILP over HTTP is `9000`, and for ILP over TCP is `9009`.
20+
21+
### PostgreSQL wire protocol
22+
23+
QuestDB exposes a [PostgreSQL wire](/docs/reference/sql/overview/#postgresql) protocol, which can be used to send SQL
24+
statements both for data definition or for data manipulation. When used for data ingestion, throughput is noticeably
25+
lower than using the ILP protocol.
26+
27+
QuestDB implements the wire protocol, allowing many third-party libraries to query QuestDB directly. Some client libraries
28+
might be incompatible if they rely heavily on PostgreSQL metadata, as QuestDB implements only a subset of it. For an
29+
overview of some key differences on QuestDB schema design, please visit our
30+
[Schema Design Essentials](/docs/guides/schema-design-essentials/) guide.
31+
32+
The default port number for the pg-wire interface is `8812`.
33+
34+
### HTTP Rest API
35+
36+
QuestDB [REST API](/docs/reference/sql/overview/#rest-http-api) can be used to issue SQL statements over HTTP. It also
37+
exposes and endpoint for importing CSV files, and for exporting tables and query results.
38+
39+
The default port number for the REST API is `9000`.
40+
41+
### Minimal HTTP server for health-check and metrics
42+
43+
QuestDB exposes an HTTP interface for monitoring. Please see the [Observability](/docs/guides/architecture/observability) section
44+
for more information.
45+
46+
The default port number for the minimal HTTP server is `9003`.
47+
48+
## Next Steps
49+
50+
- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
51+
- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
52+
- [QuestDB Documentation](/docs)
53+
54+

‎documentation/guides/architecture/observability.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
title: Observability
3+
slug: observability
4+
description: QuestDB provides real-time metrics, a health check endpoint, and logging to monitor performance and simplify troubleshooting.
5+
---
6+
7+
8+
## Observability & diagnostics
9+
10+
QuestDB provides real-time metrics, a health check endpoint, and logging to monitor performance and simplify troubleshooting.
11+
12+
- **Metrics:**
13+
QuestDB exposes detailed [metrics in Prometheus format](/docs/operations/logging-metrics/#metrics), including query
14+
statistics, memory usage, and I/O details.
15+
16+
- **Health check:**
17+
A [minimal HTTP server](/docs/operations/logging-metrics/#minimal-http-server) monitors system health.
18+
19+
- **Metadata tables:**
20+
The engine provides [metadata tables](/docs/reference/function/meta/) to query
21+
table status, partition status, query execution, and latency.
22+
23+
- **Extensive logging:**
24+
[Logging](/docs/operations/logging-metrics/) covers SQL parsing, execution, background processing, and runtime exceptions. The framework minimizes performance impact.
25+
26+
- **Real-time metric dashboards:**
27+
The web console lets you create dashboards that display per-table metrics.
28+
29+
<Screenshot
30+
alt="Metric dashboard at the QuestDB Console"
31+
title="Metric dashboard at the QuestDB Console"
32+
height={447}
33+
src="images/guides/questdb-internals/telemetry.webp"
34+
width={745}
35+
/>
36+
37+
38+
## Next Steps
39+
40+
- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
41+
- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
42+
- [QuestDB Documentation](/docs)
43+

0 commit comments

Comments
 (0)