better readme

Author: jerrinot

Dockerfile-Sample-Faker

@@ -1,4 +1,4 @@
-FROM maven:3.8-jdk-8-slim AS dep-cache
+FROM maven:3.8-jdk-11-slim AS dep-cache
 WORKDIR /opt
 COPY ./pom.xml /opt/pom.xml
 COPY ./connector/pom.xml /opt/connector/pom.xml

kafka-questdb-connector-samples/faker/readme.md

@@ -1,3 +1,67 @@
+# Sample Project: Kafka -> QuestDB
+## What does the sample project do?
+This code sample show the simplest usage of the Kafka QuestDB connector. It uses a simple node.js application to generate random data and send it to a Kafka topic. The connector is configured to read from the topic and write the data to a QuestDB table.
+
+## Prerequisites
+- Git
+- Working Docker environment, including docker-compose
+- Internet access to download dependencies
+
+The project was tested on MacOS with M1, but it should work on other platforms too. Please open a new issue if it's not working for you.
+
+## Usage:
+1. Clone this repository via `git clone https://github.com/questdb/kafka-questdb-connector.git`
+2. `cd kafka-questdb-connector/kafka-questdb-connector-samples/faker/` to enter the directory with this sample.
+3. Run `docker-compose build` to build a docker image with the sample project.
+4. Run `docker-compose up` to start the node.js producer, Apache Kafka and QuestDB containers.
+5. The previous command will generate a lot of log messages. Eventually logging should cease. This means both Apache Kafka and QuestDB are running. The last log message should contain the following text: `Session key updated`
+6. Execute the following command in shell: 
+    ```shell
+    $ curl -X POST -H "Content-Type: application/json" -d '{"name":"questdb-connect","config":{"topics":"People","connector.class":"io.questdb.kafka.QuestDBSinkConnector","tasks.max":"1","key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false","host":"questdb", "timestamp.field.name": "birthday", "transforms":"convert_birthday","transforms.convert_birthday.type":"org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.convert_birthday.target.type":"Timestamp","transforms.convert_birthday.field":"birthday","transforms.convert_birthday.format": "yyyy-MM-dd'"'"'T'"'"'HH:mm:ss.SSSX"}}' localhost:8083/connectors
+    ```
+7. The command above will create a new Kafka connector that will read data from the `People` topic and write it to a QuestDB table called `People`. The connector will also convert the `birthday` field to a timestamp.
+8. Go to the [QuestDB console](http://localhost:19000) and run `select * from 'People';` and you should see some rows.
+9. Congratulations! You have successfully created a Kafka connector that reads data from a Kafka topic and writes it to a QuestDB table!
+
+## How does it work?
+The sample project consists of 3 components:
+1. [Node.js](index.js) application that generates random JSON data and sends it to a Kafka topic. Each generated JSON looks similar to this:
+    ```json
+    {"firstname":"John","lastname":"Doe","birthday":"1970-01-12T10:25:12.052Z"}
+    ```
+2. [Docker-compose](docker-compose.yml) file that starts the node.js application, Apache Kafka, Kafka Connect and QuestDB containers.
+3. Kafka Connect configuration that defines the connector that reads data from the Kafka topic and writes it to a QuestDB table. The configuration is submitted into a running Kafka Connect cluster via the REST API.
+
+The Kafka Connect configuration looks complex, but it's quite simple. Let's have a closer look. This is how the `curl` command looks like:
 ```shell
-curl -X POST -H "Content-Type: application/json" -d '{"name":"questdb-connect","config":{"topics":"People","connector.class":"io.questdb.kafka.QuestDBSinkConnector","tasks.max":"1","key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false","host":"questdb", "timestamp.field.name": "birthday", "transforms":"convert_birthday","transforms.convert_birthday.type":"org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.convert_birthday.target.type":"Timestamp","transforms.convert_birthday.field":"birthday","transforms.convert_birthday.format": "yyyy-MM-dd'"'"'T'"'"'HH:mm:ss.SSSX"}}' localhost:8083/connectors
-```
+$ curl -X POST -H "Content-Type: application/json" -d '{"name":"questdb-connect","config":{"topics":"People","connector.class":"io.questdb.kafka.QuestDBSinkConnector","tasks.max":"1","key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false","host":"questdb", "timestamp.field.name": "birthday", "transforms":"convert_birthday","transforms.convert_birthday.type":"org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.convert_birthday.target.type":"Timestamp","transforms.convert_birthday.field":"birthday","transforms.convert_birthday.format": "yyyy-MM-dd'"'"'T'"'"'HH:mm:ss.SSSX"}}' localhost:8083/connectors
+```
+
+It uses `curl` to submit a following JSON to Kafka Connect:
+```json
+{
+  "name": "questdb-connect",
+  "config": {
+    "topics": "People",
+    "connector.class": "io.questdb.kafka.QuestDBSinkConnector",
+    "tasks.max": "1",
+    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
+    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
+    "value.converter.schemas.enable": "false",
+    "host": "questdb",
+    "timestamp.field.name": "birthday",
+    "transforms": "convert_birthday",
+    "transforms.convert_birthday.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
+    "transforms.convert_birthday.target.type": "Timestamp",
+    "transforms.convert_birthday.field": "birthday",
+    "transforms.convert_birthday.format": "yyyy-MM-dd'T'HH:mm:ss.SSSX"
+  }
+}
+```
+Most of the fields are self-explanatory. The `transforms` field is a bit more complex. It defines a [Kafka Connect transformation](https://docs.confluent.io/platform/current/connect/transforms/index.html) that converts the `birthday` string field to a timestamp. The `transforms.convert_birthday.format` field defines the format of the date. The format is a [Java SimpleDateFormat](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html) pattern. The `transforms.convert_birthday.field` field defines the name of the field that should be converted. The `transforms.convert_birthday.target.type` field defines the type of the field after the conversion. In this case, it's a timestamp.
+
+The other potentially non-obvious configuration elements are:
+1. `"value.converter.schemas.enable": "false"` It disables the schema support in the Kafka Connect JSON converter. The sample project doesn't use schemas.
+2. `"host": "questdb"` It defines the hostname of the QuestDB instance. The hostname is defined in the [docker-compose.yml](docker-compose.yml) file.
+3. `"timestamp.field.name": "birthday"` It defines the name of the field that should be used as a timestamp. It uses the field that was converted by the Kafka Connect transformation described above.
+4. The ugly value in `"transforms.convert_birthday.format": "yyyy-MM-dd'"'"'T'"'"'HH:mm:ss.SSSX"`. This part looks funny: `'"'"'T'"'"'`. In fact, it's a way to submit an apostrophe via shell which uses apostrophes to define strings. The apostrophe is required to escape the `T` character in the date format. The date format is `yyyy-MM-dd'T'HH:mm:ss.SSSX`. If you know a better way to submit the same JSON then please [open a new issue](https://github.com/questdb/kafka-questdb-connector/issues/new). 

kafka-questdb-connector-samples/readme.md

@@ -0,0 +1,5 @@
+# Samples Projects
+
+There is 1 sample projects.
+## [Faker](faker)
+Simplistic project which uses a simple node.js application to create JSON entries in Apache Kafka and QuestDB Kafka Connect Sink to feed generated data from Kafka to QuestDB.

pom.xml

@@ -16,8 +16,8 @@
 
     <properties>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <maven.compiler.source>11</maven.compiler.source>
-        <maven.compiler.target>11</maven.compiler.target>
+        <maven.compiler.source>8</maven.compiler.source>
+        <maven.compiler.target>8</maven.compiler.target>
         <kafka.scala.version>2.13</kafka.scala.version>
         <kafka.version>3.3.1</kafka.version>
         <junit.version>5.9.0</junit.version>

readme.md

@@ -2,6 +2,10 @@
 The connector reads data from Kafka topics and writes it to [QuestDB](https://questdb.io/) tables.
 The connector implements Apache Kafka [Sink Connector API](https://kafka.apache.org/documentation/#connect_development).
 
+## Pre-requisites
+* QuestDB 6.5.0 or newer
+* Apache Kafka 2.8.0 or newer, running on Java 11 or newer.
+
 ## Usage with Kafka Connect
 This guide assumes you are already familiar with Apache Kafka and Kafka Connect. If you are not then watch this [excellent video](https://www.youtube.com/watch?v=Jkcp28ki82k) or check our [sample projects](kafka-questdb-connector-samples).
 - Unpack connector ZIP into Kafka Connect `./plugin/` directory.
@@ -80,19 +84,19 @@ In production, it's recommended to [create tables manually via SQL](https://ques
 
 ## FAQ
 <b>Q</b>: Does this connector work with Schema Registry?
-
+<br/>
 <b>A</b>: The Connector does not care about serialization strategy used. It relies on Kafka Connect converters to deserialize data. Converters can be configured using `key.converter` and `value.converter` options, see the configuration section.
 
 <b>Q</b>: I'm getting this error: `org.apache.kafka.connect.errors.DataException: JsonConverter with schemas.enable requires "schema" and "payload" fields and may not contain additional fields. If you are trying to deserialize plain JSON data, set schemas.enable=false in your converter configuration.`
-
+<br/>
 <b>A</b>: This error means that the connector is trying to deserialize data using a converter that expects a schema. The connector does not use schemas, so you need to configure the converter to not expect a schema. For example, if you are using JSON converter, you need to set `value.converter.schemas.enable=false` or `key.converter.schemas.enable=false` in the connector configuration. 
 
 <b>Q</b>: Does this connector work with Debezium?
-
+<br/>
 <b>A</b>: Yes, it's been tested with Debezium as a source. Bear in mind that QuestDB is meant to be used as append-only database hence updates should be translated as new inserts. The connector supports Debezium's `ExtractNewRecordState` transformation to extract the new state of the record. The transform by default drops DELETE events so no need to handle it explicitly.
 
 <b>Q</b>: How I can select which fields to include in the target table?
-
+<br/>
 <b>A</b>: Use the ReplaceField transformation to remove unwanted fields. For example, if you want to remove the `address` field from the example above, you can use the following configuration:
 ```json
 {
@@ -112,4 +116,8 @@ In production, it's recommended to [create tables manually via SQL](https://ques
   }
 }
 ```
-See [ReplaceField documentation](https://docs.confluent.io/platform/current/connect/transforms/replacefield.html#replacefield) for more details.
+See [ReplaceField documentation](https://docs.confluent.io/platform/current/connect/transforms/replacefield.html#replacefield) for more details.
+
+<b>Q</b>: I need to run Kafka Connect on Java 8, but the connector says it requires Java 11. What should I do?
+<br/>
+<b>A</b>: The Kafka Connect-specific part of the connectors works with Java 8. The requirement for Java 11 is coming from QuestDB client itself. The zip archive contains 2 JARs: `questdb-kafka-connector-<version>.jar` and `questdb-<version>.jar`. You can use replace the latter with `questdb-<version>-jdk8.jar` from the [Maven central](https://mvnrepository.com/artifact/org.questdb/questdb/6.5.3-jdk8). Bear in mind this setup is not officially supported and you may encounter issues. If you do, please report them to us.