Configurable batch size (#393)

* Configurable batch size
* Keep 1.5s of data in cache
* Remove minimum 1024 cache size for faster rebalances in case of long processing time
* Changed property name to `js.consumer.max.batch.size`
* Use only 1.5 seconds cache size estimation aligned at batch size * concurrency
* Fix for at-least-once guarantee not ensured in case a seek happens on one partition and there are messages being fetched for other partitions
* Configurable cache size in milliseconds
* Add worker identifier to the payload for better debugging
* Fix for test flakyness
* Make `is cleared before rebalance` less flaky in case of increased time before first assignment
* Reduce flakyness of 'times out if messages are pending'
* FIXME about KIP-848 autocommit issue
* v1.7.0

Author: emasab

CHANGELOG.md

@@ -1,3 +1,16 @@
+# confluent-kafka-javascript 1.7.0
+
+v1.7.0 is a feature release. It is supported for all usage.
+
+### Enhancements
+
+1. Configurable batch size through the `js.consumer.max.batch.size` property 
+   and cache size through the `js.consumer.max.cache.size.per.worker.ms` 
+   property (#393).
+2. Fix for at-least-once guarantee not ensured in case a seek happens on one
+partition and there are messages being fetched about other partitions (#393).
+
+
 # confluent-kafka-javascript 1.6.0
 
 v1.6.0 is a feature release. It is supported for all usage.

MIGRATION.md

@@ -303,9 +303,14 @@ producerRun().then(consumerRun).catch(console.error);
   - The `heartbeat()` no longer needs to be called by the user in the `eachMessage/eachBatch` callback.
     Heartbeats are automatically managed by librdkafka.
   - The `partitionsConsumedConcurrently` is supported by both `eachMessage` and `eachBatch`.
-  - An API compatible version of `eachBatch` is available, but the batch size calculation is not
-    as per configured parameters, rather, a constant maximum size is configured internally. This is subject
-    to change.
+  - An API compatible version of `eachBatch` is available, maximum batch size
+    can be configured through the `js.consumer.max.batch.size` configuration property
+    and defaults to 32.  It is not dependent on the size of the produced batches
+    present on the broker, as these are constructed client-side. Similar to
+    the Java client configuration `max.poll.records`.
+    `js.consumer.max.cache.size.per.worker.ms` allows to
+    configure the cache size estimated based on consumption rate and defaults
+    to the cache being sized to 1.5s worth of messages.
     The property `eachBatchAutoResolve` is supported.
     Within the `eachBatch` callback, use of `uncommittedOffsets` is unsupported,
     and within the returned batch, `offsetLag` and `offsetLagLow` are supported.

ci/update-version.js

@@ -89,7 +89,7 @@ function getPackageVersion(tag, branch) {
 
   // publish with a -devel suffix for EA and RC releases.
   if (tag.prerelease.length > 0) {
-    baseVersion += '-' + tag.prerelease.join('-');
+    baseVersion += '-' + tag.prerelease.join('.');
   }
 
   console.log(`Package version is "${baseVersion}"`);