Skip to content

Producer-related configurations

Version: 1.0.0

The producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server as well as a background I/O thread that is responsible for turning these records into requests and transmitting them to the cluster.

Each cloud connection instantiates a single KafkaProducer, which is shared among the CloudPublishers.

Attention

The producer implemented in this layer is not transactional nor idempotent.

Acknowledgements

The acks config controls the criteria under which requests are considered complete. The ALL setting will result in blocking on the full commit of the record, the slowest but most durable setting.

When this property is set to ALL and the oldest in-flight message is deleted from the data service, the acknowledgement might never be received. This causes the producer to never send new messages. Hence, when using this setting it is recommended tweaking the max.inflight.requests.per.connection config, the QoS of the produced message (Publisher configurations), or the store settings (DataService layer).

Default configurations

The instantiated KafkaProducer is initialized with some default configurations that are not definable by the user:

  • retries=0: the producer will not try to resend any record whose send fails with a potentially transient error.
  • linger.ms=0: the producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. With this setting to 0, the client will not wait any amount of delay for accumulating larger batches of records. The accumulated records are always sent immediately.
  • request.timeout.ms=1000: maximum amount of time the client will wait for the response of a request.
  • delivery.timeout.ms=1000: upper bound on the time to report success or failure after send. This limits the total time that a record will be delayed prior to sending and the time to await acknowledgement from the broker.
  • key.serializer=org.apache.kafka.common.serialization.StringSerializer: the producer is able to process only records with keys of type java.lang.String.
  • value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer: the producer is able to process only records with values of type byte[].

All the other configs are defaulted according to the Kafka Producer configurations documentation.