Kafka Consumers and Consumer Groups

../../_images/KafkaConsumerConcepts01.PNG

  • Kafka consumers are typically part of a consumer group.

  • We can start off with just one consumer in a consumer group, and expand that to multiple consumers in the same consumer group. As long as there are as many partitions for the topic.

  • Once you have more consumers than partitions for a particular topic, the extra consumers would be idle and not receiving any messages.

../../_images/KafkaConsumerConcepts02.PNG

  • The way to scale in such a situation, is to create another consumer group and spreading the consumers between the consumer groups.

  • If the new consumer group (G2) has only one consumer, this consumer will get all the messages for the topic independent of what consumers in the other consumer group G1 is doing.

  • Kafka typically handles the balancing of partition processing between consumers in a consumer group and storing the consumer’s current partition offsets.

The number of Partitions

  • The number of partitions for a topic is an important parameter that determines how many consumers can be yoked to the topic.

  • It determines how a topic is scaled and how messages will be balanced across the entire cluster as brokers are added.

  • This number is specified when a topic is created. This number can be increased, but never decreased, hence care must be taken when creating a topic.

  • Usually, the number of partitions for a topic will be equal to or a multiple of the number of brokers in the cluster.

  • This allows the partitions to be evenly distributed to the brokers, evenly distributing the message load.