Consumer Group Internals: Rebalancing, Rebalancing.... -- Jason Gustafson & Onur Karaman, 8/23/16
Why Stack Overflow recommends it
Stack Overflow users link it when explaining Kafka consumer group rebalancing and coordination, since it comes from two of the protocol's main developers.
Recommended by 2 people across 2 different months, 2018 to 2021.
What it is
A conference talk from LinkedIn's Stream Processing Meetup in which Kafka developers Jason Gustafson (Confluent) and Onur Karaman (LinkedIn) explain how Kafka consumer groups work. It covers the coordination protocol introduced in Kafka 0.9, how the new consumer client uses it, how rebalancing works, and how it scales.
What people said
Recommends Mickael Maison · 2019-07 · Stack Overflow · on “How exactly does the newest version of Kafka store/persist the subscriptions from the consumers?”
The documentation has a large section on Consumer model: [LINK] Details of the consumer group protocol can be found in these 2 talks: [this video] [LINK] Topic subscriptions are not persisted on the brokers but instead consumers use a rebalance protocol to share their subscriptions and assign partitions between each other. Records are sent from the brokers "raw" and consumers are responsible for deserializing the data, hence why key.deserializer and value.deserializer are mandatory settings [LINK] .
Recommends jessica · 2021-12 · Stack Overflow · on “kafka + what chould be the root cause for Consumer group is rebalancing”
Kafka machines are installed as part of hortonworks packages , kafka version is 0.1X We run the deeg_data applications, consuming data from kafka topics On last days we saw that our application – deeg_data are failed and we start to find the root cause On kafka cluster we see the following behavior /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --group deeg_data --describe --bootstrap-server kafka1:6667 To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files> where num_of_file > 0 GC log rotation is turned off Consumer group ‘deeg_data’ is rebalancing from kafka side kafka cluster is healthy and all topics are balanced and
Cites gabrielgiussi · 2018-05 · Stack Overflow · on “Kafka Consumer error: Marking coordinator dead”
With session.timeout.ms you only control the timeouts due to heartbeats, this means that has passed session.timeout.ms milliseconds since the last heartbeat and the cluster declares you as a dead node and triggers a rebalance. Before KIP-62 [LINK] the heartbeat was sent within the poll but now is moved to a specific background thread to avoid being evicted from the cluster if you were taking more time than session.timeout.ms to call another poll() . Separating the heartbeat to a specific thread decouples the processing from telling the cluster that you are up and running, but this introduced the risk of "livelock" situations in which the process is alive, but is not making progress, so besid