atom feed9 messages in org.apache.incubator.kafka-usersRE: Only one broker has partitions
FromSent OnAttachments
Bateman, MattFeb 16, 2012 2:47 pm 
Jun RaoFeb 16, 2012 6:07 pm 
Bateman, MattFeb 17, 2012 11:05 am 
Jun RaoFeb 17, 2012 11:18 am 
Bateman, MattFeb 17, 2012 1:39 pm 
Jun RaoFeb 17, 2012 2:22 pm 
Bateman, MattFeb 17, 2012 2:57 pm 
Jun RaoFeb 17, 2012 3:02 pm 
Bateman, MattFeb 17, 2012 4:55 pm 
Subject:RE: Only one broker has partitions
From:Bateman, Matt (maba@ebay.com)
Date:Feb 17, 2012 11:05:13 am
List:org.apache.incubator.kafka-users

We're using a single topic ("test-topic") in our QA setup. Originally we were
using only one broker (id 0) in or testing environment. Recently we added two
more (id 2 and 3) for a total of 3 in our cluster. ZooKeeper shows that the
topic and partitions are registered by broker 0 as expected. They do not show
for brokers 2 and 3. I'm guessing that if a broker is not registered with
ZooKeeper when a producer sends the first message for a topic, it will never
process messages for that topic?

As a test, I sent messages into the cluster for a new topic. The producer logs
show

[2012-02-17 10:31:12,698] DEBUG Getting the number of broker partitions
registered for topic: cluster-test (kafka.producer.Producer) [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered under
topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo) [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test with available
brokers in the cluster with default number of partitions = 1
(kafka.producer.ZKBrokerPartitionInfo) [2012-02-17 10:31:12,705] DEBUG Adding following broker id, partition id for NEW
topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
(kafka.producer.ZKBrokerPartitionInfo) [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)

This seems logical, all brokers are mentioned. Then I see the following (each
broker is configured to use 20 partitions):

[2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
(kafka.producer.ProducerPool) [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of topics changed at
/brokers/topics Updated topics -> [topic1, topic2, cluster-test, topic3,
test-topic, topic4] (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of topics:
Set(topic1, topic3, topic2, test-topic, topic4)
(kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list of topics:
Set(topic4, test-topic, topic2, topic3, cluster-test, topic1)
(kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly registered
topics: Set(cluster-test)
(kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently registered list
of brokers for topic: cluster-test are Buffer(2)
(kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered list of
brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
(kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of broker partitions
for topic: cluster-test are TreeSet(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7,
2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
(kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener) [2012-02-17 10:31:13,841] DEBUG Getting the number of broker partitions
registered for topic: cluster-test (kafka.producer.Producer) [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10,
2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
(kafka.producer.Producer)

This seems strange, shouldn't each broker host each partition for the new
"cluster-topic"?

I then see the producer debug messages showing messages sent to broker 2.

What am I missing here?

Thanks,

Matt

-----Original Message----- From: Jun Rao [mailto:jun@gmail.com] Sent: Thursday, February 16, 2012 6:08 PM To: kafk@incubator.apache.org Subject: Re: Only one broker has partitions

Try enabling debug level logging in Producer class. It will show which broker
the partitioner selects.

Thanks,

On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <maba@ebay.com> wrote:

Hi All,

I feel like this was discussed at some point but couldn't find it in the docs or the mailing list archives, hopefully this isn't redundant.

We are running 3 version 0.7 brokers using an ensemble of 3 ZooKeeper nodes. We use a ZooKeeper connect string in the producers and consumers. ZooKeeper is enabled on each broker. We use the default partitioner in the producer supplying it an integer. Each broker has a unique Id.

However, only one broker has any log files. The log from that machine shows it registering the topics in ZooKeeper. The other brokers don't have any log files.

Have a I missed a basic configuration setting or misunderstood something operational? The only slight mismatch I see is we run ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.

Thanks,