I am testing ActiveMQ 5.9.0 with a replicated DB level .
Starting standalone ActiveMQ with local LevelDB storage, each call producer.send(message)takes about 1 ms. With my replicated installation with 3 zoos and 3 active brokers, producer.send(message)it takes just over 5 seconds to get back! This happens even at sync="local_mem"c <replicatedLevelDB ... >. It is always just above 5 seconds, so there seems to be a weird wait / timeout.
Does this ring a bell?
It doesn't matter if I install brokerurl on failover:(<all three brokers>)or just tcp://brokerXwhere brokerX is in the replicated LevelDB setting. The brokerX ui (hawtio) web browser does not show any noticeable delay in sending. If I go to tcp://brokerYwhere the broker is otherwise an identical broker with <persistenceAdapter ...>set to <levelDB...>instead <replicatedLevelDB...>, we will go down 1 ms for sending.
Change zookeeper tickTime etc. irrelevant.
Debug log below. As you can see, 5 seconds between "sent to the queue", but the zookeeper ping code is fast.
2014-02-19 10:45:34,719 | DEBUG | Handling request for path /jolokia | io.hawt.web.AuthenticationFilter | qtp1217711018-227
2014-02-19 10:45:34,724 | DEBUG | localhost Message ID:<hostname>-57776-1392803129562-0:0:1:1:2 sent to queue://IO_stab_test_Q | org.apache.activemq.broker.region.Queue | ActiveMQ Transport: tcp:///<ip address>:54727@61616
2014-02-19 10:45:34,725 | DEBUG | IO_stab_test_Q toPageIn: 1, Inflight: 1, pagedInMessages.size 1, enqueueCount: 27, dequeueCount: 25 | org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[localhost] Task-20
2014-02-19 10:45:34,731 | DEBUG | Handling request for path /jolokia | io.hawt.web.AuthenticationFilter | qtp1217711018-222
2014-02-19 10:45:34,735 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:34,867 | DEBUG | Handling request for path /jolokia | io.hawt.web.AuthenticationFilter | qtp1217711018-222
2014-02-19 10:45:35,403 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:35,634 | DEBUG | Handling request for path /jolokia | io.hawt.web.AuthenticationFilter | qtp1217711018-227
2014-02-19 10:45:36,071 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:36,740 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:37,410 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:38,088 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 8ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:38,623 | DEBUG | Handling request for path /jolokia | io.hawt.web.AuthenticationFilter | qtp1217711018-222
2014-02-19 10:45:38,750 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:39,420 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)
2014-02-19 10:45:39,735 | DEBUG | localhost Message ID:<hostname>-57776-1392803129562-0:0:1:1:3 sent to queue://IO_stab_test_Q | org.apache.activemq.broker.region.Queue | ActiveMQ Transport: tcp:///<ip address>:54727@61616
2014-02-19 10:45:39,737 | DEBUG | IO_stab_test_Q toPageIn: 1, Inflight: 2, pagedInMessages.size 2, enqueueCount: 28, dequeueCount: 25 | org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[localhost] Task-24
2014-02-19 10:45:40,090 | DEBUG | Got ping response for sessionid: 0x244457fceb80003 after 0ms | org.apache.zookeeper.ClientCnxn | main-SendThread(<hostname>:2181)