When implementing event based servers, there are generally 2 patterns used, the reactor pattern and the proactor pattern.  The reactor pattern can be though of as being a synchronous version of the proactor pattern.  In a reactor pattern IO events are serviced by the thread in the IO handling loop.  In a proactor pattern the thread processing the IO event loop passes off the the IO event to another thread for processing.  HawtDispatch can support both styles of IO processing.

HawtDispatch uses a fixed sized thread pool sized to match the number of cores on your system.  Each thread in the pool runs an IO handling loop.  When a NIO event source is created, it gets assigned to one of the threads.  When network events occur the source causes callbacks to occur against on the dispatch queue targeted in the event source.  Typically that target dispatch queue is set to a serial queue which the application uses to handle the network protocol.  Since it’s a serial queue, the handling of the event can be done in a thread safe way.  The proactor pattern is being used since the serial queue can execute in any of the threads in the thread pool .

To use the reactor pattern, HawtDispatch supports ‘pinning’ the serial queue to a thread.  When a dispatch source is created on a pinned dispatch queue, then the event source gets registered against the same ‘pinned’ thread.  The benefit of the reactor pattern is that it avoids some of  cross thread synchronization needed for the proactor pattern and provides cheaper GCs.  The down side to the reactor pattern is that you may have to manage reblanacing network sources across all the available thread.  Lucky HawtDispatch does support moving pinned dispatch queues and sources to different threads.

A few months ago, I experienced similar results while hacking on ActiveMQ.  I noticed that passing messages from producer connections to consumer connections was dramatically faster if the producer and consumer were being serviced by the same thread.  I decided that the next version of the broker would to be need to be built using a thread management framework which could optimize itself so that those connections could collocate onto one thread if possible.

Then I saw the the libdispatch API (it forms the foundation of the Grand Central Dispatch technology in OS X) and fell in love with it’s simplicity and power.  I realized that implementation of that API could in theory provides the threading optimizations I was looking for.  So I started hacking on HawtDispatch, a Java/Scala clone of libdispatch.

The central concepts to libdispatch and hawtdispatch are global and serial queues.  Global queues are executors which execute tasks concurrently using a fixed size thread pool. Serial queues are executors without an assigned thread and which execute tasks in FIFO order.   When tasks added to a serial queue, the serial queue gets added to a global queue so that the serial queue can execute it’s tasks.  Multiple serial queues execute concurrently on the global queue.

The overhead of a serial queue is very small, it’s just a several counters and a couple of linked lists.  You can use them like a lightweight thread. Feel free to create thousands of them.  If you squint at it just right, they allows you to use erlang style actor thread model.

Now that you have an idea how HawtDispatch is used, lets get back to what kinds of optimizations it can do to help with cross thread contention.  HawtDispatch generally uses a concurrent linked list to enqueue a task in serial queue, but there are times when it can avoid that synchronization of the concurrent linked list.  For example, if the serial queue is currently executing in the current thread, then an enqueue can just add the task to a non synchronized linked list.  HawtDispatch also supports ‘pining’ a serial queue to one of the threads in the global queue’s thread pool.  This allows you to force serial queues to collocate onto one thread so that when they do need to communicate, there is no thread contention involved.

But you still run into cases where you need to move tons of events from one serial queue to another which is executing on a different thread.  For these cases, you use a custom event source.  It allows you to  coalesce a bunch of events generated on on thread as a single event delivered to the another queue.  HawtDispatch will aggregate custom events  into a thread local (to avoid contention) and once the current thread has drained all execution queues, it will deliver those custom events to their target queues.

This post is already getting kind of long, so I’ve have to do a follow up post on how all that interacts with network IO events.  But the general idea is, yes, keeping stuff on 1 core is fast, but it won’t scale once your CPU bound, so having a framework like can HawtDispatch help minimize cross thread contention while still providing the ability to scale up to multiple cores as load increases.

Progress Software is sponsoring an Apache ServiceMix, ActiveMQ, CXF & Camel Community Day on Thursday, December 10th, at the Hyatt Hotel in Burlingame. Join us at this free event and meet committers and founders of Apache ServiceMix, ActiveMQ, CXF and Camel that have successfully implemented enterprise application and deployed these projects in production.

Should fun to meet users/developers of these kick ass Apache based projects.  If you plan on going, make sure your register for the event.  It’ll be nice to meet everyone!

… both ActiveMQ and RabbitMQ are decent brokers that will serve their purpose well in normal conditions, but put to their extremes in terms of throughput, scalability and reliability, ActiveMQ currently outperforms RabbitMQ for messaging usage in Python.

And to support that use case, newlines after the frame terminator are a natural occurrence. But it might be easier to describe it as:

• A stomp frame may have zero or more newlines preceding it’s command verb.

And for those of you on Ubuntu, Dejan Bosanac has put together an excellent guide on how to build it on ubuntu.

Double buffering may allow us fix inconistencies in those index and gets us running faster..

Apache ActiveMQ, an open source provider of enterprise messaging services, recently released version 5.1 which includes improvements in stability and performance of the message broker product. This version also includes support for priority message ordering and a Microsoft Message Queue (MSMQ) to ActiveMQ Bridge with the new msmq transport component.

There are also improvements in the monitoring module of ActiveMQ container. A new DestinationSource class was added to access the available Queues or Topics or listen to Queues/Topics being created or deleted in the container. There is a new API to help end users view available destinations and query them to find JMS statistics such as active queue count, queue depth, number of messages etc.

Read More…

route add 224.0.0.0 netmask 240.0.0.0 dev eth0

or if you have an older version of linux like me:

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

A plethora of message queuing products exist in today’s EAI market, all aimed at providing solutions to the problem of application integration. Few can however lay as much claim to fame as ActiveMQ, an open source Message Broker and JMS/Enterprise Integration Patterns provider which is licensed, developed and distributed under the open source Apache emblem. ActiveMQ provides support for Cross Language Client and Protocols as well as a powerful messaging broker which is supported in Java, C, C++, C#, Ruby, Perl, Python, PHP.

Read the full article by Ade Ayonrinde.

And now thanks StompConnect, every JMS compliant MOM in existence (which is like almost all of them) can now be talked to using Stomp! This is a huge deal since the wire protocol for most JMS servers is proprietary and and getting a pure language client for any given mom was either difficult or impossible!

• Apache ActiveMQ has GRADUATED out of the incubator and is an official Apache project!
• The ActiveMQ website has moved to http://activemq.apache.org and has received a new face lift thanks to yours truly.
• ActiveMQ 5.0 development is making huge progress and Rob has put up an excellent post outlining the upcoming features in the next version of ActiveMQ.

If you want to get a preview of those features, get the latest from the ActiveMQ trunk and kick it around. Feedback would be appreciated.

• Guaranteed delivery
• Once and only once deliveryGuaranteed
• order of deliveryServer
• push and client pullFunny

thing is that most of all that is available today with ActiveMQ! And to get really great performance, use ActiveMQ with Jetty! ActiveMQ comes with a simple little JavaScript API that allows you to access the ActiveMQ message bus using Comet style http polling. And ActiveMQ provides you all the above guarantees like all good message brokers.

Other news on this front is all the Ajax tool kits are starting to look at being able to inter-operate. At a minimum, a page with multiple tool kits will need to share it’s connections back to the server. So they will need to share an API to broker requests back to the server. I’ll keep an eye out for this API because it sounds like something that could be easily tied into ActiveMQ.

#1: I think it’s unfortunate that the “AMQP” sounds too much like it has something to do with ActiveMQ which most folks abbreviate to AMQ, as in, “Have you downloaded AMQ 4?”. This along with the fact that AMQ and AMQP are both related technologies, the first is a mom provider and the second is a wire protocol for mom providers.

#2: This is nice spec. I like the client specified “binding” concepts introduced in this spec. Perhaps these concepts can be introduced in higher level APIs like JMS one day.

#3: It seems that one of the goals of the spec is for vendors to be able to interoperate with each other. I have got a feeling that specs like WS-Notification or STOMP have a better chance at accomplishing this. Binary wire protocols are hard to implement and I doubt there will be many implementations of it. If a good open source reference implementation becomes available then this would become more plausible.

#4: It does not go into details of how the content of the messages should be encoded. This effectively means that JMS implementations will not be able to interoperate using AMQP since different implementations would encodes the content of a JMS Message differently. Even for simple things like TextMessage, would they use UTF-8 or ASCII? And it gets even more complex for messages lke StreamMessage and MapMessage.

#5: I may be wrong, but it seems like messages are always sent asynchronously. In some cases the JMS spec requires messages to be sent synchronously. Perhaps transactions can be used to simulate a synchronous send, but wouldn’t that add substantial of overhead?

In a way, the long term store (JDBC in most cases) is being used like an index into the Journal files. This increases the performance of the journal since the amount of data that needs to be stored in long term store is drastically reduced and generally of a small size which works better with JDBC batch operations. Also messages do not need to be batched up in memory (for batch insertion into the DB) thus reducing the memory impact of the message store.

The funny part is that at some time last week, somehow a wager got started about who could build the fastest message store implementation that could stay under 64 megs of memory usage when a queue was loaded up with 10,000,00 1k messages. Kaha at the current time keeps it’s indexes fully in memory. So Rob started looking into a way to optimize the down so that they could fit in 64 megs (Rob is the optimization King BTW). I went down the route of we need to only load parts of the index into memory. I even shared my algorithm concept with him as long as he did not use it, LOL. In the end we realized, it was not going to be a weekend deal to implement this stuff and it would be best to work on a single solution together.

Kaha, IS a nice API and is much more general purpose than the MessageStore APIs. Some of the problems that Kaha currently has is that it does not guarantee constancy of the indexes and it does not support transactional operations. Those are 2 things that the journal can do today, and which Kaha could do if we modified it’s DataManager so that it journaled operations instead of just storing data items. So I’m going to try to integrate many of the Journal concepts into the DataManager so that:

• The data file acts as a redo log that is ‘replayed’ on startup to bring the indexes to a consistent state
• Use async batch writes for increased throughput: micro benchmarks showed that the journal can write at about 21 megs/s while the current DataManager maxes out at 8 megs/s

Other things to consider is that since the interfaces to the Kaha APIs are based on the List and Map interfaces, there is no easy way to:

• switch between doing async and sync operations against the data files. Currently Kaha has a force() method on the store that does syncs up any pending write but this is not optimal when using async batched writes (you end up syncing on a subsequent write).
• associate a transaction with a operation against a list or a map

An idea I’ve been floating in my head is the ability to have multiple proxies to a single physical container. Each proxy could be enlisted in a different transaction or it’s flag to do sync vs. async actions changed.

As you might be able to tell by now, I’m on the Kaha crack now… bless Rob.

It’s obvious that we need to swap messages to disk when a queue needs to hold more messages than it could hold in RAM. We sometimes also call that spooling messages to disk. The issues that make this hard to implement are:

• Writing a message to disk slows you down a little, avoid it if possible. Sometimes you have no choice if the message was marked a persistent.
• Sometimes we may need to swap out even non-persistent messages.
• Avoid chucking a message out of ram if possible since loading it back from disk is REALLY slow.
• When a consumer is ready to consume a message, that message should already be in memory, waiting for it to load from disk will lead to consumer starvation.
• Even keeping lists of message references to where messages are on disk can use up too much memory. 10,000,000 disk locations in a linked list where every node in the list used only 100 bytes would still chew up about 100 megs of memory.

ActiveMQ 4.0 takes a simple approach and when sending persistent message to a Queue, it uses a MessageReference when moving a message though the Broker message dispatch process. A process that could take a while for a message to go from producer to a consumer and finally message acknowledgement. The MessageReference starts out being direct, in that it hold a reference to the message to keep it in RAM, but if the reference count drops below 1, then the direct reference is dropped. The reference count is allowed to drop to 0 when the message is just sitting in the Queue’s message list or in a consumer’s pending list. The reference count is > 0 while it’s being dispatched to a consumer. The MessageReference knows how to reload a Message from the peristence store when it’s reference count goes up above 0.

This is a quick and dirty fix and it works, but it obviously does not fix all the issues outlined initial. The shortcomings of the current solution is that:

• it is only implemented for Queues
• consumer starvation problem can exist since it does not persisted load messages asynchronously
• it keeps a list of MessageReference objects which can still exhaust JVM memory

In ActiveMQ 4.0, we have hacked in some initial support for loading up a queue with a huge number of messages without blowing up the memory usage of the JVM, but it’s a bit hacky and it may fail work right if a consumer comes back online and the consumer recovery process kicks in.

All in all 4.0 is a solid broker with a ton of new and exciting features, but personally, I would like to focus on getting 4.1 to be the broker that can handle Gigantic Queues and Topics. I’ll post some more messages on this topic in the next few hours as I recap what I’ve discovered in the last few weeks.