Apollo features:

• Stomp 1.0 Protocol Support
• Stomp 1.1 Protocol Support
• Topics and Queues
• Queue Browsers
• Durable Subscriptions on Topics
• Persistent/Reliable Messaging
• Message Expiration
• Message Swapping
• Message Selectors
• JAAS Authentication
• ACL based Authorization
• SSL/TLS Support and Certificate based Authentication
• REST Management API

Yes, it supports JMS!  Just download Apollo, start it up using the getting started guide and then check out the distribution’s examples directory.  Oh yeah, Apollo has great documentation!

I’ve re-run the STOMP benchmarks I covered in a previous post against the 1.0 release.  The benchmark uses the STOMP protocol to build an apples to apples performance comparison between all the major STOMP servers.  I’ve also built a new JMS benchmark which uses the JMS API build the same kind of performance comparisons.  Both benchmarks are open source and available to be tried out on your choice of hardware.  Or better yet, follow the simple instructions found in each project’s readme to run the benchmarks on EC2:

• https://github.com/chirino/stomp-benchmark
• https://github.com/chirino/jms-benchmark

For those of you wondering why the ActiveMQ project create new message broker as a subproject, it’s because we wanted to address the shift in the processor market to multi core for the next major release.  ActiveMQ currently employs complex thread locking which starts to become bottleneck as you increase the core count and you don’t end up fully exploiting the potential on large core count machines.  By developing the solution as a subproject, it’s easier to explore the best options without impacting current main line development of ActiveMQ 5.x.

Now that the Apollo sub project has proven itself, I think it’s time to start integrating it into ActiveMQ.  Ideally, ActiveMQ 6 switches Apollo’s messaging engine and adds/ports all the existing features we’ve come to expect out of ActiveMQ like networks of brokers and priority queues.  But we should really be talking about this on the Apache mailing lists.. join me there!

• Apache ActiveMQ 5.5.1
• Apache ActiveMQ Apollo 1.0 Beta 6 (an ActiveMQ Subproject)
• HornetQ 2.2.5
• RabbitMQ 2.7.0

STOMP is an asynchronous messaging protocol with design roots are based on HTTP protocol.  It’s simplicity has made the protocol tremendously  popular since it reduces the complexity of integrating different platforms and languages.  There are a multitude of client libraries for your language of choice to interface with STOMP servers.

The benchmark focuses on finding the maximum producer and consumer throughput for a variety of messaging scenarios.  For example, it benchmarks a combination of all the following scenario options:

• Queues or Topics
• 1, 5, or 10 Producers
• 1, 5, or  10 Consumers
• 1, 5, or 10 Destinations
• 20 byte, 1k, or 256k message bodys

The benchmark warms up each scenario for 3 seconds.  Then for 15 seconds it samples the total number of messages that were  produced/consumed every second.  Finally, the destination gets drained of any messages before benchmarking the next scenario.  The benchmark generates a little HTML report with a graph for each scenario run where it displays messaging rate for each of the servers over the 15 second  sampling interval.

I’ve run the benchmarks on a couple if different machines and I’ve posted the results at: http://hiramchirino.com/stomp-benchmark/
Note: the graphs can take a while to load as they are being generated on the client side using the excellent flot javascript library.

Since anyone can get access to an EC2 instance to reproduce those results, the rest of this article will focus on the results of the obtained on the EC2 High-CPU Extra Large Instance.  If you want to reproduce, just spin up a new Amazon Linux 64 bit AMI and then run the following commands in it:

sudo yum install -y screen
curl https://nodeload.github.com/chirino/stomp-benchmark/tarball/master | tar -zxv
mv chirino-stomp-benchmark-* stomp-benchmark
screen ./stomp-benchmark/bin/benchmark-all

Note: RabbitMQ 2.7.0 sometimes dies midway through the benchmark.  It seems RabbitMQ does not enforce very strict flow control and you can get into situations where it runs out of memory if you place too much load on it.  It seems that crash becomes more likely as you increase the core speed of the cpu or reduce the amount of physical memory on the box.  Luckily, the RabbitMQ folks are aware of the issue and hopefully will fix it by the next release.

The ‘Throughput to an Unsubscribed Topic’ benchmarking scenario is interesting to just get a idea/base line what the fastest possible rate a producer can send to server.  Since there are not attached consumers, the broker should be doing very little work since it’s just dropping all the messages that get sent to it.

The Queue Load/Unload scenarios a very important to look at if your application uses queues.  You often times run into situations where messages start accumulating in a queue with either no consumers or with not enough consumers to keep up with the producer load.  This benchmark first runs a producer for 30 seconds enqueuing non-persistant messages and then runs a producer enqueuing persistant messages  for 30 seconds.  Finally, it runs a consumer to dequeue the messages for 30 seconds.  An interesting observation in this scenario is that Apollo was the only sever which could dequeue at around the same maximum enqueue rates which is important if you ever want your consumers to catch up with fast producers.

The Fan In/Out Load Scenarios help you look at cases where you have either multiple producers or multiple consumers running against a single destination.  It helps you see how performance will be affects as you scale up the producers and consumers.  You should follow the “10 Producers” columns and “10 Consumers” rows to really get a sense of which servers do well as the you increase the number of clients on a single destination.

The Partitioned Load Scenarios look at how well the server scales as you start to increase load on multiple destinations at different message sizes.

I’ve tried to make the benchmark as fair as possible to all the contenders, all the source code to the benchmark is available on github.  Please open an issue or send me pull request if you think of ways to improve it!

The BIGGEST change in the broker architecture was a switch to a non-blocking threading model for message processing using the HawtDispatch library.  This meant changing many of the APIs to using a continuation passing style so that caller never blocks on a request.  The upside of the new architecture is that the most complex and brittle areas of multi-threaded code in ActiveMQ 5.x  have now been tremendously simplified.  Those areas are using actor style coarse grained synchronization thanks to HawtDispatch.

It is impressive how well Apollo performs. Using a little benchmarking tool, I compared the performance of Apollo and three other STOMP server implementations.  The benchmark tests different combinations of consumers, destinations, producers, persistence options, message sizes for a total of 74 common usage scenarios.  I ran the benchmarks on two different boxes.  If you want to peek at the complete benchmark reports, see Ubuntu 4 Core report and OS X 8 Core report.

Apollo ends up doing really well in comparison to other servers in most cases but in some cases it’s just mind blowing.  For example:

The  scenario above is the case where you have 10 producer and 10 consumers on a single topic moving non persistent STOMP messages with a 20 byte payload.   The graph shows that Apollo can easily sustain a total consumer processing rate of 1.2 Million messages per second while the closest contender could only reach about 105,000 thousand messages per second.

RestyGWT is a GWT generator for REST services and JSON encoded data transfer objects.  What I really like about it is that it allows you to write rich JAX-RS based services and access those services from at GWT client reusing all the same DTO objects that you use on the server side.

In other words, it just as easy to use as GWT-RPC except it’s using simple RESTful URLs and JSON encoding of data when it access server side resources.  This has the nice benefit that you can implement the server side in something other than JAVA easily.

Further information see:

• Website
• Community
• Documentation
• Download

When implementing event based servers, there are generally 2 patterns used, the reactor pattern and the proactor pattern.  The reactor pattern can be though of as being a synchronous version of the proactor pattern.  In a reactor pattern IO events are serviced by the thread in the IO handling loop.  In a proactor pattern the thread processing the IO event loop passes off the the IO event to another thread for processing.  HawtDispatch can support both styles of IO processing.

HawtDispatch uses a fixed sized thread pool sized to match the number of cores on your system.  Each thread in the pool runs an IO handling loop.  When a NIO event source is created, it gets assigned to one of the threads.  When network events occur the source causes callbacks to occur against on the dispatch queue targeted in the event source.  Typically that target dispatch queue is set to a serial queue which the application uses to handle the network protocol.  Since it’s a serial queue, the handling of the event can be done in a thread safe way.  The proactor pattern is being used since the serial queue can execute in any of the threads in the thread pool .

To use the reactor pattern, HawtDispatch supports ‘pinning’ the serial queue to a thread.  When a dispatch source is created on a pinned dispatch queue, then the event source gets registered against the same ‘pinned’ thread.  The benefit of the reactor pattern is that it avoids some of  cross thread synchronization needed for the proactor pattern and provides cheaper GCs.  The down side to the reactor pattern is that you may have to manage reblanacing network sources across all the available thread.  Lucky HawtDispatch does support moving pinned dispatch queues and sources to different threads.

A few months ago, I experienced similar results while hacking on ActiveMQ.  I noticed that passing messages from producer connections to consumer connections was dramatically faster if the producer and consumer were being serviced by the same thread.  I decided that the next version of the broker would to be need to be built using a thread management framework which could optimize itself so that those connections could collocate onto one thread if possible.

Then I saw the the libdispatch API (it forms the foundation of the Grand Central Dispatch technology in OS X) and fell in love with it’s simplicity and power.  I realized that implementation of that API could in theory provides the threading optimizations I was looking for.  So I started hacking on HawtDispatch, a Java/Scala clone of libdispatch.

The central concepts to libdispatch and hawtdispatch are global and serial queues.  Global queues are executors which execute tasks concurrently using a fixed size thread pool. Serial queues are executors without an assigned thread and which execute tasks in FIFO order.   When tasks added to a serial queue, the serial queue gets added to a global queue so that the serial queue can execute it’s tasks.  Multiple serial queues execute concurrently on the global queue.

The overhead of a serial queue is very small, it’s just a several counters and a couple of linked lists.  You can use them like a lightweight thread. Feel free to create thousands of them.  If you squint at it just right, they allows you to use erlang style actor thread model.

Now that you have an idea how HawtDispatch is used, lets get back to what kinds of optimizations it can do to help with cross thread contention.  HawtDispatch generally uses a concurrent linked list to enqueue a task in serial queue, but there are times when it can avoid that synchronization of the concurrent linked list.  For example, if the serial queue is currently executing in the current thread, then an enqueue can just add the task to a non synchronized linked list.  HawtDispatch also supports ‘pining’ a serial queue to one of the threads in the global queue’s thread pool.  This allows you to force serial queues to collocate onto one thread so that when they do need to communicate, there is no thread contention involved.

But you still run into cases where you need to move tons of events from one serial queue to another which is executing on a different thread.  For these cases, you use a custom event source.  It allows you to  coalesce a bunch of events generated on on thread as a single event delivered to the another queue.  HawtDispatch will aggregate custom events  into a thread local (to avoid contention) and once the current thread has drained all execution queues, it will deliver those custom events to their target queues.

This post is already getting kind of long, so I’ve have to do a follow up post on how all that interacts with network IO events.  But the general idea is, yes, keeping stuff on 1 core is fast, but it won’t scale once your CPU bound, so having a framework like can HawtDispatch help minimize cross thread contention while still providing the ability to scale up to multiple cores as load increases.

… a small utility for executing Java programs which are stored as artifacts like jars or bundles in a Maven repository. MOP automatically deals with the following for you transitive dependencies downloading artifacts from remote repositories and caching them locally setting up your class path

It’s a wonderfully small download that weighs in at only 3.2 megs.  Why don’t you try it out today?

Progress Software is sponsoring an Apache ServiceMix, ActiveMQ, CXF & Camel Community Day on Thursday, December 10th, at the Hyatt Hotel in Burlingame. Join us at this free event and meet committers and founders of Apache ServiceMix, ActiveMQ, CXF and Camel that have successfully implemented enterprise application and deployed these projects in production.

Should fun to meet users/developers of these kick ass Apache based projects.  If you plan on going, make sure your register for the event.  It’ll be nice to meet everyone!

… both ActiveMQ and RabbitMQ are decent brokers that will serve their purpose well in normal conditions, but put to their extremes in terms of throughput, scalability and reliability, ActiveMQ currently outperforms RabbitMQ for messaging usage in Python.

Counter that with latest AJAX rage, RESTful services with a JSON or XML representation of the data.  The nice thing about using JSON with REST services is that clients and servers can easily be built it any language.  Wouldn’t it be nice if you use JSON RESTful services from GWT with the ease of GWT RPC?

That’s exactly the reason I started hacking on RestyGWT.  It uses a very similar development model as GWT RPC.  You create a service interface who’s implementation is code generated via Deferred Binding when your GWT module is built.  The developer maps the interface methods to RESTful HTTP requests using the excelent JAX-RS annotations.  Just like in GWT RPC, the interface must be ‘asynchronous’.  You must use callback method arguments to get the request results.  But unlike GWT RPC, the data sent and received will be JSON encoded.  RestyGWT will automatically code generate JSON encoders and decoders for any classes sent or received from the rest service.

For more details see the Git Hub home page at: http://github.com/chirino/resty-gwt

As of this point, RestyGWT only implements the most basic JAX-RS annotations and provides a very simple JSON encoding scheme.  Despite that, the project is already extremely useful.  I would love here what folks think about it.

It also supports and even simpler interface remoting model which does not mandate the throwing of RemoteException or extending the RemoteObject interfaces. It even allows remoting classes which don’t implement any interfaces via ASM magic.

Documentation still needs a little work, but the project is now ASL 2.0 Licensed and looking for contributions! Please join the mailing lists to get involved.

And to support that use case, newlines after the frame terminator are a natural occurrence. But it might be easier to describe it as:

• A stomp frame may have zero or more newlines preceding it’s command verb.

So you might be asking yourself, what’s the secret sauce? Well before I get into that, let me first explain the class model that our proto compiler generates.

For every message definition in the ‘.proto’ file, the compiler will generate 3 classes:

• the message interface: is implemented by the bean and buffer classes. It has all the ‘getters’.
• the bean class: has all the ‘setters’ and ‘merge’ methods
• the buffer class: has all the encoding and decoding methods. It does not allow mutation.

The message interface also defines the freeze(), frozen(), and copy() methods which allow you to make an instance immutable, check to see if an instance is immutable, and create a mutable copy. Buffer classes are alway immutable. Bean class can transition to being immutable via freeze(). freeze() naturally returns a buffer object. copy() naturally returns a bean object.

This bean model gives substantial flexibility. Besides making it easy to transition from immutable to mutable and back, the message interface lets you implement business methods that operate against either type of instance. You could use the bean class purely in a builder style to always generate a buffer instance, or you could just use them like traditional java bean objects.

Once a bean instance is frozen, any attempts to modify the instance will throw assertion errors if assertions are enabled in your JVM. So the CPU cost of validating program correctness can can be disabled at run time.

Finally, the most important feature of the buffer class is that it holds on to either the byte array that it was created from or the frozen bean that created it, and sometimes both, after it builds one from the other. This has several implications. Firstly, once a buffer is encoded to a byte[], subsequent encoding passes are free. This is also true when a buffer is decoded, as the next encoding is free since it still retains the original encoding of the message. And the other benefit that this provides which the benchmark highlighted, is that deferred decoding is possible. A newly created buffer class will not decode the data until a field is accessed. This also true of the nested messages that are encoded in a buffer. While the outer message may get decoded, the nested message will not be decoded until it’s fields are accessed.

I’ve had been researching the protobuf encoding format for a while and really liked it. But I did not really like the Java implementation that Google had published. It was kinda clunky to use and I saw several optimizations that could be used that were missing. Optimizations that could create huge performance wins when applied to the usage patterns of an enterprise messaging system like Apache ActiveMQ. So I created a new protobuf implementation in the ActiveMQ project.

Naturally, I was curious to see how the activemq protobuf implementation stacked up against similar technologies. So I grabbed the V1 benchmark source code and added our implementation to it. If you want to do the same, apply this patch.

Once I ran the benchmark and I was very pleased with the results. I’m including the performance graphs of our impl and standard protobuf and thrift for comparison.

As it turns out, our implementation looks awesome in the benchmark! How about that decoding speed!

It’s getting late here.. so I’ll have do a follow up post explaining how come we did so much better.

And for those of you on Ubuntu, Dejan Bosanac has put together an excellent guide on how to build it on ubuntu.

It’s called Jansi and it provides support for using ANSI escape sequences in your Java console applications on Windows.

With ANSI escape sequences, you can fully control the the cursor positioning and the foreground and background color of the console text output. Here is quick example of what’s posssible:

The basic problem the plugin is trying to solve is that it is possible that central repositories get hacked and the artifacts/dependencies of our builds get replaced with
malicious versions. Right now we have no way to easily detect that
and we could potential create a release build of a project which
bundles one of those malicious dependencies. In practice this rarely
occurs, but it’s not out of the realm of possibilities.

Basically the plugin supports generating a checksum.txt file that is included as part of the project build. This file holds all the checksums for the dependencies (including the dependencies’ pom checksum). Generating/updating is induced via the use of a maven profile. This is only done when dependencies get updated.

In a normal build the plugin just validates the checksums of the downloaded dependencies against those stored in the checksum.txt file.

I wish I could move up the validation of the dependencies from their current maven life cycle locations, but it seems you can’t get the list of dependencies it gets moved up any more. Any maven mojo hackers have any work arounds for that?

First off, I love the Maven Repository Security Proposal. I think that the ‘Specified Checksums’ idea is awesome. I think it needs to be made so easy to use that folks always use it. Right now it’s a little ugly because it makes the dependency declaration much more verbose. Plus it does not seem to cover transitive dependencies that are being used during the build, and I think that those checksums NEED to be included too.

I think that what would be better is if maven provided the tools to update the checksum information in the pom.

Lets say that a build for a module is setup in some strict mode where only artifacts with known checksums are allowed. If the pom is updated to add a new dependency, I think there should be some maven command which automatically adds the checksum for the new dependency (and transitive dependencies). Artifacts that are signed with a trusted key get added without prompting, and a confirmation prompt would be given for artifacts that are not GPG trusted.

So the question is why go through all that trouble? So that folks get a trusted source distribution (out of SCM or a signed tar ball), can do a build and have a high level of guarantee that the dependencies that are being used in the source build match what was intended by the developers of the source distribution. Furthermore, it will not matter if the transitive dependencies are signed and have keys in the end user’s keyring since all the checksums are include in the build.

Now, since there could be lots of dependencies in a build, due to the use of build plugins and transitive dependencies, it might be worth storing the checksum data in a file external to pom.xml, or at least in a different xml section from the dependencies declaration.

Things to think about: Having SNAPSHOT dependencies in the build could complicate things, as the build would be tied to a particular SNAPSHOT/checksum, but maybe that’s a good thing.

Oh and it’s an Apache Project now. Yay! Seems like the project website is still not fully setup since they are migrating from SourceForge to Apache, be here’s a link to the source tree.

Double buffering may allow us fix inconistencies in those index and gets us running faster..

Apache ActiveMQ, an open source provider of enterprise messaging services, recently released version 5.1 which includes improvements in stability and performance of the message broker product. This version also includes support for priority message ordering and a Microsoft Message Queue (MSMQ) to ActiveMQ Bridge with the new msmq transport component.

There are also improvements in the monitoring module of ActiveMQ container. A new DestinationSource class was added to access the available Queues or Topics or listen to Queues/Topics being created or deleted in the container. There is a new API to help end users view available destinations and query them to find JMS statistics such as active queue count, queue depth, number of messages etc.

Read More…

Even if you have not had seen any issues with your 5.0.0 installation, I’d highly recommend you upgrade to 5.1.0 to avoid running into some of the bugs that have been addressed in the release.

route add 224.0.0.0 netmask 240.0.0.0 dev eth0

or if you have an older version of linux like me:

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

A plethora of message queuing products exist in today’s EAI market, all aimed at providing solutions to the problem of application integration. Few can however lay as much claim to fame as ActiveMQ, an open source Message Broker and JMS/Enterprise Integration Patterns provider which is licensed, developed and distributed under the open source Apache emblem. ActiveMQ provides support for Cross Language Client and Protocols as well as a powerful messaging broker which is supported in Java, C, C++, C#, Ruby, Perl, Python, PHP.

Read the full article by Ade Ayonrinde.

Update:
An interesting thread docbook and HTML is going on at The Server Side.

A high quality version of the screencast also available.