Thoughts of Hiram Chirino

Keep an eye out for ZooKeeper

noreply@blogger.com (Hiram Chirino) — Thu, 17 Jul 2008 19:04:00 +0000

Wow, I love the simplicity that ZooKeeper brings to a really hard set of distributed problems. Check out this Introductory Video that explains it more in depth. Basically group leadership/coordination and cluster wide configuration issues are taken care of if you Use ZooKeeper.

Oh and it's an Apache Project now. Yay! Seems like the project website is still not fully setup since they are migrating from SourceForge to Apache, be here's a link to the source tree.

TODO: Double Write Buffers

noreply@blogger.com (Hiram Chirino) — Thu, 17 Jul 2008 18:57:00 +0000

Note to self: investigate implementing the Double Write Buffers idea in ActiveMQ. ActiveMQ keeps several indexes into the persistent messages that it's holding and when ActiveMQ is shutdown ungracefully, we rebuild the indexes from the data logs due to them being in inconsistent state. If your queueing up millions of messages, building those indexes can take a long time.

Double buffering may allow us fix inconistencies in those index and gets us running faster..

ActiveMQ/SpecJMS/Camel Webinar

noreply@blogger.com (Hiram Chirino) — Mon, 02 Jun 2008 19:30:00 +0000

Whoa, time flies by, and I forgot to post about the upcoming webinar that I will be co-hosting with Rob Davies on June 10th. We will be covering some messaging basics, introducing Apache ActiveMQ and Apache Camel to the audience, but most interesting I think will be the section where Rob will be covering the results that IONA has been seeing benchmarking ActiveMQ against the SpecJMS2007 test suite. I totally agree with Rob's comment that "An independent benchmark is important, because it negates the chance to skew home groan tests to a vendor's strengths."

InfoQ Covers ActiveMQ 5.1 Release

noreply@blogger.com (Hiram Chirino) — Thu, 29 May 2008 19:51:00 +0000

InfoQ has posted nice article on the new features in the ActiveMQ 5.1 release versus the last 4.1 release:

Apache ActiveMQ, an open source provider of enterprise messaging services, recently released version 5.1 which includes improvements in stability and performance of the message broker product. This version also includes support for priority message ordering and a Microsoft Message Queue (MSMQ) to ActiveMQ Bridge with the new msmq transport component.

There are also improvements in the monitoring module of ActiveMQ container. A new DestinationSource class was added to access the available Queues or Topics or listen to Queues/Topics being created or deleted in the container. There is a new API to help end users view available destinations and query them to find JMS statistics such as active queue count, queue depth, number of messages etc.
Read More...

ActiveMQ 5.1.0 Release

noreply@blogger.com (Hiram Chirino) — Wed, 07 May 2008 16:56:00 +0000

For all of you who ran into issues with ActiveMQ 5.0.0 when running it in anger, I highly recommend you give the just released ActiveMQ 5.1.0 a try. This release focused focused on making the broker rock solid. It resolved several bugs which only reared their heads in high load situations. Memory leaks have been squashed and performance has even improved in several areas.

Even if you have not had seen any issues with your 5.0.0 installation, I'd highly recommend you upgrade to 5.1.0 to avoid running into some of the bugs that have been addressed in the release.

Mulitcast not working on a Linux box?

noreply@blogger.com (Hiram Chirino) — Thu, 10 Apr 2008 17:59:00 +0000

Just ran into a problem where some mutlicast tests were failing on a linux box and I could not figure out why. Did a little bit of research and found out that you may need to add a route for it first. So if you have this problem try running:

route add 224.0.0.0 netmask 240.0.0.0 dev eth0

or if you have an older version of linux like me:

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

More ActiveMQ Fanfare

noreply@blogger.com (Hiram Chirino) — Wed, 02 Jan 2008 14:59:00 +0000

Nice to see the new year starting off right. I Noticed this post over at the eaimatrix.com:

A plethora of message queuing products exist in today's EAI market, all aimed at providing solutions to the problem of application integration. Few can however lay as much claim to fame as ActiveMQ, an open source Message Broker and JMS/Enterprise Integration Patterns provider which is licensed, developed and distributed under the open source Apache emblem. ActiveMQ provides support for Cross Language Client and Protocols as well as a powerful messaging broker which is supported in Java, C, C++, C#, Ruby, Perl, Python, PHP.

Read the full article by Ade Ayonrinde.

ActiveMQ Webinar

noreply@blogger.com (Hiram Chirino) — Tue, 13 Nov 2007 15:29:00 +0000

Late notice.. but James and I are doing an ActiveMQ Webinar in an hour and half. Sign up for it and let us know what you thought about it.

Docbook is dead.. Long live HTML

noreply@blogger.com (Hiram Chirino) — Tue, 02 Oct 2007 16:22:00 +0000

Yay.. The Apache Camel project has started to generate some beautiful looking PDF documentation from standard HTML by using prince and the Boom style sheet against our wiki. We contacted the Boom folks and they cleared up the license terms of the Boom file so that it's officially open source. The Boom folks have relicensed under the very liberal MIT license.

Update:
An interesting thread docbook and HTML is going on at The Server Side.

Dip Your Toes in Some Camel Today

noreply@blogger.com (Hiram Chirino) — Wed, 26 Sep 2007 15:36:00 +0000

The James Strachan has put together an awesome intro to Apache Camel.

A high quality version of the screencast also available.

New FUSE Releases

noreply@blogger.com (Hiram Chirino) — Tue, 25 Sep 2007 16:49:00 +0000

It looks like the word is spreading that the FUSE has a new release out. The release includes enhancements across the FUSE family of products including, FUSE ESB, FUSE Message Broker, FUSE Services Framework and FUSE Mediation Router. The releases include significant performance and feature improvements and tighter integration between the FUSE ESB and all components of the product family.

For those of you who don't know, the FUSE family of products are IONA supported versions of several Apache projects like ActiveMQ and ServiceMix.

ServiceMix has Graduated

noreply@blogger.com (Hiram Chirino) — Thu, 20 Sep 2007 16:17:00 +0000

Congratulations to the ServiceMix project for graduating from the Incubator! Having participated in several projects that have graduated from the Apache incubator, I can attest that this is an awesome milestone to have completed.

For those of you who don't know what ServiceMix is, it's by far the BEST open source ESB available today. The components in this ESB use the JBI 1.0 spec to integrate so end users and component developers can avoid the vendor lock in that is so common in todays commercial ESB space.

Learn how to Use ActiveMessaging to leverage ActiveMQ from Rails

noreply@blogger.com (Hiram Chirino) — Fri, 02 Mar 2007 15:56:00 +0000

You have to checkout this great InfoQ Article on ActiveMessaging. It's an outstanding writeup on how to use ActiveMQ from Rails. I think this is just another sign that STOMP is gain momentum.

StompConnect - Finally, you can talk nativley to any JMS server from any language.

noreply@blogger.com (Hiram Chirino) — Sun, 25 Feb 2007 02:40:00 +0000

OMG, James is at it again! He whipped up the new StompConnect project at a blink of an eye. Firstly, let me introduce you to Stomp. Stomp was designed to be a super simple wire protocol for clients to talk to Message Oriented Middleware (MOM), like ActiveMQ. Since the protocol is text based and simple to implement, multiple language clients and servers were created with little effort.

And now thanks StompConnect, every JMS compliant MOM in existence (which is like almost all of them) can now be talked to using Stomp! This is a huge deal since the wire protocol for most JMS servers is proprietary and and getting a pure language client for any given mom was either difficult or impossible!

ActiveMQ Updates

noreply@blogger.com (Hiram Chirino) — Tue, 20 Feb 2007 15:20:00 +0000

Wow, it's been a while since I've posted but several exiting events have been happing in the ActiveMQ arena.

Apache ActiveMQ has GRADUATED out of the incubator and is an official Apache project!
The ActiveMQ website has moved to http://activemq.apache.org and has received a new face lift thanks to yours truly.
ActiveMQ 5.0 development is making huge progress and Rob has put up an excellent post outlining the upcoming features in the next version of ActiveMQ.

If you want to get a preview of those features, get the latest from the ActiveMQ trunk and kick it around. Feedback would be appreciated.

Web 2.0 Communication Layer: from HTTP to Comet to Internet Mess

noreply@blogger.com (Hiram Chirino) — Mon, 16 Oct 2006 21:31:00 +0000

Checkout this server side thread. Folks are starting to think about using JavaScript on the browser to access an "Internet Messaging Bus". They want to have thing like:

Guaranteed delivery
Once and only once deliveryGuaranteed
order of deliveryServer
push and client pullFunny

thing is that most of all that is available today with ActiveMQ! And to get really great performance, use ActiveMQ with Jetty! ActiveMQ comes with a simple little JavaScript API that allows you to access the ActiveMQ message bus using Comet style http polling. And ActiveMQ provides you all the above guarantees like all good message brokers.

Other news on this front is all the Ajax tool kits are starting to look at being able to inter-operate. At a minimum, a page with multiple tool kits will need to share it's connections back to the server. So they will need to share an API to broker requests back to the server. I'll keep an eye out for this API because it sounds like something that could be easily tied into ActiveMQ.

AMQP - An Interesting Start

noreply@blogger.com (Hiram Chirino) — Wed, 21 Jun 2006 19:50:00 +0000

A few days ago the AMQP spec was announced on TSS. I quickly downloaded the spec and I have some initial impressions.

#1: I think it's unfortunate that the "AMQP" sounds too much like it has something to do with ActiveMQ which most folks abbreviate to AMQ, as in, "Have you downloaded AMQ 4?". This along with the fact that AMQ and AMQP are both related technologies, the first is a mom provider and the second is a wire protocol for mom providers.

#2: This is nice spec. I like the client specified "binding" concepts introduced in this spec. Perhaps these concepts can be introduced in higher level APIs like JMS one day.

#3: It seems that one of the goals of the spec is for vendors to be able to interoperate with each other. I have got a feeling that specs like WS-Notification or STOMP have a better chance at accomplishing this. Binary wire protocols are hard to implement and I doubt there will be many implementations of it. If a good open source reference implementation becomes available then this would become more plausible.

#4: It does not go into details of how the content of the messages should be encoded. This effectively means that JMS implementations will not be able to interoperate using AMQP since different implementations would encodes the content of a JMS Message differently. Even for simple things like TextMessage, would they use UTF-8 or ASCII? And it gets even more complex for messages lke StreamMessage and MapMessage.

#5: I may be wrong, but it seems like messages are always sent asynchronously. In some cases the JMS spec requires messages to be sent synchronously. Perhaps transactions can be used to simulate a synchronous send, but wouldn't that add substantial of overhead?

Problems with Geronimo and RTC

noreply@blogger.com (Hiram Chirino) — Sat, 17 Jun 2006 16:20:00 +0000

The Apache Geronimo project has recently changed it's commit policy to a Review then Commit (RTC) policy. Which means no one can commit to the Geronimo project unless you submit a patch and then get 3 other Geronimo committers to review, apply, test, and then give you 3 +1s.

In theory, this is supposed to increase the communication between the developers. In practice, it's looking a little scary. For example, I just wanted to move some ActiveMQ integration modules from living in the ActiveMQ project to live in the Geronimo project. Now, before I submitted the patch, everyone agreed that it was good idea. I submitted the patch and now I have been waiting 12 days to get enough +1s.

Why is it taking so long? I think it's because the Geronimo project is bigger than what anybody knows. But we are lucky to have a large developer base that is specialized in different areas of the server. This specialization is what allows us to produced a server with high quality parts that in turn give us a high quality server. I don't go wasting time figuring how what's the best way to implement a Transaction Manager or Deployment system, because I trust the guys that are already working on those pieces.

With the RTC policy, if I we want to get Transaction Manager patch in, folks that might not know ANYTHING about Transaction Manager are going to be looking at those patch and pretending they understand what's going on and then giving the +/- 1's. The same thing goes for the developer tooling, JMS Messaging, etc. etc.

I guess in the long run, it may be good that folks get to know other Geronimo systems besides what they specialize in, but in short term, your going to slow down your development model by at least a factor of 10 (I've already been waiting over 10 days to do something that used to take me 10 seconds). Furthermore, since Geronimo is playing feature catch up, most developers already have enough on their plates, I doubt they also want to review and test other folks patches x3 if the previous development pace is to be sustained.

Getting Involved with Open Source

noreply@blogger.com (Hiram Chirino) — Thu, 15 Jun 2006 02:10:00 +0000

I have a younger brother-in-law that also went in to software development field, and he thinks that I'm the luckiest guy in the world since my day job is basically working on open source projects. Every once in a while he asks me, "How can I get more involved with open source?".

I always say, just find a project that you like and start contributing to it. So far, not much has happened... So Jason, screw that "find a project" stuff.. How about you pick up this issue, and implement it? It's a small little feature that lots of folks would find handy!

Blog URL Moved

noreply@blogger.com (Hiram Chirino) — Tue, 30 May 2006 18:04:00 +0000

Since I hate to be dependent on 3rd party services and URLs, I moved my Blog URL to a URL that I own: http://hiramchirino.com

They funny thing is that blogspot wasted no time when I changed how I published my blog, and someone else snatched up my old blogbucket.blogspot.com url.

Beefing up Kaha

noreply@blogger.com (Hiram Chirino) — Mon, 22 May 2006 23:40:00 +0000

Rob Davies and I spent some time last week looking at his Kaha message store implementation. In a way, it's similar to a experimental QuickJournal implementation that I had committed previously. The idea of the QuickJournal was that Journal log files were not deleted and that messages could be easily retrieved from the Journal. The journal would only checkpoint to the long term store the location of where the messages are located in the journal.

In a way, the long term store (JDBC in most cases) is being used like an index into the Journal files. This increases the performance of the journal since the amount of data that needs to be stored in long term store is drastically reduced and generally of a small size which works better with JDBC batch operations. Also messages do not need to be batched up in memory (for batch insertion into the DB) thus reducing the memory impact of the message store.

The funny part is that at some time last week, somehow a wager got started about who could build the fastest message store implementation that could stay under 64 megs of memory usage when a queue was loaded up with 10,000,00 1k messages. Kaha at the current time keeps it's indexes fully in memory. So Rob started looking into a way to optimize the down so that they could fit in 64 megs (Rob is the optimization King BTW). I went down the route of we need to only load parts of the index into memory. I even shared my algorithm concept with him as long as he did not use it, LOL. In the end we realized, it was not going to be a weekend deal to implement this stuff and it would be best to work on a single solution together.

Kaha, IS a nice API and is much more general purpose than the MessageStore APIs. Some of the problems that Kaha currently has is that it does not guarantee constancy of the indexes and it does not support transactional operations. Those are 2 things that the journal can do today, and which Kaha could do if we modified it's DataManager so that it journaled operations instead of just storing data items. So I'm going to try to integrate many of the Journal concepts into the DataManager so that:

The data file acts as a redo log that is 'replayed' on startup to bring the indexes to a consistent state
Use async batch writes for increased throughput: micro benchmarks showed that the journal can write at about 21 megs/s while the current DataManager maxes out at 8 megs/s

Other things to consider is that since the interfaces to the Kaha APIs are based on the List and Map interfaces, there is no easy way to:

switch between doing async and sync operations against the data files. Currently Kaha has a force() method on the store that does syncs up any pending write but this is not optimal when using async batched writes (you end up syncing on a subsequent write).
associate a transaction with a operation against a list or a map

An idea I've been floating in my head is the ability to have multiple proxies to a single physical container. Each proxy could be enlisted in a different transaction or it's flag to do sync vs. async actions changed.

As you might be able to tell by now, I'm on the Kaha crack now... bless Rob.

A Closer Look at the Gigantic Destination Nut

noreply@blogger.com (Hiram Chirino) — Mon, 22 May 2006 21:14:00 +0000

I exposed the gigantic destination issues that ActiveMQ has in a previous blog post. I'll take a little time to expand on the issue and why it's not simple to solve, and what ActiveMQ 4.0 does today.

It's obvious that we need to swap messages to disk when a queue needs to hold more messages than it could hold in RAM. We sometimes also call that spooling messages to disk. The issues that make this hard to implement are:

Writing a message to disk slows you down a little, avoid it if possible. Sometimes you have no choice if the message was marked a persistent.
Sometimes we may need to swap out even non-persistent messages.
Avoid chucking a message out of ram if possible since loading it back from disk is REALLY slow.
When a consumer is ready to consume a message, that message should already be in memory, waiting for it to load from disk will lead to consumer starvation.
Even keeping lists of message references to where messages are on disk can use up too much memory. 10,000,000 disk locations in a linked list where every node in the list used only 100 bytes would still chew up about 100 megs of memory.

ActiveMQ 4.0 takes a simple approach and when sending persistent message to a Queue, it uses a MessageReference when moving a message though the Broker message dispatch process. A process that could take a while for a message to go from producer to a consumer and finally message acknowledgement. The MessageReference starts out being direct, in that it hold a reference to the message to keep it in RAM, but if the reference count drops below 1, then the direct reference is dropped. The reference count is allowed to drop to 0 when the message is just sitting in the Queue's message list or in a consumer's pending list. The reference count is > 0 while it's being dispatched to a consumer. The MessageReference knows how to reload a Message from the peristence store when it's reference count goes up above 0.

This is a quick and dirty fix and it works, but it obviously does not fix all the issues outlined initial. The shortcomings of the current solution is that:

it is only implemented for Queues
consumer starvation problem can exist since it does not persisted load messages asynchronously
it keeps a list of MessageReference objects which can still exhaust JVM memory

Mapping Beans to REST

noreply@blogger.com (Hiram Chirino) — Mon, 22 May 2006 19:36:00 +0000

I'm one of those guys that thinks that REST is great technology. Sure, REST is does not do everything that SOAP can do, but I think that REST is built to be SIMPLE, something that SOAP and it's WS-* buddies forgot about.

What we are missing is a good standard way to map REST to the simple POJO programming model that most of the Java industry has been quickly adopting. Seems SeXFire Dan has good start on a way of doing that!

Dan if you read this, I would make it even simpler if by default the method names for a service are determined by convention. For example, for a given XService, the methods:

getX(...) : is automatically mapped to a HttpMethod.GET
deleteX(...): is automatically mapped to a HttpMethod.DELETE
addX(...): is automatically mapped to HttpMethod.POST
updateX(...): is automatically mapped to a HttpMethod.PUT

Scaling to Gigantic Queues and Topics

noreply@blogger.com (Hiram Chirino) — Mon, 22 May 2006 17:49:00 +0000

One of the current issues with ActiveMQ is that it's an uber fast message broker while consumers are online and consuming messages, but things start to kinda not works so great when you have a use case where you want to queue up 'work/messages' for a consumer that will be offline for days.

In ActiveMQ 4.0, we have hacked in some initial support for loading up a queue with a huge number of messages without blowing up the memory usage of the JVM, but it's a bit hacky and it may fail work right if a consumer comes back online and the consumer recovery process kicks in.

All in all 4.0 is a solid broker with a ton of new and exciting features, but personally, I would like to focus on getting 4.1 to be the broker that can handle Gigantic Queues and Topics. I'll post some more messages on this topic in the next few hours as I recap what I've discovered in the last few weeks.

noreply@blogger.com (Hiram Chirino) — Mon, 22 May 2006 17:46:00 +0000

This just a quick post on why I'm going to start blogging.

I think that I get involved in many small projects, and sometimes I don't get back some of the smaller ones. I may be deluding myself, but I'm hoping that If I post about it, one day I will re-read my blog and get back to projects that I had found interesting before. I also hope other folks may find some of these projects interesting and will help with them.