About
Disclaimer: the views expressed on this site are mine alone and do not necessarily reflect the views of my employer or its affiliated entities.
While reading Comparing the Java Serialization Options I ran across the a cool google code project which has done an excellent job benchmarking a wide variety of serialization options for java.
I’ve had been researching the protobuf encoding format for a while and really liked it. But I did not really like the Java implementation that Google had published. It was kinda clunky to use and I saw several optimizations that could be used that were missing. Optimizations that could create huge performance wins when applied to the usage patterns of an enterprise messaging system like Apache ActiveMQ. So I created a new protobuf implementation in the ActiveMQ project.
Naturally, I was curious to see how the activemq protobuf implementation stacked up against similar technologies. So I grabbed the V1 benchmark source code and added our implementation to it. If you want to do the same, apply this patch.
Once I ran the benchmark and I was very pleased with the results. I’m including the performance graphs of our impl and standard protobuf and thrift for comparison.
As it turns out, our implementation looks awesome in the benchmark! How about that decoding speed!
It’s getting late here.. so I’ll have do a follow up post explaining how come we did so much better.
9 Comments for The ActiveMQ Protobuf Implementation Rocks!
David Yu | September 18, 2009 at 6:43 am
kenton | September 18, 2009 at 11:24 am
What made you decide to write your own implementation rather than contribute patches with your optimizations to Google?
Monty Taylor | September 18, 2009 at 11:47 am
Looks interesting! Any reason you didn't work with or contribute to the main Protobuf project? (I ask because I'm a user of it and have contributed to it and would love for interesting new work to continue to make its way in)
Hiram Chirino | September 18, 2009 at 12:13 pm
David Yu,
I'll include some info on the flavor of builder pattern that we are using in my follow up post.
The problem with the build is that it's part of a large distribution. If you check out
http://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow/
you should be able to do a quick build of all using:
mvn install -Dtest=false
Hiram Chirino | September 18, 2009 at 12:32 pm
Kenton,
Firstly, I wanted to also implement a java based code generator. That way it would be easier have a cross platform build that would be easily embeddable in a maven plugin. Since the google protobuf code generator is c based, there's not really much that sharing that can be done there.
Secondly, there are a few features that google java object model implements which I personally do not need for my needs and have no interest in implementing since it would only make the code generator more complex. For example: runtime reflection of messages.
Some of the optimizations require a different object model/pattern from what is used by the google version. It would be hard to retrofit some of this stuff into that model.
All that being said, if the google folks would like me to submit the impl as a patch, I'd be happy to.
kenton | September 18, 2009 at 12:44 pm
I don't think we want to maintain two independent implementations, but I'd like to know what optimizations you made, especially if the same optimizations could be made to our implementation. I have to say I'm skeptical of your deserialization benchmark — is it really doing the same work as the official implementation?
Hiram Chirino | September 18, 2009 at 12:56 pm
BTW, it's not like I was working on this stuff in secret. I did post to the google group to see if anyone would be interested collaborating.
I assumed folks were not. 
Sam | September 22, 2009 at 2:15 am
You stated in your followup post that the deserialization is lazy. I'm wondering about this in conjunction with Kenton's skepticism regarding the deserialization performance graph. Does that graph really represent deserialization or does it represent the code simply saying "I'll do it later…" ?
Hiram Chirino | September 22, 2009 at 9:33 am
Sam,
Yes that's right. The bench mark only requires the parser to create a Bean object from the encoded byte[].
Unfortunately (for the benchmark), decoding does not kick in until some fields of the object are accessed. And even if some fields were accessed, it would only partially decode the data. It would need to access fields in all the nested messages if it wanted to test full decoding.
Curious/cant wait about the follow up post. Good job!
I'm assuming the extra Message.Builder is scratched … instead the pojo's setter methods returns the pojo (like the Builder pattern)
The "mvn install" of the project fails.
"org.apache.activemq:activemq-util:jar:6.0-SNAPSHOT" could not be resolved
Looking forward to your follow up post. Good job!