- About Scala
- Documentation
- Code Examples
- Software
- Scala Developers
Re: Scala vs. Erlang
Sat, 2009-03-07, 12:15
I've enjoyed this discussion, but I too don't want to drag it off topic too far. So this will be my last comment here, I'll leave the last word to others.
Final comments in-line.
R.
2009/3/7 Alexis Richardson <alexis.richardson@gmail.com>
I'd have to disagree with you there. The broker in question was being run by the same organisation who envisioned, created and wrote AMQP and QPid to meet their messaging needs. So .. it was created for this. Exactly this.
I believe that most messages were in the 0.1 to 1 Mb range and that gigabit ethernet and other network acceleration technologies are employed. The system transmits fully described business 'objects' with all necessary context so that any clients who subscribe to the feed can take all necessary processing action without any need for farther querying, data look-up or large quantities of specialised messages. This isn't an RTP system so fewer large messages makes more sense than floods of small messages in this scenario. In previous iterations of the same system it was found that there was little or no reuse of messages - most focussed message queues had only a single subscriber. In the newer version, with much larger more fully realised messages there are often dozens of subscribers. This level of reuse has been very beneficial.
--
Blog : http://robertlally.com
Final comments in-line.
R.
2009/3/7 Alexis Richardson <alexis.richardson@gmail.com>
Rob,You are, of course, correct. There's nothing in software that is black and white. Everything is context sensitive.
Hi there, and thanks for clarifying that. More below.
Scala folks, please tell me to shut up if this goes too off-topic.
On Sat, Mar 7, 2009 at 10:01 AM, Robert Lally <rob.lally@gmail.com> wrote:
> Alexis,
> I wouldn't contradict your general principles, you certainly know infinitely
> more than I do on the subject.
Thanks but actually, IMHO, it is very hard to say anything much about
performance without doing lots and lots and lots of different tests.
This is incredibly expensive, and often misguided. So we tend to sit
on our hands and wait for the community to pick out issues where
performance may not suffice for some use case.
If there is an issue, often it can be fixed. Less often it reveals a
more general principle -- your large files example is a great instance
of that. AMQP is not designed for file streaming, not yet anyway, but
it could be adapted to that case by chunking (see for example Nanite
from Engineyard). So this means apple-apple broker comparisons are
hard. Ho, and indeed, hum.
I'd have to disagree with you there. The broker in question was being run by the same organisation who envisioned, created and wrote AMQP and QPid to meet their messaging needs. So .. it was created for this. Exactly this.
BTW - some of you may be interested in how Facebook use erlang. Take
a look at this: http://www.facebook.com/note.php?note_id=51412338919
It seems to me that they are not using OTP...
> I rooted through some old dev notebooks to see if I could find any notes I'd
> made on this subject and I found a few hastily scribbled notes from the
> meetings.
> The important piece of data in your writing was the 300b per message figure.
Right, although more or less the same is true up to a few K, then most
messaging systems start to very very gradually degrade in either t/put
or latency or both, on the margin.
> In the area I was working the average message size was a distribution in the
> 0 to 10 Mb range.
OK. *Please* don't take this the wrong way - I realise you are
reporting an anecdote from someone else from the past! I merely wish
to comment.
That's quite a range and it does contain small messages -- though I
assume you are talking about 'large messages'. Absent other info, the
principle of maximum likelihood leads me to assume an average message
size of 5Mb.
Assuming a normal distribution of message size, let's (wlog) assume
the claim is that RabbitMQ could manage one message per second on
average in your friend's test, and therefore that Qpid was sending the
equivalent of 1,000 5Mb messages per second. Because these are
normally distributed ex hypothesi, that's around 5Gb per second. In
which case astonishment must be expressed -- even if the network used
was more than 1GigE.
I believe that most messages were in the 0.1 to 1 Mb range and that gigabit ethernet and other network acceleration technologies are employed. The system transmits fully described business 'objects' with all necessary context so that any clients who subscribe to the feed can take all necessary processing action without any need for farther querying, data look-up or large quantities of specialised messages. This isn't an RTP system so fewer large messages makes more sense than floods of small messages in this scenario. In previous iterations of the same system it was found that there was little or no reuse of messages - most focussed message queues had only a single subscriber. In the newer version, with much larger more fully realised messages there are often dozens of subscribers. This level of reuse has been very beneficial.
So being charitable we must conclude *either* that the message
distribution was not normal, *or* that the test results are
unrepresentative of a real case, *or* that the broker/client set-ups
were incomparable. I won't comment on the latter two. In the first
case, then, charitably assume that there is ONE large message per
second, and 999 small ones. This also does not make sense to me ---
it would basically imply that RabbitMQ is single threaded and Qpid is
not, which is simply not the case.
> Erlang's propensity towards pass-by-value means that every
> time you pass on a reference to a 10Mb enclosure you create a copy of it so
> the system promptly becomes memory bound to an extreme degree with many
> copies of many large structures being held.
This is interesting and I believe remains 'on topic' for this thread.
I'll comment on what RabbitMQ does. Suppose you have one broker on
one box running one erlang VM. Then, RabbitMQ will not pass by value.
RabbitMQ only passes by value for messages of up to 64b size, or
across VMs. To be honest I am not sure how erlang/OTP does this magix
but would be happy to get more info from one of the engineers if the
Scala list is interested.
In any case, I would not recommend that 10Mb data be in a single
enclosure in a messaging broker, because the broker's value arises
from abstracting the network (as a database does the disk) and so
client-broker messaging is the bottleneck and should be chunked.
> This wasn't a theoretical problem, this was based on playback of real
> messages passing through a real broker.
Understood and I hope my comments above constitute a plausible
critique of the claim.
Thanks very much indeed for posting more --- it's been very helpful.
alexis
> R.
>
> 2009/3/6 Alexis Richardson <alexis.richardson@gmail.com>
>>
>> Hi guys,
>>
>> Alexis here from RabbitMQ. Apologies for going OT but I feel the need
>> to frame some radical claims with some real world experience.
>>
>> Here is some data from 2007, that Intel published after running tests:
>>
>> http://www.intelfasterfs.com/2007/11/intel-low-latency-trading-lab-set-to-improve-financial-trading-performance/
>>
>> The key statement is "Using industry standard hardware, an OPRA feed
>> input rate of 1.3million messages per second was distributed and
>> replicated to four concurrent subscribers, via an AMQP 0-8 compliant
>> broker cluster running on a single Intel Xeon Processor 7300 based
>> server, with measured mean latency below 1.5ms".
>>
>> Please note that throughput here refers to 'OPRA messages per second',
>> but financial feeds from OPRA normally get sent as batches of 16
>> messages at a time. So divide by 16 to get the AMQP messages per
>> second. Note also that the 1.3m number quoted is an input rate. And
>> the output rate is 4x that, so the total throughput is 5x. Finally
>> note that (a) this used a C client; (b) the disk was not used -
>> everything was transient; (c) AMQP message size was about 300b per
>> message.
>>
>> What does this tell us? Actually: not much. In 2007 it was possible
>> to create a set up representing a real financial case, and make
>> RabbitMQ run reasonably, but not amaziningly, fast, on a 16 core box.
>> Since then the clustering (thanks erlang) and the features, have
>> gotten a bit better. And chip cores have gotten quite a lot faster.
>>
>> Using RabbitMQ today, and an optimised Java client for raw AMQP, I
>> would still be kind of amazed to see a total throughput (input rate +
>> output rate) of more than a few 10K per second per core. With a C
>> client, I would hope a *single* modern core doing message fanout of 4x
>> should be running at around 40-60K messages per second total
>> throughput, whether using a good erlang broker or a good Java broker.
>> In a cluster - erlang and Scala should both do well. Java may need
>> more hand coding but can be very very fast.
>>
>> Finally, it is still not possible to compare RabbitMQ with Qpid
>> fairly, because the clients and brokers don't quite interoperate. We
>> are getting there on that.
>>
>> alexis
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Mar 6, 2009 at 7:38 PM, Meredith Gregory
>> <lgreg.meredith@gmail.com> wrote:
>> > Robert,
>> >
>> > Chatting with the RabbitMQ guys, they can do 1Ks of msg/sec. Based on
>> > what
>> > you report, that would mean that Qpid can do 1Ms of msg/sec. That would
>> > be
>> > really cool. i'd love to get the details. Can you provide a link or
>> > reference for the comparison you cite?
>> >
>> > Best wishes,
>> >
>> > --greg
>> >
>> > On Fri, Mar 6, 2009 at 11:01 AM, Robert Lally <rob.lally@gmail.com>
>> > wrote:
>> >>
>> >> Hi Meredith,
>> >> I don't want to be seen as arguing for Erlang. I like it well enough,
>> >> but
>> >> I doubt I'll ever use it because I doubt I'll ever work on an
>> >> application
>> >> where up-time is vital and geriatric-snail-like performance is
>> >> acceptable.
>> >> Just to be clear, the Erlang broker had one thousandth of the message
>> >> handling capacity of the Java broker. With Erlang's neat distribution
>> >> features, you could deploy more to nodes to make up for this but then
>> >> you'd
>> >> have decreased performance from network latencies and the additional
>> >> overhead of nodes required for co-ordination. In the case that was
>> >> tested -
>> >> going from a single 128 core box to thousands of boxes doesn't seem
>> >> like a
>> >> good trade-off. You'd have to go to a cloud because, well, you'd need
>> >> one
>> >> ... just for a single app.
>> >> That said, if community support is the most important thing to you,
>> >> then
>> >> Java is a much safer bet. I would guess that anyone who's considering
>> >> Erlang
>> >> or Scala for a project is someone who likes living on the edge. Off the
>> >> beaten path ( for now, at least ) languages like Scala are inherently
>> >> risk.
>> >> Couple this with the fact that, whilst Scala can use lots of Java
>> >> libraries, it doesn't always play well with existing tools and
>> >> frameworks -
>> >> try writing a Seam application in Scala for instance. So you're not
>> >> really
>> >> getting the full benefit of the Java community there either.
>> >> In counterpoint, to my counterpoint, I like Scala because it is a good
>> >> middle ground - a nice OO/functional hybrid, good concurrency support,
>> >> a
>> >> healthy community, access to many Java libraries and more complex but
>> >> more
>> >> powerful language than Java.
>> >>
>> >> Rob Lally.
>> >>
>> >> 2009/3/6 Meredith Gregory <lgreg.meredith@gmail.com>
>> >>>
>> >>> Robert,
>> >>>
>> >>> i agree that Erlang is quite solid from a performance and fault
>> >>> tolerance
>> >>> perspective. It's got years on Scala, and even a few on Java. i also
>> >>> think
>> >>> that they've got a more stable concurrency semantic -- which is
>> >>> crucial if
>> >>> you'd like to start leveraging some of the new technologies coming
>> >>> from the
>> >>> Concurrency and Types communities on types for concurrency and
>> >>> model-checking.
>> >>>
>> >>> On the other hand, essentially what keeps me programming in Scala is
>> >>> the
>> >>> availability of the entire (and very rapidly growing) Open Source Java
>> >>> codebase. Erlang just doesn't have the same sort of reach or interop.
>> >>> It's
>> >>> not that it isn't possible, it just isn't there. So, it's very hard to
>> >>> pitch
>> >>> this in -- say -- the Enterprise, except for niche jobs that are
>> >>> mainly
>> >>> about comms.
>> >>>
>> >>> Likewise, i have found Erlang to be not quite ready for my grandmother
>> >>> to
>> >>> use when it comes to installation and configuration of it's services.
>> >>> For
>> >>> example, i spent 18 hours (a couple of years ago -- so things may have
>> >>> improved) getting their web framework up and running and it was still
>> >>> very
>> >>> brittle. And, there was little to no community support. For a
>> >>> lift-based
>> >>> app, i start with a 1-liner to generate a basic application shell,
>> >>> fill in
>> >>> my logic, and have an application i can deploy and run with a 1-liner.
>> >>> If i
>> >>> run into a bug that i can't solve, i ask the community. The response
>> >>> time is
>> >>> typically much less than 24hrs. Five years ago that kind of service
>> >>> would
>> >>> have cost an arm and a leg.
>> >>>
>> >>> One very real place for Erlang is in the cloud. In an
>> >>> infrastructure-level service, say cloud-level messaging, no one knows
>> >>> or
>> >>> cares what's "under the hood" unless it some how affects features set,
>> >>> performance, fault-tolerance, etc.
>> >>>
>> >>> Best wishes,
>> >>>
>> >>> --greg
>> >>>
>> >>> On Fri, Mar 6, 2009 at 4:49 AM, Robert Lally <rob.lally@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> As with many things in the software world, I don't think that there's
>> >>>> a
>> >>>> black and white answer to this question.
>> >>>> One thing that is worth considering, is the performance
>> >>>> characteristics
>> >>>> of your application and your reasons for distributing it. If you're
>> >>>> hope is
>> >>>> to get better performance, then for certain classes of problems you
>> >>>> might be
>> >>>> better off with Erlang than something that runs on the JVM but most
>> >>>> of the
>> >>>> time, I don't think that would be true.
>> >>>> A friend, who worked on the Java AMQ broker Apache Qpid, compared it
>> >>>> to
>> >>>> the Erlang Rabbit AMQ implementation and was startled by the results.
>> >>>> Running playback of real client transactions through it resulted in
>> >>>> 1000
>> >>>> times better performance from the Java version. That's quite a
>> >>>> difference. I
>> >>>> don't know how a Scala implementation would fare here, but I assume
>> >>>> that
>> >>>> Scala performance would be more in line with Java performance than
>> >>>> Erlang.
>> >>>> That's the only case I know of, of two parallel implementations of a
>> >>>> real world project. If anyone knows of any others, it would be
>> >>>> interesting
>> >>>> to hear of them - particularly if the results were very different
>> >>>> from this.
>> >>>> When you read Erlang literature, a great deal of the work done with
>> >>>> it
>> >>>> is combined Erlang/C projects. It seems that Erlang is used to
>> >>>> provide
>> >>>> up-time and C is used to provide the performance.
>> >>>> R.
>> >>>>
>> >>>> 2009/3/4 Franco Lombardo <f_lombardo@hotmail.com>
>> >>>>>
>> >>>>> This times, the Erlang programming language is getting more and more
>> >>>>> interest among the smartest guys in the programming community. I
>> >>>>> think
>> >>>>> that
>> >>>>> the major advantages of Erlang compared to Scala are:
>> >>>>>
>> >>>>> 1) Erlang's syntax seems simpler (that doesn't imply that
>> >>>>> programming
>> >>>>> in
>> >>>>> Erlang is simpler than programming in Scala)
>> >>>>> 2) It's very simple to create distributed Erlang programs. I know
>> >>>>> that
>> >>>>> this
>> >>>>> is possible also in Scala, but I think there is not enough
>> >>>>> documentation
>> >>>>> about this (neither "Scala by Example" nor "Programming in Scala"
>> >>>>> talks
>> >>>>> about this topic).
>> >>>>> 3) Erlang is designed for "foult tollerance". In Erlang the "link"
>> >>>>> function
>> >>>>> can link processes, and when a process die the linked process is
>> >>>>> informed.
>> >>>>> Moreover a monitor process can respawn the died process or cleanup
>> >>>>> the
>> >>>>> system. Is there somthing like this in Scala.
>> >>>>>
>> >>>>> Thanks in advance from a Scala enthusiast.
>> >>>>>
>> >>>>> Bye
>> >>>>>
>> >>>>> Franco
>> >>>>>
>> >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> >>>>> http://www.francolombardo.net
>> >>>>> Scala, Java, As400.....
>> >>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Blog : http://robertlally.com
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> L.G. Meredith
>> >>> Managing Partner
>> >>> Biosimilarity LLC
>> >>> 806 55th St NE
>> >>> Seattle, WA 98105
>> >>>
>> >>> +1 206.650.3740
>> >>>
>> >>> http://biosimilarity.blogspot.com
>> >>
>> >>
>> >>
>> >> --
>> >> Blog : http://robertlally.com
>> >
>> >
>> >
>> > --
>> > L.G. Meredith
>> > Managing Partner
>> > Biosimilarity LLC
>> > 806 55th St NE
>> > Seattle, WA 98105
>> >
>> > +1 206.650.3740
>> >
>> > http://biosimilarity.blogspot.com
>> >
>
>
>
> --
> Blog : http://robertlally.com
>
--
Blog : http://robertlally.com
Sat, 2009-03-07, 13:17
#2
Re: Scala vs. Erlang
I'd just like to say that I welcome such discussion. It's probably
better in scala-debate@ than scala@, but I'm glad to be on the kind of
list where nobody did answer your call to "tell me to shut up".
2009/3/7 Alexis Richardson :
> Rob,
>
> On Sat, Mar 7, 2009 at 11:14 AM, Robert Lally wrote:
>> I've enjoyed this discussion, but I too don't want to drag it off topic too
>> far. So this will be my last comment here, I'll leave the last word to
>> others.
>> Final comments in-line.
>
> Agreed. If others find this of interest I shall rejoin the conv.
>
> A couple of clarifications below.
>
>
>>> If there is an issue, often it can be fixed. Less often it reveals a
>>> more general principle -- your large files example is a great instance
>>> of that. AMQP is not designed for file streaming, not yet anyway, but
>>> it could be adapted to that case by chunking (see for example Nanite
>>> from Engineyard). So this means apple-apple broker comparisons are
>>> hard. Ho, and indeed, hum.
>>
>> I'd have to disagree with you there. The broker in question was being run by
>> the same organisation who envisioned, created and wrote AMQP and QPid to
>> meet their messaging needs. So .. it was created for this. Exactly this.
>
> AMQP is designed by a Working Group consisting of multiple firms of
> which our RabbitMQ team has been one since way back. I know the JPM
> guys extremely well and what they do with AMQP. File streaming has
> been deprecated from the protocol and not implemented. Qpid does not
> interoperate with RabbitMQ yet, though we are all working on that. We
> have a mailing list for RabbitMQ where you can look for comments about
> performance.
>
>
>>> Assuming a normal distribution of message size, let's (wlog) assume
>>> the claim is that RabbitMQ could manage one message per second on
>>> average in your friend's test, and therefore that Qpid was sending the
>>> equivalent of 1,000 5Mb messages per second. Because these are
>>> normally distributed ex hypothesi, that's around 5Gb per second. In
>>> which case astonishment must be expressed -- even if the network used
>>> was more than 1GigE.
>>
>> I believe that most messages were in the 0.1 to 1 Mb range and that gigabit
>> ethernet and other network acceleration technologies are employed.
>
> Again:
>
> Ceteris paribus... If all the messages are at the *lowest* end of your
> range, i.e. 100Kb then claiming 1,000x faster suggests the minimal
> claim is:
>
> * RabbitMQ sends one 100Kb message per second
> * Qpid sends 1,000 of these per second
>
> The latter implies a total network throughput of 1,000 * 100,000 * 8
> bits = 800 Mbit/sec. It is very hard to get above 700 Mbit/sec on
> 1GigE without saturating it. But let's say that Qpid is sending 1,000
> messages per second of 100Kb each, then the claim that RabbitMQ is
> 1,000 slower implies the claim that RabbitMQ can send at most 1x 100Kb
> message per second. People are very welcome to test this claim and
> draw their own conclusions.
>
>
>
>
>> The system transmits fully described business 'objects' with all necessary
>> context so that any clients who subscribe to the feed can take
>> all necessary processing action without any need for farther querying, data
>> look-up or large quantities of specialised messages. This isn't an RTP
>> system so fewer large messages makes more sense than floods of small
>> messages in this scenario. In previous iterations of the same system it was
>> found that there was little or no reuse of messages - most focussed message
>> queues had only a single subscriber. In the newer version, with much larger
>> more fully realised messages there are often dozens of subscribers. This
>> level of reuse has been very beneficial.
>
> I agree, which is why RabbitMQ does not copy by value for messages
> larger than 64 bytes.
>
> Cheers
>
> alexis
>
Sun, 2009-03-08, 02:37
#3
Re: Scala vs. Erlang
All,
i really appreciate this sort of discussion. It's an engineering discussion: focused on crisp clear argumentation; aimed at arriving to a common understanding of a relevant topic. Thanks to all who have participated.
Best wishes,
--greg
On Sat, Mar 7, 2009 at 4:12 AM, Ricky Clarkson <ricky.clarkson@gmail.com> wrote:
--
L.G. Meredith
Managing Partner
Biosimilarity LLC
806 55th St NE
Seattle, WA 98105
+1 206.650.3740
http://biosimilarity.blogspot.com
i really appreciate this sort of discussion. It's an engineering discussion: focused on crisp clear argumentation; aimed at arriving to a common understanding of a relevant topic. Thanks to all who have participated.
Best wishes,
--greg
On Sat, Mar 7, 2009 at 4:12 AM, Ricky Clarkson <ricky.clarkson@gmail.com> wrote:
I'd just like to say that I welcome such discussion. It's probably
better in scala-debate@ than scala@, but I'm glad to be on the kind of
list where nobody did answer your call to "tell me to shut up".
2009/3/7 Alexis Richardson <alexis.richardson@gmail.com>:
> Rob,
>
> On Sat, Mar 7, 2009 at 11:14 AM, Robert Lally <rob.lally@gmail.com> wrote:
>> I've enjoyed this discussion, but I too don't want to drag it off topic too
>> far. So this will be my last comment here, I'll leave the last word to
>> others.
>> Final comments in-line.
>
> Agreed. If others find this of interest I shall rejoin the conv.
>
> A couple of clarifications below.
>
>
>>> If there is an issue, often it can be fixed. Less often it reveals a
>>> more general principle -- your large files example is a great instance
>>> of that. AMQP is not designed for file streaming, not yet anyway, but
>>> it could be adapted to that case by chunking (see for example Nanite
>>> from Engineyard). So this means apple-apple broker comparisons are
>>> hard. Ho, and indeed, hum.
>>
>> I'd have to disagree with you there. The broker in question was being run by
>> the same organisation who envisioned, created and wrote AMQP and QPid to
>> meet their messaging needs. So .. it was created for this. Exactly this.
>
> AMQP is designed by a Working Group consisting of multiple firms of
> which our RabbitMQ team has been one since way back. I know the JPM
> guys extremely well and what they do with AMQP. File streaming has
> been deprecated from the protocol and not implemented. Qpid does not
> interoperate with RabbitMQ yet, though we are all working on that. We
> have a mailing list for RabbitMQ where you can look for comments about
> performance.
>
>
>>> Assuming a normal distribution of message size, let's (wlog) assume
>>> the claim is that RabbitMQ could manage one message per second on
>>> average in your friend's test, and therefore that Qpid was sending the
>>> equivalent of 1,000 5Mb messages per second. Because these are
>>> normally distributed ex hypothesi, that's around 5Gb per second. In
>>> which case astonishment must be expressed -- even if the network used
>>> was more than 1GigE.
>>
>> I believe that most messages were in the 0.1 to 1 Mb range and that gigabit
>> ethernet and other network acceleration technologies are employed.
>
> Again:
>
> Ceteris paribus... If all the messages are at the *lowest* end of your
> range, i.e. 100Kb then claiming 1,000x faster suggests the minimal
> claim is:
>
> * RabbitMQ sends one 100Kb message per second
> * Qpid sends 1,000 of these per second
>
> The latter implies a total network throughput of 1,000 * 100,000 * 8
> bits = 800 Mbit/sec. It is very hard to get above 700 Mbit/sec on
> 1GigE without saturating it. But let's say that Qpid is sending 1,000
> messages per second of 100Kb each, then the claim that RabbitMQ is
> 1,000 slower implies the claim that RabbitMQ can send at most 1x 100Kb
> message per second. People are very welcome to test this claim and
> draw their own conclusions.
>
>
>
>
>> The system transmits fully described business 'objects' with all necessary
>> context so that any clients who subscribe to the feed can take
>> all necessary processing action without any need for farther querying, data
>> look-up or large quantities of specialised messages. This isn't an RTP
>> system so fewer large messages makes more sense than floods of small
>> messages in this scenario. In previous iterations of the same system it was
>> found that there was little or no reuse of messages - most focussed message
>> queues had only a single subscriber. In the newer version, with much larger
>> more fully realised messages there are often dozens of subscribers. This
>> level of reuse has been very beneficial.
>
> I agree, which is why RabbitMQ does not copy by value for messages
> larger than 64 bytes.
>
> Cheers
>
> alexis
>
--
L.G. Meredith
Managing Partner
Biosimilarity LLC
806 55th St NE
Seattle, WA 98105
+1 206.650.3740
http://biosimilarity.blogspot.com
Sun, 2009-03-08, 02:37
#4
Re: Scala vs. Erlang
All,
i really appreciate this sort of discussion. It's an engineering discussion: focused on crisp clear argumentation; aimed at arriving to a common understanding of a relevant topic. Thanks to all who have participated.
Best wishes,
--greg
On Sat, Mar 7, 2009 at 4:12 AM, Ricky Clarkson <ricky.clarkson@gmail.com> wrote:
--
L.G. Meredith
Managing Partner
Biosimilarity LLC
806 55th St NE
Seattle, WA 98105
+1 206.650.3740
http://biosimilarity.blogspot.com
i really appreciate this sort of discussion. It's an engineering discussion: focused on crisp clear argumentation; aimed at arriving to a common understanding of a relevant topic. Thanks to all who have participated.
Best wishes,
--greg
On Sat, Mar 7, 2009 at 4:12 AM, Ricky Clarkson <ricky.clarkson@gmail.com> wrote:
I'd just like to say that I welcome such discussion. It's probably
better in scala-debate@ than scala@, but I'm glad to be on the kind of
list where nobody did answer your call to "tell me to shut up".
2009/3/7 Alexis Richardson <alexis.richardson@gmail.com>:
> Rob,
>
> On Sat, Mar 7, 2009 at 11:14 AM, Robert Lally <rob.lally@gmail.com> wrote:
>> I've enjoyed this discussion, but I too don't want to drag it off topic too
>> far. So this will be my last comment here, I'll leave the last word to
>> others.
>> Final comments in-line.
>
> Agreed. If others find this of interest I shall rejoin the conv.
>
> A couple of clarifications below.
>
>
>>> If there is an issue, often it can be fixed. Less often it reveals a
>>> more general principle -- your large files example is a great instance
>>> of that. AMQP is not designed for file streaming, not yet anyway, but
>>> it could be adapted to that case by chunking (see for example Nanite
>>> from Engineyard). So this means apple-apple broker comparisons are
>>> hard. Ho, and indeed, hum.
>>
>> I'd have to disagree with you there. The broker in question was being run by
>> the same organisation who envisioned, created and wrote AMQP and QPid to
>> meet their messaging needs. So .. it was created for this. Exactly this.
>
> AMQP is designed by a Working Group consisting of multiple firms of
> which our RabbitMQ team has been one since way back. I know the JPM
> guys extremely well and what they do with AMQP. File streaming has
> been deprecated from the protocol and not implemented. Qpid does not
> interoperate with RabbitMQ yet, though we are all working on that. We
> have a mailing list for RabbitMQ where you can look for comments about
> performance.
>
>
>>> Assuming a normal distribution of message size, let's (wlog) assume
>>> the claim is that RabbitMQ could manage one message per second on
>>> average in your friend's test, and therefore that Qpid was sending the
>>> equivalent of 1,000 5Mb messages per second. Because these are
>>> normally distributed ex hypothesi, that's around 5Gb per second. In
>>> which case astonishment must be expressed -- even if the network used
>>> was more than 1GigE.
>>
>> I believe that most messages were in the 0.1 to 1 Mb range and that gigabit
>> ethernet and other network acceleration technologies are employed.
>
> Again:
>
> Ceteris paribus... If all the messages are at the *lowest* end of your
> range, i.e. 100Kb then claiming 1,000x faster suggests the minimal
> claim is:
>
> * RabbitMQ sends one 100Kb message per second
> * Qpid sends 1,000 of these per second
>
> The latter implies a total network throughput of 1,000 * 100,000 * 8
> bits = 800 Mbit/sec. It is very hard to get above 700 Mbit/sec on
> 1GigE without saturating it. But let's say that Qpid is sending 1,000
> messages per second of 100Kb each, then the claim that RabbitMQ is
> 1,000 slower implies the claim that RabbitMQ can send at most 1x 100Kb
> message per second. People are very welcome to test this claim and
> draw their own conclusions.
>
>
>
>
>> The system transmits fully described business 'objects' with all necessary
>> context so that any clients who subscribe to the feed can take
>> all necessary processing action without any need for farther querying, data
>> look-up or large quantities of specialised messages. This isn't an RTP
>> system so fewer large messages makes more sense than floods of small
>> messages in this scenario. In previous iterations of the same system it was
>> found that there was little or no reuse of messages - most focussed message
>> queues had only a single subscriber. In the newer version, with much larger
>> more fully realised messages there are often dozens of subscribers. This
>> level of reuse has been very beneficial.
>
> I agree, which is why RabbitMQ does not copy by value for messages
> larger than 64 bytes.
>
> Cheers
>
> alexis
>
--
L.G. Meredith
Managing Partner
Biosimilarity LLC
806 55th St NE
Seattle, WA 98105
+1 206.650.3740
http://biosimilarity.blogspot.com
Rob,
On Sat, Mar 7, 2009 at 11:14 AM, Robert Lally wrote:
> I've enjoyed this discussion, but I too don't want to drag it off topic too
> far. So this will be my last comment here, I'll leave the last word to
> others.
> Final comments in-line.
Agreed. If others find this of interest I shall rejoin the conv.
A couple of clarifications below.
>> If there is an issue, often it can be fixed. Less often it reveals a
>> more general principle -- your large files example is a great instance
>> of that. AMQP is not designed for file streaming, not yet anyway, but
>> it could be adapted to that case by chunking (see for example Nanite
>> from Engineyard). So this means apple-apple broker comparisons are
>> hard. Ho, and indeed, hum.
>
> I'd have to disagree with you there. The broker in question was being run by
> the same organisation who envisioned, created and wrote AMQP and QPid to
> meet their messaging needs. So .. it was created for this. Exactly this.
AMQP is designed by a Working Group consisting of multiple firms of
which our RabbitMQ team has been one since way back. I know the JPM
guys extremely well and what they do with AMQP. File streaming has
been deprecated from the protocol and not implemented. Qpid does not
interoperate with RabbitMQ yet, though we are all working on that. We
have a mailing list for RabbitMQ where you can look for comments about
performance.
>> Assuming a normal distribution of message size, let's (wlog) assume
>> the claim is that RabbitMQ could manage one message per second on
>> average in your friend's test, and therefore that Qpid was sending the
>> equivalent of 1,000 5Mb messages per second. Because these are
>> normally distributed ex hypothesi, that's around 5Gb per second. In
>> which case astonishment must be expressed -- even if the network used
>> was more than 1GigE.
>
> I believe that most messages were in the 0.1 to 1 Mb range and that gigabit
> ethernet and other network acceleration technologies are employed.
Again:
Ceteris paribus... If all the messages are at the *lowest* end of your
range, i.e. 100Kb then claiming 1,000x faster suggests the minimal
claim is:
* RabbitMQ sends one 100Kb message per second
* Qpid sends 1,000 of these per second
The latter implies a total network throughput of 1,000 * 100,000 * 8
bits = 800 Mbit/sec. It is very hard to get above 700 Mbit/sec on
1GigE without saturating it. But let's say that Qpid is sending 1,000
messages per second of 100Kb each, then the claim that RabbitMQ is
1,000 slower implies the claim that RabbitMQ can send at most 1x 100Kb
message per second. People are very welcome to test this claim and
draw their own conclusions.
> The system transmits fully described business 'objects' with all necessary
> context so that any clients who subscribe to the feed can take
> all necessary processing action without any need for farther querying, data
> look-up or large quantities of specialised messages. This isn't an RTP
> system so fewer large messages makes more sense than floods of small
> messages in this scenario. In previous iterations of the same system it was
> found that there was little or no reuse of messages - most focussed message
> queues had only a single subscriber. In the newer version, with much larger
> more fully realised messages there are often dozens of subscribers. This
> level of reuse has been very beneficial.
I agree, which is why RabbitMQ does not copy by value for messages
larger than 64 bytes.
Cheers
alexis