On Thu, 08 May 1997 13:45:54 -0400 Brad Knowles <[log in to unmask]> said:
> The problem is that the Internet side of the equation is not the
>limiting factor here.
You asked me if I could do better on the Internet side of the equation,
not on the internal side. I know nothing about your internal mail system
and am not in a position to make comments about it.
> I'm confident it could handle twice or even three times that load
>without too much difficulty, if there weren't problems elsewhere.
*shrug* Likewise, our servers can handle more mail than they are handling
today (they are far from saturation). I don't see how this is relevant to
the discussion.
> Typically, we recieve about twice as much as we send. That's still a
>boatload of mail that we send, but since the expansion factor per
>outbound email message is low (on average, about two recipients per
>message), we don't get your economies of scale with hundred or thousands
>of recipients from a single message all served by the same set of MXes,
>etc....
I believe I did tell you that our average number of RCPT TO: is on the
order of 2. I don't doubt that if you look at a whole day's of activity,
somewhere you'll find one transaction that had hundreds or possibly even
thousands of recipients served through the same MX. But, on the average,
we have two recipients per transaction. Very few of our mailing lists
have thousands of recipients to begin with, let alone thousands at the
same host.
> Not exactly. Since virtually all mail that is sent is transmitted as
>soon as it is recived by the other end, it makes very little use of
>connection caching.
You are again thinking in sendmail terms. This simply does not apply to
LSMTP.
> Since you've got extremely large economies of scale due to large
>numbers of recipients typically served by a set of MXes,
Again I don't.
>and you've surely optimized your delivery to make maximum use of
>connection caching, the actual number of SMTP connections you make is
>actually probably quite small.
This is correct and is one of the reasons why LSMTP outperforms sendmail.
However you would also get this economy with a large workload comprised
of a lot of independent messages.
> Making SMTP connections is relatively cheap, since you can choose
>whether or not to have another queue runner fork off
Fork? What is that? :-)
> Since receiving mail is inherently interrupt-driven, and you can't
>force the other end to make use of connection caching (you can only sit
>there and wait to accept multiple messages per connection if the other
>end chooses to send them that way), what I want to count is the total
>number of SMTP connections you receive per day.
The total number of SMTP connections I receive per day is limited by the
intrinsic nature of my workload and not by the capacity of the server.
This is a bit like asking me how many concurrent processes LSMTP can
support, well it supports one, so sendmail "wins". What you really want
to know however is how much AOL-style mail LSMTP could pump in and out
and on what type of box, and the correct approach is to go through the
specs/bid/prototype process and not go on making meaningless comparisons
between software with completely different design and internal structure.
> I will pass on to the mail systems development management that you
>want to re-write the AOL Internet mail gateway system using LSMTP. If
>that's something we can do in parallel with our other efforts (and
>without a great deal of support required to teach you how the back end
>works), then they might be willing to listen.
We can look into that as well if it makes sense (I just don't have the
information to have an opinion on whether it does or not), but to
clarify, this is not what I was proposing. I was thinking along the lines
of having LSMTP front end your existing system. It would bear the brunt
of all the incoming connections, do basic filtering (reject mail that is
not for AOL users, or whatever your policies are), etc. Then it would
pass the mail back to your existing unix servers, using the exact number
of connections of your choice, which naturally would be cached. Your unix
servers would run with a controlled, tunable number of sendmail processes
and would not have to open/close connections all the time. You would keep
your more advanced filtering tools on sendmail. Likewise, outbound mail
would be sent to the LSMTP servers through a sendmail.cf rule and your
sendmails would not develop a queue. In other words, your unix systems
would work optimally and you could retain your existing interface to the
internal system, whatever this may be. This is clearly not the perfect
solution, but my goal was to minimize the amount of development work on
AOL's side. Basically, you don't need any new code at all, just a small
sendmail.cf change.
> He keeps remarking on occasion how much mail could be handled by VM
>SMTP, but he's changed his tune a bit since we found a bug in that code
>with regards to the way it handles MX RRs.
I'm going to make a very rough exaggeration here, but it wouldn't be
entirely wrong to say that LSMTP is a refined version of the VM SMTP
design, ported to affordable hardware and implemented in a more efficient
way (fully asynchronous I/O, etc). Back when Matt left IBM, the VM SMTP
outperformed sendmail easily on slower machines. Unfortunately
development more or less froze when Matt left and the software has not
been improved since then. Nevertheless, it did scale to 1.1M daily
deliveries (highest recorded figure). The VM SMTP is not implemented very
efficiently but is remarkably scalable, given the workloads that were
actually being processed at the time it was developed.
Eric
|