On Thu, 08 May 1997 13:45:54 -0400 Brad Knowles <[log in to unmask]> said:

>    The problem  is that the  Internet side of  the equation is  not the
>limiting factor here.

You asked me if  I could do better on the Internet  side of the equation,
not on the internal side. I  know nothing about your internal mail system
and am not in a position to make comments about it.

>    I'm confident  it could handle twice  or even three times  that load
>without too much difficulty, if there weren't problems elsewhere.

*shrug* Likewise, our servers can handle more mail than they are handling
today (they are far from saturation). I don't see how this is relevant to
the discussion.

>    Typically, we recieve about twice as much as we send. That's still a
>boatload  of mail  that  we send,  but since  the  expansion factor  per
>outbound  email message  is low  (on average,  about two  recipients per
>message), we don't get your economies of scale with hundred or thousands
>of recipients from a single message all  served by the same set of MXes,
>etc....

I believe I  did tell you that our  average number of RCPT TO:  is on the
order of 2. I don't doubt that if  you look at a whole day's of activity,
somewhere you'll find one transaction  that had hundreds or possibly even
thousands of recipients served through the  same MX. But, on the average,
we have  two recipients per  transaction. Very  few of our  mailing lists
have thousands  of recipients to begin  with, let alone thousands  at the
same host.

>    Not exactly. Since virtually all mail that is sent is transmitted as
>soon as  it is recived  by the  other end, it  makes very little  use of
>connection caching.

You are again  thinking in sendmail terms. This simply  does not apply to
LSMTP.

>    Since you've  got extremely  large economies of  scale due  to large
>numbers of recipients typically served by a set of MXes,

Again I don't.

>and  you've  surely optimized  your  delivery  to  make maximum  use  of
>connection caching,  the actual number  of SMTP connections you  make is
>actually probably quite small.

This is correct and is one of the reasons why LSMTP outperforms sendmail.
However you would  also get this economy with a  large workload comprised
of a lot of independent messages.

>    Making SMTP  connections is relatively  cheap, since you  can choose
>whether or not to have another queue runner fork off

Fork? What is that? :-)

>    Since receiving  mail is inherently interrupt-driven,  and you can't
>force the other end to make use  of connection caching (you can only sit
>there and wait  to accept multiple messages per connection  if the other
>end chooses to  send them that way),  what I want to count  is the total
>number of SMTP connections you receive per day.

The total number of SMTP connections I  receive per day is limited by the
intrinsic nature  of my workload and  not by the capacity  of the server.
This is  a bit  like asking  me how many  concurrent processes  LSMTP can
support, well it  supports one, so sendmail "wins". What  you really want
to know however  is how much AOL-style  mail LSMTP could pump  in and out
and on what  type of box, and  the correct approach is to  go through the
specs/bid/prototype process and not  go on making meaningless comparisons
between software with completely different design and internal structure.

>    I will pass  on to the mail systems development  management that you
>want to  re-write the AOL Internet  mail gateway system using  LSMTP. If
>that's  something we  can do  in parallel  with our  other efforts  (and
>without a great deal  of support required to teach you  how the back end
>works), then they might be willing to listen.

We can look  into that as well if  it makes sense (I just  don't have the
information  to have  an  opinion on  whether  it does  or  not), but  to
clarify, this is not what I was proposing. I was thinking along the lines
of having LSMTP  front end your existing system. It  would bear the brunt
of all the incoming connections, do  basic filtering (reject mail that is
not for  AOL users, or  whatever your policies  are), etc. Then  it would
pass the mail back to your  existing unix servers, using the exact number
of connections of your choice, which naturally would be cached. Your unix
servers would run with a controlled, tunable number of sendmail processes
and would not have to open/close connections all the time. You would keep
your more advanced  filtering tools on sendmail.  Likewise, outbound mail
would be  sent to the LSMTP  servers through a sendmail.cf  rule and your
sendmails would  not develop a queue.  In other words, your  unix systems
would work optimally and you could  retain your existing interface to the
internal system,  whatever this may be.  This is clearly not  the perfect
solution, but my  goal was to minimize the amount  of development work on
AOL's side. Basically, you  don't need any new code at  all, just a small
sendmail.cf change.

>    He keeps remarking on occasion how  much mail could be handled by VM
>SMTP, but he's changed his tune a bit  since we found a bug in that code
>with regards to the way it handles MX RRs.

I'm going  to make  a very  rough exaggeration here,  but it  wouldn't be
entirely wrong  to say  that LSMTP is  a refined version  of the  VM SMTP
design, ported to affordable hardware and implemented in a more efficient
way (fully asynchronous  I/O, etc). Back when Matt left  IBM, the VM SMTP
outperformed   sendmail   easily   on  slower   machines.   Unfortunately
development more  or less froze when  Matt left and the  software has not
been  improved since  then.  Nevertheless,  it did  scale  to 1.1M  daily
deliveries (highest recorded figure). The VM SMTP is not implemented very
efficiently but  is remarkably  scalable, given  the workloads  that were
actually being processed at the time it was developed.

  Eric