At 8:47 AM -0500 12/22/03, Stan Horwitz wrote
>We have a few lists on our Listserv 1.8d server (under Unix with qmail)
>that have something like 40,000 subscriptions with the subscriptions all
>pointing to the same email server. When email to these lists goes out
>(they're moderated lists), it stresses our email server complex. What I am
>wondering is if there is a way to control the size of the batched emails
>that get forwarded to our central email server on a per list basis.
>
>For example, say I have a list called EVERYONE-HERE and I have 40,000
>subscriptions on that list that all point to the server email.temple.edu,
>instead of having Listserv batch all 40K subscriptions at once, it would
>be nice to break up the batch into four groups of 10,000 subscriptions
>with a 15 minute interval between each group. Is this possible? If so,
>how?

Multiple smtp workers help this a bit.  We have 18 workers set up for
normal mail and one for our large lists.  Here are some lines from
our go.user file.

...
SMTP_FORWARD_1="18*lists.psu.edu;POOL=M-"
SMTP_FORWARD_2="lists.psu.edu;POOL=L"
SMTP_FORWARD_3="lists.psu.edu(22:30-06:00);POOL=P"
DEFAULT_LIST_POOL="M"
export SMTP_FORWARD_1 SMTP_FORWARD_2 SMTP_FORWARD_3 DEFAULT_LIST_POOL
...

Our large lists use a keyword of "Delivery-Pool= L" to direct traffic
to this worker.

L-Soft sent me the following prose that describes this facility:

SMTP worker pool support for prioritizing high volumes of outbound mail.

LISTSERV version 1.8d implements a new SMTP worker pool feature. SMTP worker
pools allow a site to run large lists on a server and also (with appropriate
hardware) to set up high-speed, low-volume delivery nodes for command responses
and the like.

SMTP worker pools sound quite complex but are very simple once you understand
what they actually do. LISTSERV recognizes 27 mail delivery pools, called A
through Z plus a default pool, called '-' (hyphen). By default, LISTSERV puts
all outgoing mail in the default pool, and the workers process mail for all the
pools. You can tell LISTSERV to put certain types of messages in different
pools, as follows:

The new "Delivery-Pool= x,y" list header keyword (must be set by the LISTSERV
maintainer) lets you define the delivery pool for postings to the list (x) and
for administrative messages related to the list (y). The latter includes
bounces, error monitoring reports, welcome messages, etc. It does NOT include
command responses of any kind (even if there is just one command and it is for
the list) or responses to postings (posting rejection, request for cookie,
forward to editor, etc), which are considered to be command replies and always
go to the default pool. If y is missing, x defines the pool for both postings
and administrative mail.

The new DEFAULT_LIST_POOL keyword, which defines site defaults for the above
(space separated).

The new POOL=x parameter for DISTRIBUTE.

The new DEFAULT_DIST_POOL keyword, which defines the default value for the
above.

If you use any of the above, LISTSERV will start assigning outgoing mail to the
various pools, while the default pool continues to receive command replies and
spontaneous messages that are not tied to any particular list (system errors,
etc). By default, all the workers are set to process mail from any of
the pools,
so it will not have any effect until you reconfigure the workers to only manage
some of the pools:

SMTP_FORWARD_n=blah blah;POOL=xyzt

(use POOL=* for the default behaviour where all pools are processed). So, each
individual worker can be configured to manage either a specific pool, a series
of pools, or all the pools (the latter is probably not useful). Note that you
must always provide at least one worker to process the default pool (POOL=-),
since you will always have messages in this pool. Obviously you don't need to
provide workers for pools you do not use.

In the current implementation, the pools are one-letter prefixes to
the xxx.MAIL
filename, rather than separate directories. This is a little bit less
efficient,
but it makes it a lot simpler for a LISTSERV maintainer to visually check the
contents of the queue. Here is a concrete example where we define pool L to be
large lists, X to be a high-priority express service for small lists, and M to
be the default pool for normal lists:

DEFAULT_LIST_POOL=M
SMTP_FORWARD_1=2*FAST.LSOFT.COM;FAST-BACKUP.LSOFT.COM;POOL=X-
SMTP_FORWARD_2=REGULAR1.LSOFT.COM;BACKUP1.LSOFT.COM;POOL=M
SMTP_FORWARD_3=REGULAR2.LSOFT.COM;BACKUP2.LSOFT.COM;POOL=M
SMTP_FORWARD_4=5*BULK.LSOFT.COM;BACKUP3.LSOFT.COM;POOL=L

A simpler setup, where we create a fast channel for command replies, would be:

DEFAULT_LIST_POOL=M
SMTP_FORWARD_1=2*FAST.LSOFT.COM;FAST-BACKUP.LSOFT.COM;POOL=-
SMTP_FORWARD_2=whatever we had before;POOL=M

This keeps list messages routed as before and routes command replies to a new,
high-speed, low-volume box or LSMTP instance.

Incidentally, assigning a worker to a single pool is faster because it can then
look for files called M*.MAIL and so forth. A worker that manages
multiple pools
needs to scan for *.MAIL and skip the files it does not service. Either way,
introducing pools adds some measure of overhead corresponding to the time spent
skipping over files which are not serviced. This would only be a problem in
practice if you had a bunch of workers configured to skip most of the files.

Worker pools can get quite interesting for specialized applications when
combined with another new worker feature, open time. You can now define that
certain servers should only be used during certain times of the day:

SMTP_FORWARD_n=BULKBOX.XYZ.COM(09:00-17:00,19:00-23:00);OTHERBOX.XYZ.COM

This defines a worker that will use BULKBOX during the times specified, and
OTHERBOX the rest of the time. A typical use is to bring bulk delivery boxes to
the rescue during peak interactive hours, while making sure not to
use them when
they are processing large postings. But you can also use it like this:

SMTP_FORWARD_n=SOMEBOX.XYZ.COM(23:00-06:00);POOL=W

This defines a worker that is only allowed to do useful work from 11pm to 6am.
If this is the only worker that services pool W, well, you won't be seeing much
activity in that pool outside these hours. Not probes, not welcome messages,
nothing (but commands can still be sent and the command reply will be
delivered). This could be a good way to shut off a very large list, or then to
cause a large mailing to start off at a very precise time. The workers are
accurate to the minute. Likewise, the workers will stop on a dime (after
finishing the current message) at the end of the opening interval. Currently
there is no weekday support; however if is really needed, it can be implemented
with a daily reboot.


--
Cheerfulness is contagious, but don't wait to catch it from others.
Be a "carrier".

Bill Verity - 814-865-4758  Fax: 814-863-7049
215A Computer Building - Information Technology Services, Penn State University
At the office - on my Mac, of course ;-)