Description of the changes for release 1.7f of LISTSERV
-------------------------------------------------------
April 2nd, 1993
**************
* Highlights *
**************
Version 1.7f introduces the following major enhancements:
- Most of DISTRIBUTE and the sections of incoming mail processing which
most depend on the size of the mailing list have been rewritten in
PASCAL, improving performance by a factor of 3 to 40 for DISTRIBUTE
jobs. A distribution to a test list with 8,500 recipients now takes a
mere 2.5 seconds of CPU time on a 3090-180x (over 100 seconds with
1.7e). Even small machines can now host lists with thousands of
recipients. Furthermore, with release 1.7f and LMail it is now more
economical, in most cases, to be on the DISTRIBUTE backbone than to
have the files duplicated upstream and delivered via RSCS.
- Significant productivity and ease-of-use enhancements have been made to
the mailing list functions: automatic digests and indexes, list topics,
dual-headers option for plain-VMSmail and PC users, easier command
submission for QuickMail and All-in-One users, easier retrieval of
indexed list postings through the generic 'listname-search-request'
mailbox, etc.
- New functions have been added to facilitate list maintenance: automatic
deletion of nonexistent accounts when the error report is in LMail
format (also supported by MX V3.2 and PMDF V4.2), list header keywords
parser to verify keyword syntax, subscription confirmation to prevent
addresses that cannot be replied to from being added to the list,
farewell message, subscription and posting filtering under list owner
control, more robust duplicate postings rejection, etc.
- A new driver has been written for the LISTS (list-of-lists) database,
with performance improvements of more than an order of magnitude.
Running the LISTS database under 1.7f requires less system resources
than simply managing the LIST GLOBAL summary with 1.7e. Since the
change is transparent to end users and since only a couple dozen sites
run the LISTS database, the technical aspects will not be described in
these release notes; refer to the LSTSRV-L archives for more
information.
- When used in conjunction with version 1.1e or higher of LMail, a 1.7f
backbone server may implement a "Global List Exchange" allowing users
to reach all public lists in the LISTSERV network as though they were
all located on the same machine. See the LSTSRV-L archives for a full
description.
- The database functions have been made 20-30% faster.
- Support for VM/ESA release 2 and CMS release 9 has been added. Given
the stability of current service levels of CMS release 6, the
"unsupported environment" warnings have been removed and current levels
of CMS release 6 are now supported.
****************
* Availability *
****************
Release 1.7 is available free of charge to sites which qualify under one
(or more) of the categories listed below:
a. Members of CREN or one of the CREN Cooperating Networks, excluding:
1. Members of the European Academic and Research Network (EARN).
2. Members whose computing facilities are located in an EEC country.
b. Members of EARN, CREN or a CREN Cooperating Network whose computing
facilities are located in a nordic country (Denmark, Finland, Iceland,
Norway, Sweden).
c. Members of EARN whose computing facilities are not located in an EEC
country and which have formally applied (in writing) for a license,
and complied with the appropriate EARN directives. International
organizations are not considered as belonging to the EEC, regardless
of where their computing facilities are physically located.
Release 1.6 licenses from EEC countries remain valid for release 1.6, but
will not be carried over to release 1.7.
*******************************
* Supported operating systems *
*******************************
Release 1.7 introduces the following changes to the list of supported
operating systems:
- (1.7a) Withdrawal of support for CMS release 3.
- (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA.
- (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes.
- (1.7b) Support for the Shared File System, with a few restrictions
concerning the setup of the A and D-disks and the default filepool
setting. See the corresponding section of the 1.7b release notes for
more information.
- (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes.
- (1.7e) Support for the Shared File System for the D-disk.
- (1.7f) Support for VM/ESA 1.2, CMS release 9 and CMS release 6.
Support for other versions of CP and CMS is unchanged from 1.6e.
**************************
* External Compatibility *
**************************
Release 1.7f is generally compatible with 1.7e, from the perspective of
the end-user (including list managers and maintainers), and with the
exception of the changes listed below. Release 1.7f introduces major
changes to the LISTSERV internals, making compatibility with locally
developed extensions or local modifications problematic. These changes
are briefly described in the next section, and only affect sites which
made alterations or additions to the LISTSERV code. Changes which affect
all LISTSERV sites are highlighted below; note that more details are
available, when appropriate, from the longer descriptive section of these
release notes.
1. For the reasons described on the NODMGT-L list, the hashing function
used by the Internet<->BITNET lookup code has been inverted. Some
sites whose ':internet.' tags were not globally unique may find that
their Internet and BITNET addresses are no longer recognized as being
the same and will have to correct their ':internet.' information.
Refer to the NODMGT-L and LSTOWN-L archives for more information.
2. The DISTRIBUTE command no longer removes duplicate recipients. If you
have applications which submit their own DISTRIBUTE jobs, you will
have to ensure that they do not supply the same mailbox twice as the
second and following occurrences will no longer be removed.
3. The daily GET quotas for sites using the default LSV$GETQ exit have
been changed from 20 files and 256k to 50 files and 2M. Sites with
good connectivity may want to consider further increases in these
quotas (MAXGET and MAXGETK configuration variables).
4. Lists operating with "Safe= Yes" (which is the default when the local
mailer is is Netdata-capable) have a BSMTP origin of
[log in to unmask] VM systems running XMAILER set the spool
fileid from the BSMTP origin, rather than from the RFC822 header, and
mail from safe lists will thus appear as 'owner-xx MAIL' in RDRLIST.
LMail sites are not affected. See the LSTOWN-L archives for a
comprehensive description of this change.
5. The behaviour of the LIST command was changed with respect to mailing
lists for which no dummy VM userid has been created. Previously
LISTSERV assumed the list had just been created and was not yet
operational, and displayed a warning to that effect. But with the
advent of LMail it is no longer necessary to create a dummy account
if the list is to carry only RFC822 mail from sites which have a
RFC822-compliant mailer (this excludes BITNET sites whose mail
software delivers directly via NJE and not through the registered
BITNET mailers). Such lists are operational, but only for RFC822
mail, and this is what the LIST command now reports.
Release 1.7f is to be installed directly on top of the base 1.7e code,
and includes all the fixes available from LISTSERV@SEARN for release 1.7e
(ie all the "17Ennnt FIX" files from filelist "LFIXES"), with the
exception of those containing replacement "recommended user exits", which
always have to be installed manually.
**************************
* Internal Compatibility *
**************************
Major changes have been made to LISTSERV internals. The most important
ones are briefly described below; refer to the LSTSRV-L archives for more
information.
1. Release 1.7f introduces PERMVARS FILE, a book-keeping file similar in
function to LASTING GLOBALV but without the 255-bytes restrictions.
Ultimately, all the information kept in LASTING GLOBALV will be moved
to this new PERMVARS FILE. As a rule of thumb, when the last REXX
program requiring access to the information is converted to PASCAL,
the code is changed to use the "PMV" routines rather than GLOBALV. As
this happens, LASTING GLOBALV entries will become obsolete and they
may or may not be migrated to PERMVARS FILE by the installation exit.
With release 1.7f, the "held list" information is migrated and copied
over to PERMVARS FILE as the new code installs itself.
2. The information formerly in CRC FILE and LUPDTIME FILE will now be
kept in PERMVARS FILE. The old files are erased and their contents
are not copied to PERMVARS FILE.
3. Digest checkpoint variables have been migrated, but not copied over,
to PERMVARS FILE. Site which had installed development fix 17E-002D
will see the next issue of each digest claim to be the "first ever".
This will only happen once.
4. While LSVDIST2 EXEC is still present, it is no longer the command
entry point for the DISTRIBUTE command. Any call to LSVDIST2 in local
applications must be replaced with calls to LSVDS4('DISTRIBUTE') -
see LSVXMAIL EXEC for an example.
The following REXX files have been become obsolete with release 1.7f and
are deleted during migration:
LSVBLC LSVDFOPT LSVHELD LSVPRSUM LSVXAM LSV00D
The following files have become obsolete with release 1.7f:
PEERS DISTSUM
CRC FILE
DS1WNG FILE
LUPDTIME FILE
**************************
* Compatibility Warnings *
**************************
This new section describes planned changes which may affect compatibility
(both external and internal). The release in which the change is planned
to be introduced is indicated, with the letter 'x' denoting an unknown
release of the specified major version (not necessarily posterior to the
last release explicitly mentioned). Note that conversion of more REXX
files to PREXX is assumed to take place with each new release and is not
indicated here.
- (1.7g) The DELETE * (NETWIDE command will be removed, as it does not
scale up and places an unacceptable burden on LISTSERV sites. Sites
running LMail, MX V3.2 or PMDF V4.2 (or higher) need not submit netwide
delete jobs, as the delivery errors generated by their mailers will be
automatically processed by LISTSERV (starting from 1.7f).
- (1.7x) The internal format of FILELIST/FILEID and attendant files such
as AFDLIST FILE will change dramatically. Applications which alter them
directly will no longer work.
- (1.7x) The internal format of LIST and attendant files (CACHE, STATS,
et al) will change dramatically. Applications which alter them directly
will no longer work.
- (1.7x) LSVFRD1, LSVFWR1 and RXLSVUHF will be eventually removed. You
should avoid using them.
********************************
* Minor fixes and enhancements *
********************************
This section has been removed for this release, due to significant
changes made to key functions such as incoming mail files processing,
DISTRIBUTE, the list keyword processor, etc. About 10,000 lines of new
PASCAL code have been written to replace existing internals. As a rule of
thumb, and as explained in the 1.7c release notes, it is not possible to
generate the "Minor fixes and enhancements" section for code which has
been redesigned and rewritten from scratch in PASCAL.
REMINDER: Manual installation is not supported for release 1.7, and the
code that implemented it has been removed with version 1.7e. Automatic
installation with local password validation must be used instead. Sites
which insist on installing manually should note that shipments contain an
installation pre- and post-processing exit. If that exit is not executed,
or if it is executed in a different order or environment than during
automatic installation, serious problems may occur.
******************************************************
* Miscellaneous usability/human factors enhancements *
******************************************************
- The acknowledgements LISTSERV sends to confirm that a message has been
posted now include the date and subject line.
- LISTSERV will now accept up to 10 invalid commands in a mail message
before command processing stops. This makes it easier for users of
systems such as QuickMail, All-in-one, PROFS, OV/VM or any other mail
system which inserts a letterhead-like header in the message body
before the text supplied by the user to send commands to LISTSERV.
Previously it was necessary for the user to either remove the
letterhead lines manually before sending the message, or instruct
LISTSERV to ignore them by typing '// job' before the first command.
While this still works of course, the users can now send the commands
directly and simply ignore the error messages generated by LISTSERV as
it attempts to process the letterhead. Note that these letterheads are
usually in the local language and nearly impossible to identify.
- Unsolicited mail notices (formerly with "Subject: Message") now include
the beginning of the first message line in the subject field. This is
mostly useful for LISTSERV maintainers, who may receive a lot of these
messages in case of system problem - repeated occurrences of one or
more unexpected problems.
- Subscription renewal confirmation requests no longer have a Reply-To:
pointing to the list owner. This makes it possible for users to reply
to the message and type the CONFIRM command.
- Welcome messages are now passed back to the list owner if they cannot
be delivered because the subscriber's address was invalid.
- A "farewell message" can be defined for a mailing list, using the same
procedure as for the welcome message but with a filetype of FAREWELL.
The farewell message is sent to any user who signs off the list by
himself, but not to users who are removed by the list owner, as it is
expected to contain a message to users who chose to leave on their own,
rather than to misbehaving users. The first line of this farewell
message must be a 'Subject:' field.
- Double quotes surrounding a user name (usually entered by mistake) are
now internally removed, to avoid addresses of the type:
From: "\"John Smith\"" <[log in to unmask]>
which are perfectly correct but which many mail systems cannot handle.
- The 'all-request' mailbox has been provided as a convenient means for
the LISTSERV maintainer to send important warnings (unexpected
downtime, changes in server configuration, etc) to all list owners.
- The "Sender=" list header keyword has been extended to allow the
specification of a second mailbox, to be used for IETF headers. The
first value is used, as before, for non-IETF headers and is expected to
contain the name and address of the list, or the keywords LIST or NONE.
The second mailbox is used for IETF headers; if it is omitted, the
generic 'owner-listname' mailbox is substituted. Example:
Sender= "Test list <[log in to unmask]>","[log in to unmask]"
- When delivering messages to a mailing list, LISTSERV scans the mail
header for the name of the poster. But some user interfaces do not
provide any personal name - just an e-mail address, and often one that
does not have much in common with the person's real name. In such cases
LISTSERV will now check if the poster is subscribed to the list and, if
so, it will use the name registered in the subscription.
**************************************
* Usability: Change to short headers *
**************************************
The 'Organization:' field is now included in short headers (SHORTHDR and
SHORTBSMTP options), as it provides useful information that is generally
desired by the human reader, and does not require any special user
interface to exploit.
The criteria for adding fields to the short headers are that they should
be useful to non-technical subscribers with unsophisticated user
interfaces, such as plain VMSmail, PEEK, and so on. By definition, fields
which are useful only with a specialized user interface, such as MIME
fields, do not belong in the short header. Full headers (FULLHDR and
FULLBSMTP) contain all fields, including MIME fields, and the list owner
selects the default header type he finds most appropriate for his user
population. This has been the case from day one. Every month, someone or
other states the opposite on a public forum, usually to try to convince
people not to choose LISTSERV to run their mailing lists. You can help
preventing this misinformation from being spread by posting a copy of
this paragraph in answer to any statement that LISTSERV and MIME are not
compatible. Thank you.
*********************************
* Usability: New DUALHDR option *
*********************************
Quite a lot of non-technical users are relying on non-RFC822 user
interfaces for reading their mail. Quite often these user interfaces are
user-friendly, quality implementations of a proprietary mail protocol
which the users are proficient with, but which happens not to lend itself
to bidirectional mapping to RFC822. The users may have a good reason for
using this particular program, and they complain that it is not always
clear what list the postings come from, or who posted them. Other users
have very primitive mail programs which do not preserve the original
RFC822 header and may not even have a "message subject" concept. The user
knows which list the message came from, but not who posted it, making
private replies impossible.
To solve this problem, a new type of headers has been added: DUALHDR
(minimum abbreviation: DUAL). Dual headers are regular short (SHORTBSMTP)
headers followed by a second header inside the message body. This second
header shows what list the message is coming from ('Sender:'), the name
and address of the person who posted it ('Poster:'), the poster's
organization, if present, and the message subject. The date is not shown
because even the most primitive mail programs appear to supply a usable
message date.
**************************************
* Usability: New list keyword parser *
**************************************
The list keyword parser has been enhanced to perform basic syntax
checking, to be more tolerant in its handling of commas and blanks, and
to provide a permanent solution to the problem of lowercase tokens in
keywords whose value is normally translated to upper case.
Whenever a list is updated via the PUT command, the syntax of the various
keywords is checked and a report is mailed back to the command originator
if errors were found. Serious errors prevent the list from being stored,
while the list is updated if only warnings are issued. The keywords are
checked individually for proper syntax and consistency within the keyword
itself (for instance, "Digest= No,A1" would be rejected, even though both
sub-values are syntactically valid). For now, "global" consistency
(across the various keywords) is not checked; this might be added in a
future version if the current level of checking turns out to be
insufficient. Note also that a few keywords are not checked at all,
either because it is not possible or because the programming effort it
would require is not worth the additional safety; for instance, it would
be possible to check that the "Newsgroup=" keyword respects the xx.yy.zz
syntax, but without checking that the newsgroup actually exists this
would not be very useful. Here is a sample error report:
-------------------------------------------------------------------------
The following problems have been detected in the list header:
> Default-Options= XYZ,...
Error: Invalid delivery option - "XYZ".
> NATEBOOK= NO
Warning: There is no keyword by that name - check the spelling.
> Filter= YES,...
Error: Invalid parameter value. Choose one from the following (without the
quotes): "Also", "Only" or "Safe".
> Digest= ...,X1/401,...
Error: Invalid directory specification - incorrect syntax or directory does not
exist.
> Sizelim= XYZ
Error: Value must be numeric.
PUT operation rejected, old list remains unchanged.
-------------------------------------------------------------------------
The new parser ignores blanks after or before commas, removing one of the
most frequent causes for mistakes ("Reply-To= List, Respect" becomes
valid with 1.7f). Furthermore, any text enclosed in double quotation
marks stays in mixed case. Unlike version 1.7e, which supported this
mechanism only if the first character of the keyword was a double-quote,
with 1.7f you can use this anywhere in the keyword text. For instance,
* Owner= ERIC@SEARN,"eric"@sunic.sunet.se
defines ERIC@SEARN and [log in to unmask] as list owners.
***********************************************
* Usability/Productivity: Digests and indexes *
***********************************************
Release 1.7f introduces an automatic digestification function allowing
subscribers who do not have the time to read large numbers of messages as
they arrive to subscribe to a digestified or indexed version of the list.
The list owner decides whether digests are available or not, the
frequency at which they are issued and the day of week or time of day
when the digest should be distributed. The digests conform to RFC1153
with an acceptable deviation from the recommended subject line (verified
with the RFC author).
Digests are larger messages containing all the postings made by list
subscribers over a certain period of time. Unlike real-world digests,
LISTSERV digests are not edited; what you see is exactly what was posted
to the list. The only difference is that you get all the messages for a
given day, week or month in a single batch. This is mostly useful if you
are just "listening in" to the list and prefer to read the postings at
your leisure. Digests are kept separately from list archives and can be
made available for mailing lists which do not archive postings (ie which
run with "Notebook= No").
Indexes, on the other hand, only provide a few lines of information for
each posting: date and time, number of lines, name and address of poster,
subject. The actual text is not included. You select just the messages
you are interested in, and order them from the server. This is useful for
mailing lists where most messages really don't interest you at all, or as
an alternative to SET NOMAIL: when you come back from vacations, you can
quickly order the messages you are most interested in. Note that, since
indexes are not useful without the ability to order a copy of the
messages you do want to read, they are not made available unless the list
is archived and digests are enabled.
Users sign up for digestified rather than immediate delivery with 'SET
listname DIGests', while indexes are selected with 'SET listname INDex'.
These two new options are alternatives to MAIL and NOMAIL. When switching
around between these delivery options, users will observe the following
behaviour (digests will be assumed to be daily for the sake of clarity):
- When switching to NOMAIL: delivery stops immediately. The day's digest
is not sent, as the user is assumed to request immediate termination of
traffic from the list.
- When switching from any option to DIGESTS or INDEX: mail delivery stops
immediately, and the first index or digest may contain some items the
user has already seen (if switching from MAIL to DIGESTS/INDEX). This
is because the digests and indexes are global to the list - they are
the same for everyone, just like regular issues of newspapers.
- When switching from DIGESTS or INDEX to MAIL, the current, unfinished
digest or index is immediately mailed to the user. New messages are
delivered normally, as they arrive. Thus, a "trick" to get a copy of
the current digest is to switch to MAIL and then back to DIGESTS. You
can send both commands in the same mail message to make sure they are
executed together.
The list owner controls the availability and frequency of digests through
the new "Digest=" list header keyword, which defaults to "Digest= No" for
lists without an archive and "Digest= Yes,Same,Daily" for archived lists.
Again, it is not necessary for the list to be archived to keep a digest;
LISTSERV just attempts to avoid having to store large amounts of digest
data on its A-disk for lists which, lacking a "Notebook= Yes,xxx"
keyword, do not specify any suitable filemode for the digest data.
Conversely, having daily as the default frequency keeps the additional
cost in disk space to a minimum.
The syntax of the keyword is "Digest= Yes,where,frequency,when,max" when
digests are enabled, or then "Digest= No". The second parameter is a disk
or directory specification, just as with the "Notebook=" keyword, or
"Same", which means that the digest must be stored on the same disk as
the list archives. The third parameter is either "Daily" (the default),
"Weekly" or "Monthly". The third parameter is optional and specifies when
the digest is to be actually distributed. For daily digests, specify
'hh:ss' or just 'hh' in the usual 00-23 scale (24 is also accepted for
midnight). For weekly digests, specify a weekday such as "Tuesday". For
monthly digests, you may specify a number from 1 to 31 corresponding to
the day of the month when the digest will be distributed, although this
is not recommended. The purpose here is to make it possible for digests
to be issued at mid-month rather than on the first of the month - if you
code a number larger than 28, you may not get a digest every month.
Finally, the last and optional parameter takes the form "Size(nnnn)" and
specifies the maximum size the digest is allowed to reach before a
"special issue" is cut. Bear in mind that most unix systems do not accept
messages larger than 100 kilobytes, so values larger than 1500 should be
avoided. The default is to have virtually no limit - 10,000 lines.
The list owner must take special care when disabling digests for a list,
as LISTSERV does not presently have any facility which would allow it to
alter subscription options automatically on the basis of changes to the
list header. Subscribers who had opted for digests would continue not to
receive mail as it arrives, but will not get the digests either. The best
way to solve this problem is to announce the change long enough in
advance, so that people can switch back before digests are suspended. The
reason nothing has been done to remove this limitation is that it is not
expected to be a frequent condition. Daily digests take up very little
disk space and there is no reason to disable them for a typical list.
*****************************************
* Usability: Changes to the SET command *
*****************************************
As the number of subscription options increases (especially with the
addition of digests/indexes and topics, whose behaviour depends on the
list configuration), so do the potential for confusion and the risk of
having activated the wrong options by mistake. Another concern for
"secure" lists ("Validate= All commands") is that the new digest and
topics options are likely to cause subscribers to issue a lot more SET
commands, which must all be confirmed by the list owner.
To address both problems, the SET command has been modified to return a
status report to the user after successful completion. This status report
shows all the options which are currently active (in the same format as a
'QUERY listname' command), and is always sent by mail so that it can be
conveniently filed for future reference. This may be annoying to power
users who know exactly what they are doing, but the benefits to novice
users outweigh this small inconvenience. Finally, since the user is
always informed of the change, it is no longer necessary to require list
owner confirmation for "Validate= All" lists, which otherwise could have
become unmanageable.
*************************************************************
* Usability: Miscellaneous changes to the SUBSCRIBE command *
*************************************************************
When a subscription request is sent via mail and the user did not supply
his name (that is, when LISTSERV is sent just 'SUBSCRIBE XYZ-L'), the
name will be taken from the mail header. If the mail header does not
include a personal name (as in 'From: [log in to unmask]), LISTSERV will
search its signup file as before and possibly reject the subscription.
But as long as your mail program supplies a personal name, you will be
able to subscribe to mailing lists without having to retype your name
every time.
Novice VMS users often supply all-caps personal names because they did
not know they had to put double quotes around their name when typing the
SEND command; there is a similar problem with users on the LISTSERV host
who use the MSG command to communicate with LISTSERV because they learnt
it from a computer expert who demonstrated with a command where case
doesn't matter, and did not realize that the user would not know to use
TELL when case does matter. In both cases the warning LISTSERV used to
issue was not very helpful, as it only pointed out the problem without
explaining how to solve it. On the other hand, LISTSERV has no way to
know why exactly the name arrived in upper case and a description of all
the possible operating system specific problems would be inappropriate
and confusing (many users don't even know what kind of system they are
using). From now on, LISTSERV simply converts the name to mixed case,
inserting capitals where appropriate, and completes the subscription
normally. A similar change was made to the ADD command.
************************************************************
* Productivity: Optional subscription confirmation feature *
************************************************************
One problem plaguing some mailing lists is one-way or non-repliable
addresses. Most of the time this is due to a small number of faulty
gateways, which one can just ban from the list until their maintainers
address the problem. But sometimes the very topic of the list is bound to
attract a large number of people connected through such gateways, and the
amount of domains to filter out becomes unmanageable. This is
particularly problematic when the gateway keeps quiet for a few days, and
suddenly returns hundreds of delivery errors with a promise to keep doing
so every day for another 6 days.
This problem can be avoided by probing the return address before
accepting the subscription. If the address cannot be replied to, only one
delivery error will be bounced to the list owner (perhaps for several
days, but there will be a single undeliverable message). With a gateway
that waits 3 days before sending its first delivery error, however, there
can be hundreds of messages "in the pipe" if the subscription is accepted
directly.
This probing is activated by specifying "Subscription= Open,Confirm" in
the list header. When a user attempts to subscribe to the list, he is
mailed a confirmation request with a randomly generated "confirmation
code". The procedure for confirming the subscription is simple - you just
reply to the message, type "ok", and send. If the return address does not
work, the request will never be confirmed and the user will not be
subscribed. And since the user cannot guess the confirmation code he will
be assigned, he cannot "cheat" by sending the confirmation along with his
request.
The subscription request also expires after a certain amount of time, as
determined by the "Confirm-Delay=" keyword (the default is 48h). Many
unreliable gateways have a turnaround time of several days, and this is
another way to filter them: if the confirmation delay is low enough, they
will never manage to subscribe and you will not have to put up with
gateways which take a week to realize that the subscriber's account has
expired and return a week worth's of delivery errors. On the other hand,
if you do want to let these people in, you will have to increase the
confirmation delay to a week or so (1 week = 168h).
************************************************************
* Productivity/Reliability: LMail support and "safe" lists *
************************************************************
Both LMail and current versions of XMAILER (R2.10 or higher) provide
support for the administrative 'owner-listname' and 'listname-request'
mailboxes introduced in release 1.7b. Previously this support was only
available via a patch to XMAILER R2.08 which only a handful of sites had
applied. Now that over 50% of LISTSERV sites (and 70% of backbone sites)
are running LMail or XMAILER R2.10, LISTSERV is finally in a position to
use the standard Internet conventions for error return addresses to
decrease the risk of mailing loops. Note again that without one of the
mailers mentioned above there is no way for LISTSERV to intercept
delivery errors sent to the 'owner-listname' mailboxes and that these
errors are likely to be lost, without anyone ever being notified. You
must not activate the use of these mailboxes unless your mailer supports
them.
For sites which do run a mailer that supports the administrative owner-*
mailboxes, a new list header keyword, "Safe= Yes/No", controls the e-mail
address LISTSERV places in the BSMTP MAIL FROM: field, which is where
well-behaved mailers will return delivery errors. With "Safe= No", these
errors are sent to the list address as before, hopefully to be
intercepted by the loop detector and passed on to the list owner. With
"Safe= Yes", the error address is set to 'owner-listname', and delivery
errors sent to that address are passed on to the list owner without the
risk of creating a mailing loop. The default is "Safe= Yes" if you are
running LMail or XMAILER R2.10 with NDMAIL=1 in LOCAL SYSVARS. Again, you
must not attempt to use "Safe= Yes" if your mailer does not support the
owner-* mailboxes.
IMPORTANT: The use of "Safe= Yes" does not guarantee that all errors will
go to the 'owner-listname' mailbox. Unfortunately, there are many non
compliant mailers which will continue to send the error back to the list
(usually because it is listed in the 'Reply-To:' or 'Sender:' field). The
use of the "Safe= Yes" option significantly decreases the potential for
mailing loops, but not enough to actually code "Loopcheck= No", unless
you are sure that all your subscribers have compliant mailers.
Another productivity change which has already been announced on various
mailing lists is support for automatic deletion of users whose account
has expired or whose system has permanently disconnected. When the
delivery error is generated by LMail (any version), MX V3.2 or higher, or
PMDF V4.2 or higher, which all implement the same delivery error format,
LISTSERV may be able to automatically process the delivery error and take
action based on the value of the "Auto-Delete=" list header keyword.
If you have coded "Auto-Delete= No", or if the delivery error is not in
LMail format and LISTSERV cannot understand it, LISTSERV simply passes it
to the list owner as before. Otherwise LISTSERV processes the message
automatically. The algorithm may be refined in a future version, but at
present the following steps are taken:
- LISTSERV looks for "permanent" errors (no such user, no such host, and
so on). If the failing recipients are subscribed to the list, LISTSERV
removes them and notifies you. No action is required from you.
- If there are permanent errors for users LISTSERV could not find on the
list (for instance because the account subscribed to the list is a
totally different one which forwards mail to a dead account), or if
there are only "temporary" errors (host unreachable for 3 days, system
error, disk quota exceeded, and so on), LISTSERV further examines the
"Auto-Delete=" keyword and passes the message to the list owner unless
you specified "Auto-Delete= Yes,Full-Auto".
- In a future version, LISTSERV might keep track of temporary errors when
the list is running in full-auto mode and remove users after a certain
amount of temporary errors. For now, it simply ignores them.
- When running in full-auto mode, LISTSERV never passes back a delivery
error unless it took action on it. This means that certain errors may
remain unsolved, as LISTSERV presently ignores temporary errors and
some of them are virtually permanent (if the owner of the account has
left but for some reason his account was not closed, his disk quota is
bound to remain exceeded until someone takes action). Full-auto mode is
to be used only when you positively do not have the time to handle the
delivery errors LISTSERV is sending you every day.
The default value is "Auto-Delete= No" for lists with "Validate= All" and
"Auto-Delete= Yes,Semi-Auto" for other lists.
*************************************************
* Usability: Subscription and posting filtering *
*************************************************
In order to give list owners better control over troublesome users or
gateways, and to remove the generic ban on some traditionally reserved
usernames such as 'root' or 'postmaster', a new filtering system has been
added to supplement the SERVE OFF mechanism (which is restricted to the
LISTSERV maintainer and affects all lists on the server).
This is implemented via a new list header keyword, "Filter=", which is
checked when a user attempts to post or subscribe to a list (but not when
the list owner issues an ADD command). The first word of this keyword is
either "Only", "Also" or "Safe", and it is followed by a list of patterns
such as 'X400MAIL@*' or '*@*.XYZ.EDU' (without the quotes). If "Also" is
specified, your filter is used in addition to the standard LISTSERV
filter; this is useful to register additional looping mailers, to prevent
users behind broken gateways from subscribing until the problem is
addressed, or to ban anonymous posters. LISTSERV has two built-in
filters: a "minimal" one, which is used for safe lists, and a "safe" one
(similar to the one used by 1.7e) which is used for lists running with
"Safe= No". That is, the unsafe lists need a safe filter to avoid mailing
loops; safe lists only need the minimal filter, but can be made even
safer by selecting "Filter= Safe". This, however, prevents usernames such
as 'root' or 'postmaster' from posting to the list, because they are
included in the safe filter.
If "Filter= Only" is used, the addresses you specify are the only ones
which LISTSERV prevents from posting to the list; you should not use this
option unless you also code "Safe= Yes", and even then you will want to
ask your LISTSERV maintainer for permission. In fact, this option has
been added mostly for LISTSERV maintainers with very specific problems to
solve. The minimal filter is very small and you should never need to
override it.
Messages sent to the LISTSERV userid for execution are always checked
with the minimal filter, as people with userids such as 'root' would
otherwise not be allowed to subscribe to lists which were set up to allow
them. This may cause some extra traffic while LISTSERV and various
gateways engage in ping-pong wars, but after 10 iterations the gateways
will be served off and this will stop.
Note that LISTSERV extracts as many e-mail addresses as it can from the
userid being checked and runs them all through the filter. For instance
if your list receives mail from [log in to unmask], LISTSERV
will check [log in to unmask], [log in to unmask] and
'mailer@searn' (via the 'internet.' tag). Generally speaking, userids
intercepted by the 1.7e filter will not pass the safe 1.7f filter.
**************************************
* Usability: Support for list topics *
**************************************
List topics provide a way to run a mailing list (preferrably moderated)
where several sub-topics are being discussed in parallel but some
subscribers are only interested in a subset of the topics. For instance,
a working group might have general discussions, decisions, and messages
related to meetings. People who cannot attend the meetings can then opt
out of last calls for hotel reservations and discussions about seafood
restaurants, whereas people who have no time to follow the discussions
can elect to get just the decisions. At any rate, such a compartmented
list requires a certain discipline in order to be successful, as the
posters must label their messages to indicate which topic(s) they belong
to.
Through the "Topics=" keyword, the list owner can define up to 11 topics
for the list. For instance, the list owner could code:
Topics= News,Benchmarks,Meetings,Beta-tests
********************************************************
* WARNING - YOU MUST NEVER REORDER THE TOPICS= KEYWORD *
********************************************************
To save disk space, LISTSERV remembers which topics users have selected
through their ordering in the "Topics=" keyword. That is, "News" is
"topic number 1" for LISTSERV, "Benchmarks" is "topic number 2", and so
on. This means you can change the name of a topic without requiring users
to alter their subscriptions (for instance, you could decide that "Tests"
is a better name than "Beta-tests" and just make the change). However,
you must never change the order of the topics in the "Topics=" keyword.
If you want to remove a topic, replace it with a comma. For instance, to
remove the "Meetings" topic, you would change the keyword to:
Topics= News,Benchmarks,,Beta-tests
This restriction might be removed in a future release.
Topic names can contain any character except space, colon and comma; the
use of double quotes or equal signs is discouraged, as they require
special attention when coding list header keywords. In addition, topic
names may not start with a plus or minus sign, and the words ALL, NONE,
RE, OTHER and OTHERS are reserved.
Posters label their messages through the subject field. LISTSERV first
skips any possible sequence of 'Re:' keywords, and takes anything to the
left of a colon as a list of topics, separated by commas. The posting is
considered to belong to all the topics listed before the colon. If none
of these topics is valid for the list, it is classified in a special,
12th topic, "Other". If some of the topics are valid but others are
undefined, the invalid ones are ignored. At any rate the subject field is
left unchanged. Here is an example:
Subject: Benchmarks,News: Benchmarks for XYZ now available!
Messages which should be read by everyone can be posted to the special
topic "All". Topic names can be shortened to any unambiguous
abbreviation. In our example, "Be" is ambiguous because it could be
either "Beta-tests" or "Benchmarks", but "Bench" is acceptable.
Subscribers select the topics they wish to receive with the SET command.
The syntax is 'SET listname TOPICS: xxx' where 'xxx' can be:
- A list of all the topics the user wishes to receive. In that case these
topics replace any other topics the user may have subscribed to before.
For instance, after 'SET XYZ-L TOPICS: NEWS BENCH', the user will
receive news and benchmarks, and nothing else.
- Updates to the list of topics the user currently receives. A plus sign
indicates a topic that should be added, a minus sign requests the
removal of a topic. For instance, 'SET XYZ-L TOPICS: +NEWS -BENCH' adds
news and removes benchmarks. If a topic name is given without a + or -
sign, + is assumed: 'SET XYZ-L TOPICS: +NEWS BENCH' adds news and
benchmarks. The first topic name must have the plus sign to show that
this is an addition, and not a replacement.
- A combination of the above, mostly useful to enable all but a few
topics: 'SET XYZ-L TOPICS: ALL -MEETINGS'.
The colon after the keyword TOPICS: is optional, and TOPICS= is also
accepted. Do not forget to include the special OTHER topic if you want to
receive general discussions which were not labelled properly. On the
other hand, if you only want to receive properly labelled messages you
should not include it. ALL does include OTHER.
A "Default-Topics=" list header keyword is available to define the
initial topics for new subscribers. The syntax is the same as for the SET
command, except that topic names are separated by commas in the usual
fashion and that the first topic may not start with a + or - sign (there
is nothing to add to, as this is a new subscription). This is similar to
"Default-Options=" in that it does not affect existing subscribers. Users
who signed up before topics were enabled on the list are automatically
subscribed to all topics.
Finally, it is important to note that topics are active only when your
subscription is set to MAIL. Digests are indexes always contain all the
postings that were made, because the same digest is prepared and sent to
all the subscribers. Depending on the success of topics support, this
restriction might be lifted in a future release.
************************************
* Serviceability: New LSV$DNT exit *
************************************
A new exit, LSV$DNT EXEC, has been added to make customization of DOMAIN
NAMES-based routing more convenient. This exit is similar to LMail's
LSM$DNT, but note that the format of the tables is different: you cannot
use your LSM$DNT EXEC directly for LISTSERV.
After rebuilding its DOMAIN NAMESUM file, LISTSERV looks for LSV$DNT EXEC
on any accessed disk. If it finds it, it invokes it and the exit can then
modify the DOMAIN NAMESUM file as it sees fit. With release 1.7f the
various entries in the file no longer need to be sorted from longest to
shortest domain name. For the maintainer's convenience, the code that
sorts this file has not been removed - it makes it easier to see how mail
for a given domain is going to be routed, using ALL/.XX (just the top
level domain) under XEDIT. The LSV$DNT exit may insert entries out of
sort order, but in that case that is what the final file will look like.
LISTSERV does not sort the file after calling LSV$DNT because the purpose
of this exit is to ultimately control the final contents of the file.
*****************************************
* Reliability: PASCAL region monitoring *
*****************************************
Servers with a large workload and/or many large mailing lists may report
unexpected "Machine storage exhausted" REXX errors even though the
virtual storage size is 12M or more. This problem, which is difficult to
avoid, is due to the fact that LISTSERV is being rewritten in PASCAL
(and, indeed, the "core" functions of LISTSERV are now mostly in that
language) while there are still a number of storage-hungry commands
written in REXX. The PASCAL compiler, which like most IBM compilers was
designed for MVS and made to work on VM using OS simulation, works on the
assumption that it is the only application running in the virtual
machine. While it will not grab storage it has no use for, it never
releases storage it no longer needs; instead, it keeps it around for the
next time a PASCAL routine requests some storage. This unused storage is
not available to REXX and may cause "Machine storage exhausted" errors
(or error 1001 from LSVFILER). Rebooting usually solves the problem.
Release 1.7f introduces a "PASCAL region watchdog" which monitors the
amount of storage the PASCAL run-time environment has allocated (use the
SHOW STORAGE command to check out on that value). During end-of-job
cleanup, LISTSERV resets the level-2 PASCAL run-time environment if this
value is unreasonably high. This RTE corresponds to the PASCAL
subroutines called by REXX command processors and by the REXX code that
processes new postings to mailing lists (in an all-PASCAL implementation,
this RTE would not be present). In most cases this is sufficient to
recover locked storage. However, storage held by the level-1 RTE (the one
that corresponds to LISTSERV's main program) cannot be reclaimed without
rebooting the server. SHOW STORAGE has no way to tell the level-1 and
level-2 allocations apart and can only report their sum.
|
