Description of the changes from release 1.5o to 1.6a of LISTSERV
--------------------------------------------------------------
June 1st, 1989
***********
* Warning *
***********
The planned availability of release 1.6a of LISTSERV is the week of June
19th-23rd. The following description applies to the beta-test programme,
which has been started on May 31st; minor design changes or additions of
new functions can be made before the final version is released. In order
to save network bandwidth, this document will probably not be re-posted
in its final form to the LSTSRV-M list; instead, a smaller file listing
the changes made to it will be sent, so that, in order to obtain the
current description of the changes from release 1.5o to 1.6a, you should
search the LSTSRV-M archives for all items with a subject starting with
"Changes from release 1.5o to 1.6a".
****************
* Requirements *
****************
LISTSERV release 1.6a is supported on any hardware/operating system
configuration supporting release 1.5o. In particular, support for VM/XA
SP (in 370 mode) is available in the "base" version of 1.6a (no "fix"
shipment required).
Release 1.6 of LISTSERV is available only to BITNET and NetNorth sites,
for political reasons. EARN sites now receive support for the EARN
version of LISTSERV from Turgut Kalfaoglu <TURGUT@TREARN>, to whom EARN
users should address any and all inquiries regarding LISTSERV.
*****************
* Compatibility *
*****************
Release 1.6a is fully compatible with 1.5o, with the exception of the
behavioural changes noted under the heading "Use of BSMTP to reduce the
load on Internet gateways" and of the removal of the DIST1 command, which
is now handled by the DIST2 processor. The syntax of the DIST2 command is
a superset of that of DIST1, and the results produced are compatible with
those produced by DIST1; however, some of the messages are different, and
the selected distribution path might not be the same.
Release 1.6a includes all the fixes contained in the 'FIX15O1' service
shipment. It can be installed directly on top of the base 1.5o version.
LSVDIST and LSVBITWT have been made obsolete by release 1.6a; LSVBITWT
MODULE and ASSEMBLE have consequently been removed from TOOLS FILELIST.
The following files can be erased from LISTSERV's disks once the
migration is complete: LSVBITWT ASSEMBLE, LSVBITWT MODULE, LSVBITWT
HELPCMS, LSVDIST EXEC and LSVDISTX EXEC.
***************************
* Important fix: LSVCARDR *
***************************
The release 1.5o version of LSVCARDR had a bug which could cause LISTSERV
to enter a disabled loop or die with an addressing or protection
exception. This was the most frequent cause of LISTSERV crashes, with the
well-known storage fragmentation problem causing the REXX interpreter to
stop with a "Machine storage exhausted" message, and about which little
or nothing can be done. This bug, which was triggered by a transient
abnormal behaviour of CP release 5, has been fixed in 1.6a.
*************************************************************
* Removal: DIST1 command now handled by the DIST2 processor *
*************************************************************
Support for the DIST1 command has been withdrawn a long time ago, but it
had been kept for compatibility reasons. The DIST2 command is compatible
with DIST1, both in terms of syntax and of results produced; in addition,
it solves many of the problems inherent to the DIST1 algorithm (among
which is the well-known "No recipient for this server, probable
inconsistency in routing tables" message), and it requires about 10 times
less CPU time. The amount of complaints about the aforesaid DIST1 message
has not significantly decreased since the inception of DIST2 and,
although the answer is always the same, it does take a lot of time for
the LISTSERV Coordinators to answer these questions. It has therefore
been decided to remove the DIST1 command, and to automatically process it
as a DIST2 command. In addition, lists with "Mail-via= DIST1" will now be
treated as if "Mail-via= DIST2" had been specified.
********************************************************
* Procedural change: New ":foreign" tag in PEERS NAMES *
********************************************************
Pending a final decision from the EARN Executive Committee, it is assumed
that EARN does not want the PEERS NAMES and LINKSWT FILE to be updated on
EARN servers by Eric Thomas. It is in any case clear that, in the long
run, this will not be possible: two "obvious" reasons are the change in
the format of the LINKSWT FILE (see below), and the need to establish a
LISTSERV "gateway" between the two networks. LISTSERV has therefore been
modified to recognize a ":foreign.YES" tag in a PEERS NAMES entry as
meaning that the server in question, although known by LISTSERV, is not
to receive any of the files that are normally distributed to all servers
when a new version is received from the LISTSERV Master Coordinator. The
exact meaning of this tag may change in the future, depending on how
exactly it is decided to loosen the bond between EARN and non-EARN
servers.
**************************************************************
* Performance: "Sent file" messages now discarded by LSVIUCV *
**************************************************************
In order to improve the performance of "hub" LISTSERV machines, and
particularly the ones which run the LMON package, LSVIUCV has been
modified to process the following RSCS messages at the assembler level:
1. "Sent file" and "Spooled to" are counted and discarded without
returning to the REXX level.
2. "File enqueued on link" messages are discarded without returning to
the REXX level.
Some hub LISTSERVs used to be held 100% busy for several minutes (periods
of up to 15 minutes have been actually seen) discarding such messages at
peak time (the inbound message rate being then over 10/second). The SHOW
command (see below) will display the counter mentioned in point 1.
********************************************************
* Enhancement: 32-bits weight support for LINKSWT FILE *
********************************************************
Release 1.5o did not allow the distance between any two nodes in the
network to exceed 32768 "weighted hops". It also did not make it possible
to define links faster than the "standard" 9600bps, yet not "infinitely
fast" (as CTC links are usually assumed to be). This conflicted with the
goals of the new BITEARN NODES format, and also made it impossible to
accurately represent the topology of a network where more and more
64k/56k are being introduced.
With release 1.6a, the distance and link weight counters have been
changed to 32-bits, which required a change in the format of the link
summary file, formerly known as BITEARN LINKSUM. To avoid any possible
confusion, the new link summary file has been called BITEARN LINKSUM2;
this makes it impossible to use the old LSVBITFD with the new file, or
vice-versa.
The format of the link weights file (LINKSWT FILE) has also been enhanced
to make it possible to define a "default" weight different from 1, ie to
rate a "standard" 9600bps line as weighing more than 1, which in turn
makes it possible to properly declare the faster 64k lines.
This change has been accompanied by a performance improvement of a factor
of 4 for LSVBITFD (the program that calculates distances between nodes),
and by a minor enhancement to LSVBITGN, which now produces detailed
warning messages when invalid entries are found in LINKSWT FILE. LSVBITWT
is now obsolete, and can be erased once the migration is complete.
*******************************************************************
* Enhancement: GET quota no longer stops transmission of packages *
*******************************************************************
With release 1.6a, the retrieval of a package cannot be interrupted by a
"quota exceeded" error. That is, if LISTSERV has decided to allow you to
retrieve the package, it will not "stop in the middle" as it used to do
under 1.5o. If your daily quota has already been exceeded when you ask
for the package, it will, of course, refuse to send it.
For technical reasons related to the way the "GET package" function is
implemented (using recursion) and to the calling convention for the GET
quota exit (LSV$GETQ EXEC), it was not possible to calculate the total
size of the package in advance and present it to the exit for validation.
The result is that what the exit will be called for is the first file of
the package, whatever that file happens to be, and the call will be
suppressed for all the other files in the package. If nested packages are
used, only the first (outermost) package will cause the exit to be
called, so all the files required to run the package are indeed sent.
*********************************************************************
* Enhancement: Use of BSMTP to reduce the load on Internet gateways *
*********************************************************************
New support has been added to release 1.6a to reduce the load on the
various Internet gateways through the use of BSMTP. The gateways store
each individual message in a separate disk file, regardless of the number
of recipients of that message. The "LISTSERV standard" means of
distributing list postings is to build a different RFC822 header for each
recipient, whose 'To:' field points to the person(s) in question. To
limit the size of the header, it is clearly not possible to include the
name and RFC822 address of all recipients of the message in a single
RFC822 header; indeed, LISTSERV limits the amount of addresses in a given
'To:' field (and hence in a given message) to 5, and only recipients
located at the same node are "batched together" in this fashion. This
clearly makes it impossible to send a single message to the internet
gateway with a single header corresponding to all the recipients of the
mail.
In order to solve this problem, a new form of RFC822 header has been
introduced in release 1.6a. In addition to the default "cleansed"
SHORTHDR and to the comprehensive FULLHDR, it is now possible for users
to select (via the 'SET listname option' command) two new types of
headers: SHORTBSMTP and FULLBSMTP (minimum abbreviation: SHORTB and
FULLB). These contain the same fields as SHORTHDR and FULLHDR
(respectively), with the only difference that the 'To:' field contains a
constant string rather than the address of the recipient(s). This string
is normally 'Multiple recipients of list xxxx <xxxx@yyyy>', but it can
get changed to 'Multiple recipients <BSMTP@LIST>' if the message gets
distributed via LISTSERV sites that have not yet upgraded to 1.6a (these
servers would not forward the original string while processing the
DISTRIBUTE job, and the final 1.6a server would therefore have to use
something else instead). In any case, the name of the list will appear in
the 'Sender:' field, as usual; the only reason why it has been decided to
put it into the 'To:' field at all is that some mail user agents like
Rice MAIL display the contents of this field on the incoming mail menu
and make it possible to select messages based on its contents.
In order to save postmasters and list owners the tremendous amount of
time that would be required to update the entries of all Internet
recipients of BITNET lists to take advantage of this new facility, it has
been decided that subscribers which are signed up with a domain-type
address (eg 'CUNYVM.CUNY.EDU' as opposed to 'CUNYVM') will automatically
receive a SHORTBSMTP header, unless they issue a SET command to
explicitly request another form of header (eg 'SET XYZ-L SHORTHDR').
Recipients signed up under an NJE address will continue to get the
standard LISTSERV header they are used to (they can, however, explicitly
request a BSMTP header if they so desire).
The SHORTBSMTP and FULLBSMTP options will, of course, be valid choices
for the "Default-options=" list header keyword. The output of a 'QUERY
listname' command will show 'Header= Short' (resp 'Full') for a normal
short (resp full) header, and 'Short(BSMTP)' (resp 'Full(BSMTP)') for a
BSMTP one. It should be noted, however, that BSMTP headers are only
supported for lists which are DISTRIBUTEd (ie "Mail-via= DISTRIBUTE" or
"Mail-via= DIST2" is present in the list header). If the list is not
distributed, normal (non-BSMTP) headers of the same kind are produced for
the recipients which had selected BSMTP. The purpose of this restriction
is to avoid a duplication of code which was felt to be unjustified, as
the only way to significantly reduce the load imposed by a list on the
network is to use the DISTRIBUTE function. We should therefore
concentrate our efforts on improving this function, rather than designing
ways to make it slightly less inefficient to bypass it.
Finally, it should be noted that the BSMTP support is release dependent,
and that it will not work as documented unless all the servers in the
distribution path (from the one hosting the list to the one performing
the final delivery) are upgraded to 1.6a. Minor problems will be
experienced if this is not the case:
1. The server hosting the list is still running 1.5o or older: the 'SET
listname xxxBSMTP' command will be rejected, and distribution will
proceed using normal headers, as it used to do under 1.5o.
2. The server performing the final delivery is still running 1.5o or
older: a normal header will be generated, and the 'To:' field will
read 'To: BSMTP <userid@nodeid>'.
3. Both host and final servers run 1.6a or higher, but some servers in
the DISTRIBUTE path are still running 1.5o or older: the contents of
the 'To:' field of BSMTP headers will read 'To: Multiple recipients
<BSMTP@LIST>'.
*******************************************************************
* Enhancement: "Editor=" keyword now processed as a 'mon-address' *
*******************************************************************
The "Editor=" keyword, which used to be handled as a series of RFC822
addresses, is now processed as a 'mon-address' (see LISTKEYW MEMO), which
makes it possible to authorize the owners or subscribers of a list not to
have to go through the moderator to mail to it (eg 'Editor= userid@node,
(listname),Owner(listname),etc'). The first address in the keyword
('userid@node' in the example above) is still the one to which mail will
be forwarded if the sender does not match one of the items in the
"Editor=" keyword; because the order in which recipients are listed when
"generic" items such as 'Owner(listname)' are used is unpredictable, the
first item in the "Editor=" field must always be a full RFC822 address:
"Editor= JOHN@LONDON,Owner(SMOG-L)" is valid, but "Editor= Owner(SMOG-L),
Postmasters" is not because the order in which the postmasters and the
various owners of list SMOG-L will be merged into the "Editor=" keyword
is unpredictable and the definition of the keyword would therefore be
ambiguous.
**************************************************************
* Enhancement: Prototype version of the UDD included in 1.6a *
**************************************************************
The current PROTOTYPE version of the LISTSERV UDD (User Directory
Database) has been included in release 1.6a of LISTSERV. This function is
not currently supported for production use, and still lacks a couple of
functions (eg the NAD cannot yet act on behalf of his users), but you are
encouraged to give it a try and send your comments to UDD-L@CEARN, a
public forum for discussing issues related to the UDD. You may also wish
to subscribe to this list.
For more information about the functions provided by the UDD, please read
the archives of the UDD-L@CEARN list.
************************************************************
* Enhancement: Batching of recipients with DISTRIBUTE MAIL *
************************************************************
The DIST2 processor has been enhanced to batch up to 5 recipients located
on the same node in a single outbound mailfile when processing a
DISTRIBUTE MAIL job. This will reduce the traffic whenever several
recipients at a site running MAILER but not LISTSERV are subscribed to
the same list; it will also ensure consistency in the delivery of
distribution list postings, which will now be handled the same way
regardless of whether or not the list operates with "Mail-via=
DISTRIBUTE".
*************************************************************
* Enhancement: Basic statistics on the activity of LISTSERV *
*************************************************************
LISTSERV now collects "coarse" statistics on its activity. These are
mostly counters which are kept in storage (never written to disk), and
give some information about the amount of activity which has taken place
since LISTSERV was last rebooted. More information and optional automatic
recording/checkpointing to disk would also be useful, but not enough time
was available to implement these functions.
These statistics are displayed whenever you issue a STOP command, unless
the QUIET option is used (you will probably want to change the "shut down
all servers" procedure of your operators to do TELL LISTSERV QUIET STOP).
They are, in any case, displayed on the LISTSERV console log. An easy way
to collect them is to have some server issue a 'TELL LISTSERV MAIL STOP
REBOOT' command every day and collect the resulting output file. There is
also a new command, 'SHOW', which allows you (or any user) to display
these counters, along with some more information on the status of
LISTSERV (number of queued jobs/messages, if any, average CPU time
utilization, etc). The output from this command might look like this:
-------------------------------------------------------------------------
>LISTSERV@XYZ is a backbone server running version 1.6a under VM/SP
>RELEASE 4 HPO LEVEL 42 and CMS release 4. The average CPU time used by
>LISTSERV is 3.7%. Since LISTSERV was last rebooted (on 31 May 1989 at
>23:59, ie 11h35 ago), the following requests have been processed:
>
>- Interactive messages from users 261
>- Sent file/Spooled to messages 1233 (ignored, useless network traffic)
>- Network (RSCS) status messages 99 (excludes Sent file/Spooled to)
>
>- Postings to distribution lists 50 (49 via DISTRIBUTE)
>- DISTRIBUTE jobs (mail files) 182
>- DISTRIBUTE jobs (non-mail files) 9
>- Other reader files (jobs, etc) 158
>
>- Database searches (interactive) 1
>- Database searches (batch mode) None
-------------------------------------------------------------------------
You are strongly advised to reboot LISTSERV on a regular basis,
preferrably once a day, if you want to make use of these counters, as
some of them (notably the "Sent file" one) may increase very quickly and
produce very large numbers which are not very "friendly" to the human
reader.
|
