Description of the changes from release 1.5o to 1.6a of LISTSERV -------------------------------------------------------------- June 1st, 1989 *********** * Warning * *********** The planned availability of release 1.6a of LISTSERV is the week of June 19th-23rd. The following description applies to the beta-test programme, which has been started on May 31st; minor design changes or additions of new functions can be made before the final version is released. In order to save network bandwidth, this document will probably not be re-posted in its final form to the LSTSRV-M list; instead, a smaller file listing the changes made to it will be sent, so that, in order to obtain the current description of the changes from release 1.5o to 1.6a, you should search the LSTSRV-M archives for all items with a subject starting with "Changes from release 1.5o to 1.6a". **************** * Requirements * **************** LISTSERV release 1.6a is supported on any hardware/operating system configuration supporting release 1.5o. In particular, support for VM/XA SP (in 370 mode) is available in the "base" version of 1.6a (no "fix" shipment required). Release 1.6 of LISTSERV is available only to BITNET and NetNorth sites, for political reasons. EARN sites now receive support for the EARN version of LISTSERV from Turgut Kalfaoglu <TURGUT@TREARN>, to whom EARN users should address any and all inquiries regarding LISTSERV. ***************** * Compatibility * ***************** Release 1.6a is fully compatible with 1.5o, with the exception of the behavioural changes noted under the heading "Use of BSMTP to reduce the load on Internet gateways" and of the removal of the DIST1 command, which is now handled by the DIST2 processor. The syntax of the DIST2 command is a superset of that of DIST1, and the results produced are compatible with those produced by DIST1; however, some of the messages are different, and the selected distribution path might not be the same. Release 1.6a includes all the fixes contained in the 'FIX15O1' service shipment. It can be installed directly on top of the base 1.5o version. LSVDIST and LSVBITWT have been made obsolete by release 1.6a; LSVBITWT MODULE and ASSEMBLE have consequently been removed from TOOLS FILELIST. The following files can be erased from LISTSERV's disks once the migration is complete: LSVBITWT ASSEMBLE, LSVBITWT MODULE, LSVBITWT HELPCMS, LSVDIST EXEC and LSVDISTX EXEC. *************************** * Important fix: LSVCARDR * *************************** The release 1.5o version of LSVCARDR had a bug which could cause LISTSERV to enter a disabled loop or die with an addressing or protection exception. This was the most frequent cause of LISTSERV crashes, with the well-known storage fragmentation problem causing the REXX interpreter to stop with a "Machine storage exhausted" message, and about which little or nothing can be done. This bug, which was triggered by a transient abnormal behaviour of CP release 5, has been fixed in 1.6a. ************************************************************* * Removal: DIST1 command now handled by the DIST2 processor * ************************************************************* Support for the DIST1 command has been withdrawn a long time ago, but it had been kept for compatibility reasons. The DIST2 command is compatible with DIST1, both in terms of syntax and of results produced; in addition, it solves many of the problems inherent to the DIST1 algorithm (among which is the well-known "No recipient for this server, probable inconsistency in routing tables" message), and it requires about 10 times less CPU time. The amount of complaints about the aforesaid DIST1 message has not significantly decreased since the inception of DIST2 and, although the answer is always the same, it does take a lot of time for the LISTSERV Coordinators to answer these questions. It has therefore been decided to remove the DIST1 command, and to automatically process it as a DIST2 command. In addition, lists with "Mail-via= DIST1" will now be treated as if "Mail-via= DIST2" had been specified. ******************************************************** * Procedural change: New ":foreign" tag in PEERS NAMES * ******************************************************** Pending a final decision from the EARN Executive Committee, it is assumed that EARN does not want the PEERS NAMES and LINKSWT FILE to be updated on EARN servers by Eric Thomas. It is in any case clear that, in the long run, this will not be possible: two "obvious" reasons are the change in the format of the LINKSWT FILE (see below), and the need to establish a LISTSERV "gateway" between the two networks. LISTSERV has therefore been modified to recognize a ":foreign.YES" tag in a PEERS NAMES entry as meaning that the server in question, although known by LISTSERV, is not to receive any of the files that are normally distributed to all servers when a new version is received from the LISTSERV Master Coordinator. The exact meaning of this tag may change in the future, depending on how exactly it is decided to loosen the bond between EARN and non-EARN servers. ************************************************************** * Performance: "Sent file" messages now discarded by LSVIUCV * ************************************************************** In order to improve the performance of "hub" LISTSERV machines, and particularly the ones which run the LMON package, LSVIUCV has been modified to process the following RSCS messages at the assembler level: 1. "Sent file" and "Spooled to" are counted and discarded without returning to the REXX level. 2. "File enqueued on link" messages are discarded without returning to the REXX level. Some hub LISTSERVs used to be held 100% busy for several minutes (periods of up to 15 minutes have been actually seen) discarding such messages at peak time (the inbound message rate being then over 10/second). The SHOW command (see below) will display the counter mentioned in point 1. ******************************************************** * Enhancement: 32-bits weight support for LINKSWT FILE * ******************************************************** Release 1.5o did not allow the distance between any two nodes in the network to exceed 32768 "weighted hops". It also did not make it possible to define links faster than the "standard" 9600bps, yet not "infinitely fast" (as CTC links are usually assumed to be). This conflicted with the goals of the new BITEARN NODES format, and also made it impossible to accurately represent the topology of a network where more and more 64k/56k are being introduced. With release 1.6a, the distance and link weight counters have been changed to 32-bits, which required a change in the format of the link summary file, formerly known as BITEARN LINKSUM. To avoid any possible confusion, the new link summary file has been called BITEARN LINKSUM2; this makes it impossible to use the old LSVBITFD with the new file, or vice-versa. The format of the link weights file (LINKSWT FILE) has also been enhanced to make it possible to define a "default" weight different from 1, ie to rate a "standard" 9600bps line as weighing more than 1, which in turn makes it possible to properly declare the faster 64k lines. This change has been accompanied by a performance improvement of a factor of 4 for LSVBITFD (the program that calculates distances between nodes), and by a minor enhancement to LSVBITGN, which now produces detailed warning messages when invalid entries are found in LINKSWT FILE. LSVBITWT is now obsolete, and can be erased once the migration is complete. ******************************************************************* * Enhancement: GET quota no longer stops transmission of packages * ******************************************************************* With release 1.6a, the retrieval of a package cannot be interrupted by a "quota exceeded" error. That is, if LISTSERV has decided to allow you to retrieve the package, it will not "stop in the middle" as it used to do under 1.5o. If your daily quota has already been exceeded when you ask for the package, it will, of course, refuse to send it. For technical reasons related to the way the "GET package" function is implemented (using recursion) and to the calling convention for the GET quota exit (LSV$GETQ EXEC), it was not possible to calculate the total size of the package in advance and present it to the exit for validation. The result is that what the exit will be called for is the first file of the package, whatever that file happens to be, and the call will be suppressed for all the other files in the package. If nested packages are used, only the first (outermost) package will cause the exit to be called, so all the files required to run the package are indeed sent. ********************************************************************* * Enhancement: Use of BSMTP to reduce the load on Internet gateways * ********************************************************************* New support has been added to release 1.6a to reduce the load on the various Internet gateways through the use of BSMTP. The gateways store each individual message in a separate disk file, regardless of the number of recipients of that message. The "LISTSERV standard" means of distributing list postings is to build a different RFC822 header for each recipient, whose 'To:' field points to the person(s) in question. To limit the size of the header, it is clearly not possible to include the name and RFC822 address of all recipients of the message in a single RFC822 header; indeed, LISTSERV limits the amount of addresses in a given 'To:' field (and hence in a given message) to 5, and only recipients located at the same node are "batched together" in this fashion. This clearly makes it impossible to send a single message to the internet gateway with a single header corresponding to all the recipients of the mail. In order to solve this problem, a new form of RFC822 header has been introduced in release 1.6a. In addition to the default "cleansed" SHORTHDR and to the comprehensive FULLHDR, it is now possible for users to select (via the 'SET listname option' command) two new types of headers: SHORTBSMTP and FULLBSMTP (minimum abbreviation: SHORTB and FULLB). These contain the same fields as SHORTHDR and FULLHDR (respectively), with the only difference that the 'To:' field contains a constant string rather than the address of the recipient(s). This string is normally 'Multiple recipients of list xxxx <xxxx@yyyy>', but it can get changed to 'Multiple recipients <BSMTP@LIST>' if the message gets distributed via LISTSERV sites that have not yet upgraded to 1.6a (these servers would not forward the original string while processing the DISTRIBUTE job, and the final 1.6a server would therefore have to use something else instead). In any case, the name of the list will appear in the 'Sender:' field, as usual; the only reason why it has been decided to put it into the 'To:' field at all is that some mail user agents like Rice MAIL display the contents of this field on the incoming mail menu and make it possible to select messages based on its contents. In order to save postmasters and list owners the tremendous amount of time that would be required to update the entries of all Internet recipients of BITNET lists to take advantage of this new facility, it has been decided that subscribers which are signed up with a domain-type address (eg 'CUNYVM.CUNY.EDU' as opposed to 'CUNYVM') will automatically receive a SHORTBSMTP header, unless they issue a SET command to explicitly request another form of header (eg 'SET XYZ-L SHORTHDR'). Recipients signed up under an NJE address will continue to get the standard LISTSERV header they are used to (they can, however, explicitly request a BSMTP header if they so desire). The SHORTBSMTP and FULLBSMTP options will, of course, be valid choices for the "Default-options=" list header keyword. The output of a 'QUERY listname' command will show 'Header= Short' (resp 'Full') for a normal short (resp full) header, and 'Short(BSMTP)' (resp 'Full(BSMTP)') for a BSMTP one. It should be noted, however, that BSMTP headers are only supported for lists which are DISTRIBUTEd (ie "Mail-via= DISTRIBUTE" or "Mail-via= DIST2" is present in the list header). If the list is not distributed, normal (non-BSMTP) headers of the same kind are produced for the recipients which had selected BSMTP. The purpose of this restriction is to avoid a duplication of code which was felt to be unjustified, as the only way to significantly reduce the load imposed by a list on the network is to use the DISTRIBUTE function. We should therefore concentrate our efforts on improving this function, rather than designing ways to make it slightly less inefficient to bypass it. Finally, it should be noted that the BSMTP support is release dependent, and that it will not work as documented unless all the servers in the distribution path (from the one hosting the list to the one performing the final delivery) are upgraded to 1.6a. Minor problems will be experienced if this is not the case: 1. The server hosting the list is still running 1.5o or older: the 'SET listname xxxBSMTP' command will be rejected, and distribution will proceed using normal headers, as it used to do under 1.5o. 2. The server performing the final delivery is still running 1.5o or older: a normal header will be generated, and the 'To:' field will read 'To: BSMTP <userid@nodeid>'. 3. Both host and final servers run 1.6a or higher, but some servers in the DISTRIBUTE path are still running 1.5o or older: the contents of the 'To:' field of BSMTP headers will read 'To: Multiple recipients <BSMTP@LIST>'. ******************************************************************* * Enhancement: "Editor=" keyword now processed as a 'mon-address' * ******************************************************************* The "Editor=" keyword, which used to be handled as a series of RFC822 addresses, is now processed as a 'mon-address' (see LISTKEYW MEMO), which makes it possible to authorize the owners or subscribers of a list not to have to go through the moderator to mail to it (eg 'Editor= userid@node, (listname),Owner(listname),etc'). The first address in the keyword ('userid@node' in the example above) is still the one to which mail will be forwarded if the sender does not match one of the items in the "Editor=" keyword; because the order in which recipients are listed when "generic" items such as 'Owner(listname)' are used is unpredictable, the first item in the "Editor=" field must always be a full RFC822 address: "Editor= JOHN@LONDON,Owner(SMOG-L)" is valid, but "Editor= Owner(SMOG-L), Postmasters" is not because the order in which the postmasters and the various owners of list SMOG-L will be merged into the "Editor=" keyword is unpredictable and the definition of the keyword would therefore be ambiguous. ************************************************************** * Enhancement: Prototype version of the UDD included in 1.6a * ************************************************************** The current PROTOTYPE version of the LISTSERV UDD (User Directory Database) has been included in release 1.6a of LISTSERV. This function is not currently supported for production use, and still lacks a couple of functions (eg the NAD cannot yet act on behalf of his users), but you are encouraged to give it a try and send your comments to UDD-L@CEARN, a public forum for discussing issues related to the UDD. You may also wish to subscribe to this list. For more information about the functions provided by the UDD, please read the archives of the UDD-L@CEARN list. ************************************************************ * Enhancement: Batching of recipients with DISTRIBUTE MAIL * ************************************************************ The DIST2 processor has been enhanced to batch up to 5 recipients located on the same node in a single outbound mailfile when processing a DISTRIBUTE MAIL job. This will reduce the traffic whenever several recipients at a site running MAILER but not LISTSERV are subscribed to the same list; it will also ensure consistency in the delivery of distribution list postings, which will now be handled the same way regardless of whether or not the list operates with "Mail-via= DISTRIBUTE". ************************************************************* * Enhancement: Basic statistics on the activity of LISTSERV * ************************************************************* LISTSERV now collects "coarse" statistics on its activity. These are mostly counters which are kept in storage (never written to disk), and give some information about the amount of activity which has taken place since LISTSERV was last rebooted. More information and optional automatic recording/checkpointing to disk would also be useful, but not enough time was available to implement these functions. These statistics are displayed whenever you issue a STOP command, unless the QUIET option is used (you will probably want to change the "shut down all servers" procedure of your operators to do TELL LISTSERV QUIET STOP). They are, in any case, displayed on the LISTSERV console log. An easy way to collect them is to have some server issue a 'TELL LISTSERV MAIL STOP REBOOT' command every day and collect the resulting output file. There is also a new command, 'SHOW', which allows you (or any user) to display these counters, along with some more information on the status of LISTSERV (number of queued jobs/messages, if any, average CPU time utilization, etc). The output from this command might look like this: ------------------------------------------------------------------------- >LISTSERV@XYZ is a backbone server running version 1.6a under VM/SP >RELEASE 4 HPO LEVEL 42 and CMS release 4. The average CPU time used by >LISTSERV is 3.7%. Since LISTSERV was last rebooted (on 31 May 1989 at >23:59, ie 11h35 ago), the following requests have been processed: > >- Interactive messages from users 261 >- Sent file/Spooled to messages 1233 (ignored, useless network traffic) >- Network (RSCS) status messages 99 (excludes Sent file/Spooled to) > >- Postings to distribution lists 50 (49 via DISTRIBUTE) >- DISTRIBUTE jobs (mail files) 182 >- DISTRIBUTE jobs (non-mail files) 9 >- Other reader files (jobs, etc) 158 > >- Database searches (interactive) 1 >- Database searches (batch mode) None ------------------------------------------------------------------------- You are strongly advised to reboot LISTSERV on a regular basis, preferrably once a day, if you want to make use of these counters, as some of them (notably the "Sent file" one) may increase very quickly and produce very large numbers which are not very "friendly" to the human reader.