Description of the changes for release 1.7d of LISTSERV ------------------------------------------------------- September 14th, 1992 ************** * Highlights * ************** Version 1.7d introduces the following major enhancements: - Significant parts of the LISTSERV supervisor have been rewritten in PREXX, providing a stable basis for future developments while improving robustness and performance. The main performance improvements for this release include the LIST GLOBAL, REVIEW and NODESGEN commands, generation of internal tables such as PEERS and DOMAIN NAMESUM, parsing of RFC822 headers, domain-style address resolution, maintenance of large lists and processing of incoming job files in Netdata format. - Support for the ':oldnode.'/':newnode.' tags of BITEARN NODES, which make it possible to automatically alter list and file subscriptions for nodes that leave BITNET or change their nodeid. - Automatic conversion of Netdata, Disk Dump or Card Dump DISTRIBUTE jobs to plain text when the recipient is only reachable via mail. - Improved subject in notifications sent by LISTSERV to list owners simplify list maintenance. - Relief for sites suffering from problems with the XEDIT SORT command and/or XEDIT failures when building the PEERS NAMESUM file, and unable/unwilling to apply the IBM APAR correcting the problem. Release 1.7d no longer uses XEDIT for sorting internal tables. - The REXX code has been updated in preparation for an incompatible change to the REXX interpreter which will probably be introduced with CMS release 9 (no official statement yet). This does not mean that 1.7d supports CMS9 - this will not be possible until CMS9 becomes available. The only thing that can be said is than older versions of LISTSERV will definitely not work under CMS9 (if the REXX change is introduced then). **************** * Availability * **************** Release 1.7 is available free of charge to sites which qualify under one (or more) of the categories listed below: a. Members of CREN or one of the CREN Cooperating Networks, excluding: 1. Members of the European Academic and Research Network (EARN). 2. Members whose computing facilities are located in an EEC country. b. Members of EARN, CREN or a CREN Cooperating Network whose computing facilities are located in a nordic country (Denmark, Finland, Iceland, Norway, Sweden). c. Members of EARN whose computing facilities are not located in an EEC country and which have formally applied (in writing) for a license, and complied with the appropriate EARN directives. International organizations are not considered as belonging to the EEC, regardless of where their computing facilities are physically located. Release 1.6 licenses from EEC countries remain valid for release 1.6, but will not be carried over to release 1.7. ******************************* * Supported operating systems * ******************************* Release 1.7 introduces the following changes to the list of supported operating systems: - (1.7a) Withdrawal of support for CMS release 3. - (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA. - (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes. - (1.7b) Support for the Shared File System, with a few restrictions concerning the setup of the A and D-disks and the default filepool setting. See the corresponding section of the 1.7b release notes for more information. - (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes. Support for other versions of CP and CMS is unchanged from 1.6e. ************************** * External Compatibility * ************************** Release 1.7d is generally compatible with 1.7c, from the perspective of the end-user (including list managers and maintainers), and with the exception of the changes listed below. Release 1.7d introduces major changes to the LISTSERV internals, making compatibility with locally developed extensions or local modifications problematic. These changes are briefly described in the next section, and only affect sites which made alterations or additions to the LISTSERV code. Changes which affect all LISTSERV sites are highlighted below; note that more details are available, when appropriate, from the longer descriptive section of these release notes. 1. The obsolete GETFID command has been removed, as announced in the 1.7c release notes. This command had been added for the benefit of a software package which was never completed or distributed. 2. The DISTOFF configuration variable is no longer available. 3. The "Formcheck=" list header keyword is no longer honoured; all lists now operate as per "Formcheck= No". This keyword, which in retrospect was an incentive for BITNET nodes to install mail user interfaces compatible with RFC822 or at least IBM NOTE, is now causing a lot of confusion to little avail. 4. The resource usage summary produced by the DISTRIBUTE command when used with the DEBUG option has been removed. Its purpose was to separate the resource usage of the DISTRIBUTE process itself from that of the job processor. The job processor overhead has become negligible, whereas the REXX exit overhead, not accounted for in the DISTRIBUTE resource summary, has become a significant factor. 5. When sending a non-mail file to a list with "Files= Yes" and "Send= Editor", LISTSERV will no longer forward the file to the editors, as this turned out to generate a lot of confusion, especially when the editor was on a non-BITNET machine. Instead, LISTSERV informs the file originator that he should submit the file to the editors himself. 6. The COUNTRY option of the REVIEW command produces slightly different output (see the section on the REVIEW command below). Release 1.7d is to be installed directly on top of the base 1.7c code, and includes all the fixes available from LISTSERV@SEARN for release 1.7c (ie all the "17Cnnnt FIX" files from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits", which always have to be installed manually. ************************** * Internal Compatibility * ************************** Major changes have been made to LISTSERV internals. The most important ones are briefly described below; refer to the LSTSRV-L archives for more information. 1. LIST files are no longer in sort order, and may contain blank lines. 2. BITEARN INTNJE and BITEARN NODESUM2 have been replaced with BITEARN NODESUM3. Standalone utilities such as RXLSVNNI will continue to operate outside the LISTSERV environment, provided they have access to the new file and to the new LSWNS3 MODULE. 3. SIGNUP FILE and PWLIST FILE have been replaced with SIGNUP2 FILE, which must be manipulated exclusively through the LSVU@N function. Please note that /WHOIS and /WHERE are not supported by the author of LISTSERV. Contact their respective authors for information. 4. LSVBESTP EXEC has been replaced with a LSVBSP PREXX entry point which is only partly compatible (LSVBESTP used to blindly INTERPRET one of its arguments). 5. LSVRECV now supports CARD format and generates RECFM V disk files for PUNCH format input files. 6. Authors of local command packages should note that LISTSERV does not guarantee that the program stack is in any particular state when the command processor is invoked. MAKEBUF and DROPBUF should be used wherever necessary. The following REXX files have been converted to PREXX with release 1.7d: LSVADM LSVALLOW LSVBESTP LSVCKB LSVDNSUM LSVFSEND LSVGFD LSVMONAD LSVOWN LSVPRM LSVRECV LSVSORT LSVU@N LSV82I The following MODULE files have become obsolete with release 1.7d: LSVCARDR LSVDDFID RXLSVSPU ************************** * Compatibility Warnings * ************************** This new section describes planned changes which may affect compatibility (both external and internal). The release in which the change is planned to be introduced is indicated, with the letter 'x' denoting an unknown release of the specified major version (not necessarily posterior to the last release explicitly mentioned). Note that conversion of more REXX files to PREXX is assumed to take place with each new release and is not indicated here. - (1.7e) The internal format of FILELIST/FILEID and attendant files such as AFDLIST FILE will change dramatically. Applications which alter them directly will no longer work. - (1.7x) Support for "Mail-via= Direct" will be removed, causing all mail distribution to take place via DISTRIBUTE. - (1.7x) The internal format of LIST and attendant files (CACHE, STATS, et al) will change dramatically. Applications which alter them directly will no longer work. - (1.7x) LSVDIAD4, LSVFRD1, LSVFWR1, RXLSVUHF and LSVPUNCH will be eventually removed. You should avoid using them. ******************************** * Minor fixes and enhancements * ******************************** REMINDER: Manual installation is not supported for release 1.7. Automatic installation with local password validation should be used instead. Sites which insist on installing manually should note that shipments contain an installation pre- and post-processing exit. If that exit is not executed, or if it is executed in a different order or environment than during automatic installation, serious problems may occur. In the first releases of 1.7, this happened to only cause obsolete files not to be erased, thus the only impact was loss of disk space. With release 1.7d, the impact is much more serious. This section contains a brief description of all the "small yet not unimportant worries" that have been fixed in release 1.7d, along with the affected releases, if known ("All" meaning "any release in which the facility is available", ie the problem was present since the very beginning). Minor enhancements which do not warrant a detailed description are also included; they are all located at the bottom of the list, and can be identified by the word "New" in the "Affected releases" column. Affected releases Description of the bug, problem or enhancement -------- ---------------------------------------------- 1.7a Date field incorrect in BITEARN database for non-NJE entries. 1.7a CPU usage ratio reported by the SHOW command was incorrect under VM/SP and HPO due to the session time counters being reset by system accounting checkpoints (CP ACNT), which (as seen by LISTSERV) can take place at arbitrary intervals. CPU usage had to be calculated from session CPU and CONNECT time, yielding a result that, while valid, did not correspond to the time since the last LISTSERV reboot. Under VM/XA and ESA, the result could be invalid (over 100%) because accounting checkpoints do not reset session counters and their value can exceed the capacity of the fixed-width QUERY TIME fields. A more sophisticated method is being used to solve the problem for both SP/HPO and XA/ESA systems. All '.BITNET' suffix was not consistently removed everywhere it should have been. All DISTRIBUTE FILE to mail-only recipient used to set the SMTP 'MAIL FROM:' to the LISTSERV address, thus causing delivery errors to be indirectly sent to the local postmaster, who could do nothing about them. The return address is now set to point to the job originator. New The 'SHOW NODEntry' command can now display member and site entry. It was previously limited to NJE entries only. New The new 'SHOW NAD nodeid' command will display a list of the addresses LISTSERV recognizes as node administrators for the node in question. New The node description and nodeid are now displayed when a PRINT ADDRESS or PRINT CONTACT report is requested after searching the BITEARN database. New (Cosmetics) Comments in RFC822 address fields not properly concatenated to "name" field. Warning messages could be repeated without apparent reason if the parser had to use a given RFC822 field for multiple purposes. Some parser messages improved. New MYDOMAIN configuration variable auto-configured from local node entry in BITEARN NODES if ':internet.' tag available. MYDOMAIN is vitally important to the mailing loop detector. New Allow 'QUERY * FOR user' and 'SET * FOR user' for list owners as well. Lists not owned by the invoker are of course not searched/updated. New Because MAILFORM files are actually programs, quotes have to be doubled when a single quote is desired in the message (as in "It''s a good idea to..."). Failure to do so usually caused a REXX error, which was reported to the postmaster and to LISTSERV coordination. Postmasters usually assumed it was a bug in LISTSERV and took no action. Mispaired quotes will no longer cause a REXX error; instead, a warning will be inserted in the text indicating that unpaired quotes have been detected and suggesting to contact the list owner. Note that in some cases the error cannot be detected. This is a temporary solution, pending the replacement of MAILFORM's by a more robust (but less flexible) system. New To make life easier for list owners and moderators using computer systems which only accept lower-case addresses, the "Owner=" and "Editor=" keywords now accept lower-case usernames provided that the username is enclosed in double quotation marks and stands on a line by its own. Example: Owner= "eric"@SEARN.SUNET.SE works Owner= "[log in to unmask]" does NOT work Owner= joe@xyzvm,"jack"@xyz.edu does NOT work The requirement to use the first listed form only is a temporary restriction pending some additional changes to the LISTSERV internals. Be careful when updating the headers of peered lists: this syntax only works with 1.7d. Note that LISTSERV never supported blanks in user names, quoted or not. New Entries for deleted nodes are now removed from SIGNUP2 file during the NODESGEN process, to save disk space. New The default values for FILEMAXL and MAILMAXL, which dated back to 1986, have been doubled. Since 1.7c, a backbone server which does not issue warnings about lack of storage at startup should be able to process 20,000 lines jobs. **************************************************************** * Performance/Reliability: Changes to BITEARN NODES processing * **************************************************************** The BITEARN INTNJE and BITEARN NODESUM2 files have been replaced with a new file, BITEARN NODESUM3, designed to easily accomodate additional tables as the need arises. Four new tables have already been incorportated into the file. Information about administrative networks (CREN, EARN, CAREN, etc) is now extracted directly from BITEARN NODES rather than COUNTRY FILE, which it was difficult to keep up to date. Information about BITNET mailers is also extracted directly from BITEARN NODES. LISTSERV no longer needs access to XMAILER NAMES, and synchronization problems are avoided while improving lookup performance. The NODESGEN process has been converted to PREXX to facilitate the development of functions requiring changes to the network tables through a performance improvement of a factor of about 5 in CPU time and better overall performance on I/O and paging constrained systems. The whole process now takes less than 3 seconds of CPU time on a 3090-J, about half of which is due to post-processing code in REXX (filelist refresh, etc). *********************************************************** * Usability: Support for the ':oldnode.'/':newnode.' tags * *********************************************************** The use of a more efficient language for the NODESGEN process has made it possible to add support for the ':oldnode.' and ':newnode.' BITEARN NODES tags, whose exact syntax is described in the NEWTAGS DESCRIPT file available from NETSERV. IMPORTANT: LISTSERV implements a superset of the definitions described in the november 1991 edition of NEWTAGS DESCRIPT. The document is being updated to extend the syntax of the tags, but, at the time of this writing, the new version of NEWTAGS DESCRIPT was still being proofread. You can already make use of the new syntax, as both UPDATE and VERNODES have already been changed to recognize it; it is only the formal description which has been delayed. LISTSERV uses the information in these tags to update subscriptions in mailing lists, AFD/FUI lists, and the user registration file (SIGNUP2 FILE), which contains full names, passwords and user preferences. In addition, nodes with a ':newnode.' tag are saved in the BITEARN NODESUM3 table, and any command from a user at the old node, or subscription request for such a user by a list owner, is automatically translated to the new address. This prevents new entries with the old node name from being entered in mailing lists or other subscription files; this is particularly important when the new address is an Internet hostname, as the node will not have the benefit of a ':oldnode.' tag in its new entry to take care of any left-over subscription, since it will never have a new entry. ********************************************************************* * Usability: Enhanced subject on administrative mail to list owners * ********************************************************************* In order to make it easier for list owners to process large amounts of administrative mail from LISTSERV, the 'Subject:' field of these messages has been modified as follows: <1.7c> Delivery error notice sent to list XYZ-L <1.7d> XYZ-L: error report from sun3.abc.edu <1.7c> New subscription to list XYZ-L <1.7d> XYZ-L: [log in to unmask] joined the list <1.7c> User signing off the XYZ-L list <1.7d> XYZ-L: [log in to unmask] left the list This makes it possible for experienced list owners to process most messages without having to actually read the message text, without making any difference to inexperienced owners. ************************************************************* * Usability/Performance: Enhancements to the REVIEW command * ************************************************************* The REVIEW command has been partly rewritten in PREXX and enhanced to allow the user to indicate how he wants the subscribers list to be sorted. This is accomplished through a new option, 'BY', which must be followed by one of the following field names (minimum abbreviations are shown in upper case, as usual): - 'Name': sort by subscriber name. LISTSERV uses some heuristics in an attempt to separate the first name and last name from any possible organization name, title or affiliation, and sorts the list by last name, then first name. There are of course cases where this does not have the expected effect. - 'NODEid' or 'Hostname': sort by nodeid/hostname (whatever is to the right of the '@' sign in the subscriber's address - the two options are synonyms). - 'Userid/USERName': sort by username (the part to the left of the '@' sign). - 'Country/ies': sort by country of origin and produce a statistics summary listing the various countries encountered and the number of subscribers in each country. Note that the list is sorted by country name (Switzerland), and not by country code (CH). Sorting by country causes comments with the country names to be inserted in the list (see example below), since it would otherwise be difficult to associate country names with e-mail addresses. Because of this difficulty, you may not use the country name as a secondary sort field; that is, 'REVIEW XYZ-L (BY (COUNTRY NAME)' is valid, but 'BY (NAME COUNTRY)' would be rejected. ---------- Example of comments inserted for the COUNTRY option ---------- * Netherlands: SOND0008@HASARA11 peter vons U001222@HEARN Jos Wennmacker EVERS@HUTRUU53 Evert Jan Evers / Univ. Utrecht, NL * Spain: ZCOCBF01@EBCESCA1 Cristina Blanxer * Sweden: ERIC@SEARN Eric Thomas ------------------------------------------------------------------------- The default sort fields are 'BY (HOSTNAME USERNAME)', for compatibility. Unless requested otherwise, entries with the same primary field are then sorted by hostname/username to ensure results are predictable (ie that reviewing the same list twice with the same options always gives the same results). Finally, the old COUNTRY options (as in 'REVIEW XYZ-L (COUNTRY') is now a synonym for 'BY COUNTRY'. There is no way to get the country statistics report without also ordering the subscribers list by country name. *************************************************************** * Usability: Changed 'Reply-To:' field on administrative mail * *************************************************************** Most admnistrative messages sent by LISTSERV to end users now have a 'Reply-To:' field pointing to 'LISTNAME-request@LOCALNODE', an address which LISTSERV expects to point to the list owners (see next paragraph). This prevents hasty users from inadvertently replying to the LISTSERV userid, only to discover that the long letter they had not saved as been processed as a set of commands and discarded. In order for the address in question to work as advertised, you must have the automatic owner-/-request support installed in your mailer. If you are running version 2.08 of the VM mailer, you can get the necessary changes from LISTSERV@SEARN; please refer to the description posted to the LSTSRV-L archives on 26 Nov 1991 for more details. If you are running version 2.09 or higher, it should be possible for you to activate this function simply by setting a configuration variable; the syntax is unknown at the time of this writing, as 2.09 was not yet available. ***************************************************************** * Performance/Reliability: Internal incoming spool file decoder * ***************************************************************** In order to improve performance when processing incoming spool files in Netdata format, which might become the norm rather than the exception due to planned changes to the VM mailer, and to decrease the dependence on unexpected changes to standard VM commands, LISTSERV now has its own internal decoder for incoming spool files. This decoder accepts the three major formats (Netdata, DISK and CARD), plus of course "raw" PUNCH files. It is considerably faster than native commands, except for CARD format, but even then performance is better since the data does not need to be staged through a disk file. Noteworthy points: 1. You may now submit jobs to LISTSERV in CARD format. Previous versions did not support CARD for input files. 2. With CARD and DISK formats, only the first file in the deck is extracted and processed. As mentioned in previous release notes, LISTSERV does not accept multiple jobs per spool file because it would not be able to properly checkpoint them. Jobs are generally transferred to the postmaster for manual examination and retry in case of severe failure, and this requires each job to reside in a separate file. 3. LISTSERV does not support the pre-VM/SP (1970's) version of the DISK format. While there is no VM/370 system on BITNET, some very old emulator software might still use this ancient format. ************************************************************************* * Usability: Automatic conversion of DISTRIBUTE to mail-only recipients * ************************************************************************* The addition of the internal spool file decoder has made it possible for LISTSERV to automatically convert DISTRIBUTE FILE jobs to plain text when the recipient is only reachable via mail. Scenario: a user sends a file to a list without a mail envelope (ie via SENDFILE, SEND/FILE or equivalent), because the file is large or because he does not want it logged to the archives. The typical example is a large postscript file which may need to be revised and resent a few times. Unfortunately, some of the recipients do not have BITNET access and can be reached only via mail. Former behaviour: LISTSERV encapsulates the NJE file in an RFC822 envelope and delivers it to the mail-only users. If the file was sent in an encoded format such as Netdata or DISK DUMP, the data in question will contain binary characters which will not survive delivery. In any case, the recipient is unlikely to have software that can decode such files. New behaviour: LISTSERV examines the first record of the NJE file and realizes that it contains binary characters. It runs the file through its internal decoder and gets a successful return code, confirming that the file was indeed in a supported network format (see note on VMSDUMP below). LISTSERV sends the decoded text to the recipients, along with the following message: ----------------------- Message sent to recipient ----------------------- Note: conversion from a BITNET transmission format not suitable for mail delivery was locally attempted. This type of conversion may sometimes require "choices" to be made by the conversion program, based on the (lack of) support for various file formats on the target operating system. The "choices" made by LISTSERV may not be the ones you expected, since it does not know anything about the system you are using. However, you would not have been able to use the file at all if it had not been converted. If you have trouble using the file as you received it, please contact the person who sent it and arrange for an alternate delivery method. ------------------------------------------------------------------------- Important remarks: 1. LISTSERV can only decode the formats it understands. In particular, VMSDUMP format and some types of Netdata files originating from MVS systems cannot be decoded. In such cases, behaviour is unchanged from the previous versions. Note that VMSDUMP is a proprietary format for which no specifications are available. 2. The decoded file might still be unusable if it was, for instance, an executable file. LISTSERV attempts the conversion on the assumption that binary characters do not survive ASCII to EBCDIC translation in mail gateways anyway, thus there is a choice between something that might work and something that assuredly will not work. 3. The server that converts the file is the server that performs the final delivery to the user. Backbone servers which relay the file onwards do not examine the delivery lists of other servers. Thus, submitting a job to a 1.7d server is no guarantee that conversion will take place for all recipients; it is the version of the server delivering to the recipient that matters. *********************************************************************** * Performance: New BITEARN DISTSUM2 file improves large distributions * *********************************************************************** In order to provide acceptable performance, the DISTRIBUTE process uses a cache file, formerly BITEARN DISTSUM, to save the results of its CPU-intensive topological calculations. This file used to contain information about both BITNET and domain-style addresses, with a ratio depending on the type of list hosted by the server. The conversion of the domain-style address resolver to PREXX has made it possible to remove domain-style addresses from this file, thus saving disk space while decreasing lookup time as the file becomes much smaller. This was a good opportunity to change the caching algorithm and the format of the file, to further improve performance. The new file is called BITEARN DISTSUM2. For a typical list with 150 subscribers, a performance improvement of 30-40% was achieved with an empty cache, while still saving about 10% when all the entries are in the cache. A more significant improvement should be achieved with large (1000+ subscribers) lists, but no benchmark is available at this time. Note that this only affects the server originating the distribution - the savings for "relay" servers are minimal.