Description of the changes for release 1.7e of LISTSERV ------------------------------------------------------- November 11th, 1992 ************** * Highlights * ************** Version 1.7e introduces the following major enhancements: - Support for the popular VMSDUMP format, both for incoming and outgoing files. BACKUP savesets can now be stored as native LISTSERV files and conveniently retrieved in VMSDUMP format. - Support for MIME (RFC1341) for outgoing files. Internet users may now order text files via e-mail without having to worry about truncated lines or incorrect ASCII to EBCDIC translation. - LISTSERV internals, DISTRIBUTE and inter-server communication methods were changed in preparation for release 2.09 of the VM mailer, which will have the ability to handle lines longer than 80 bytes. Note that it was unfortunately not possible to test this code in a realistic setup, and further adjustments might thus be needed before LISTSERV can actually use that feature of MAILER V2.09. - Performance and robustness improvements through the use of private formatting routines rather than IBM-supplied commands for the generation of files in Netdata and other NJE delivery formats (except CARD DUMP). This change makes it possible to use a SFS directory for LISTSERV's work disk (D-disk). - Additional improvements have been made to the notifications sent by LISTSERV to list owners such that the proposed commands can be executed without retyping by simply forwarding the notification back to LISTSERV. **************** * Availability * **************** Release 1.7 is available free of charge to sites which qualify under one (or more) of the categories listed below: a. Members of CREN or one of the CREN Cooperating Networks, excluding: 1. Members of the European Academic and Research Network (EARN). 2. Members whose computing facilities are located in an EEC country. b. Members of EARN, CREN or a CREN Cooperating Network whose computing facilities are located in a nordic country (Denmark, Finland, Iceland, Norway, Sweden). c. Members of EARN whose computing facilities are not located in an EEC country and which have formally applied (in writing) for a license, and complied with the appropriate EARN directives. International organizations are not considered as belonging to the EEC, regardless of where their computing facilities are physically located. Release 1.6 licenses from EEC countries remain valid for release 1.6, but will not be carried over to release 1.7. ******************************* * Supported operating systems * ******************************* Release 1.7 introduces the following changes to the list of supported operating systems: - (1.7a) Withdrawal of support for CMS release 3. - (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA. - (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes. - (1.7b) Support for the Shared File System, with a few restrictions concerning the setup of the A and D-disks and the default filepool setting. See the corresponding section of the 1.7b release notes for more information. - (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes. - (1.7e) Support for the Shared File System for the D-disk. Support for other versions of CP and CMS is unchanged from 1.6e. ************************** * External Compatibility * ************************** Release 1.7e is generally compatible with 1.7d, from the perspective of the end-user (including list managers and maintainers), and with the exception of the changes listed below. Release 1.7e introduces major changes to the LISTSERV internals, making compatibility with locally developed extensions or local modifications problematic. These changes are briefly described in the next section, and only affect sites which made alterations or additions to the LISTSERV code. Changes which affect all LISTSERV sites are highlighted below; note that more details are available, when appropriate, from the longer descriptive section of these release notes. 1. The changes to support mail items with lines longer than 80 bytes are not transparent for peered lists if some of the peers are running a version which does not support the long lines. More details are provided further down. 2. The changes to DISTRIBUTE to support mail items with lines longer than 80 bytes can cause problems with LISTEARN and older versions of LISTSERV. See further down for more details. 3. Release 1.7e internally uses DISTRIBUTE for all postings to mailing lists, in order to avoid having to duplicate the code that had to be added to support CRC verification of messages with long lines. When "Mail-via= Direct" is specified, a new internal DISTRIBUTE option is used to prevent the message from being forwarded to other servers, but the message is still formatted and delivered to the mailer by DISTRIBUTE code. As a result, the SHOW command no longer displays the amount of postings that were processed through DISTRIBUTE; instead, it shows the total amount of recipients. 4. LISTSERV now removes control characters from mail messages by default and expands tabs in the process. Control characters usually have undesirable and unexpected results, as they seldom survive the double ASCII->EBCDIC->ASCII translation and cause the recipient's terminal to execute sequences different from the ones meant by the message sender - not to mention the case of ASCII control characters on EBCDIC terminals. Furthermore, certain combinations of control characters were found to create problems with IBM's SMTP (due to unexpected CRLF sequences appearing in the middle of a record after translation). Lists which absolutely require control characters must have a "Translate= No" keyword added to their header. Release 1.7e is to be installed directly on top of the base 1.7d code, and includes all the fixes available from LISTSERV@SEARN for release 1.7d (ie all the "17Dnnnt FIX" files from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits", which always have to be installed manually. ************************** * Internal Compatibility * ************************** Major changes have been made to LISTSERV internals. The most important ones are briefly described below; refer to the LSTSRV-L archives for more information. 1. FFORMAT/FCLASS information is no longer kept in GLOBALV variables, and must be accessed/changed via calls to LSVCD1. 2. LSVRECV no longer forces the filemode to D if a value was specified. 3. The UUENCODE and LISTSERV-Punch file formats are now provided by built-in LISTSERV code rather than standalone MODULE's. As a result, PDUTIL, LSVPUNCH and LSV$UUE will be automatically erased after installation of 1.7e. If you have local applications which depend on them, make a copy before installing 1.7e. Furthermore, the format of SYSFF FILE has changed; if you had installed new file formats on your server, you will need to make a few adjustments. See the comments in SYSFF FILE for more information. 4. Authors of local command packages should note that LISTSERV does not guarantee that the program stack is in any particular state when the command processor is invoked. MAKEBUF and DROPBUF should be used wherever necessary. The following REXX files have been converted to PREXX with release 1.7e: LSV$UUE LSVMAIL LSVSENDF MIG_* The following MODULE files have become obsolete with release 1.7e: LSVPUNCH PDUTIL ************************** * Compatibility Warnings * ************************** This new section describes planned changes which may affect compatibility (both external and internal). The release in which the change is planned to be introduced is indicated, with the letter 'x' denoting an unknown release of the specified major version (not necessarily posterior to the last release explicitly mentioned). Note that conversion of more REXX files to PREXX is assumed to take place with each new release and is not indicated here. - (1.7x) The internal format of FILELIST/FILEID and attendant files such as AFDLIST FILE will change dramatically. Applications which alter them directly will no longer work. - (1.7x) The internal format of LIST and attendant files (CACHE, STATS, et al) will change dramatically. Applications which alter them directly will no longer work. - (1.7x) LSVDIAD4, LSVFRD1, LSVFWR1 and RXLSVUHF will be eventually removed. You should avoid using them. ******************************** * Minor fixes and enhancements * ******************************** REMINDER: Manual installation is not supported for release 1.7, and the code that implemented it has been removed with version 1.7e. Automatic installation with local password validation must be used instead. Sites which insist on installing manually should note that shipments contain an installation pre- and post-processing exit. If that exit is not executed, or if it is executed in a different order or environment than during automatic installation, serious problems may occur. This section contains a brief description of all the "small yet not unimportant worries" that have been fixed in release 1.7e, along with the affected releases, if known ("All" meaning "any release in which the facility is available", ie the problem was present since the very beginning). Minor enhancements which do not warrant a detailed description are also included; they are all located at the bottom of the list, and can be identified by the word "New" in the "Affected releases" column. Affected releases Description of the bug, problem or enhancement -------- ---------------------------------------------- 1.7d Servers running on VM/SP or VM/HPO were occasionally unable to purge incoming job files if a severe error occured during processing; the job would be reprocessed about 15 minutes later and cause multiple error notifications. 1.7d LISTSERV will no longer generate addresses of the form: From: "(Joe Smith)" <[log in to unmask]> These addresses were not generated intentionally and are unnecessarily complex, but they are perfectly valid. There exist popular mail programs which cannot handle them, but that only means they are in error. 1.7a The recipients counts for network-wide lists produced by LIST SUMMARY were incorrectly calculated. The totals reported by pre-1.7e code happened to be lower than the correct values at the time the fix was written. All Some components of LISTSERV were unable to append to files with more than 64k records, due to the unexpected behaviour of FSWRITE with RECNO=0 but without FORM=E. New Plural forms ('Owners' and 'Editors') are now supported for access-level keywords, with the same meaning as the singular forms. New JOIN and LEAVE are now accepted as synonyms for SUBSCRIBE and SIGNOFF, respectively, in order to improve compatibility with Mailbase. Mailbase appears to be the most robust and comprehensive unix-based list manager currently available, with the added advantage of not confusing the users by claiming to be a unix version of the BITNET LISTSERV. The addition of these two synonyms will hopefully remove an obstacle to the use of Mailbase by a larger community. New In order to make it easier to implement "sub-lists" (a la Mailbase) with LISTSERV, the syntax of the "Subscription=" keyword has been enhanced to support access-control specifications, in addition to the previously supported keywords ("Open", "Closed" and "By_Owner"). Sub-lists can be convenient when a rather large forum is to discuss several sub-topics in parallel and participants are to be free to join and leave the various sub-lists as they see fit. This change makes it possible to implement sub-lists where membership is not open. New A side-effect of the previous change is that "Subscription= Public" is now a synonym for "Subscription= Open", removing one of the most common mistakes made by new list owners. New LISTSERV no longer needs to swap its work minidisk (192) from mode D to mode Z and back. This makes it possible for the work disk to be replaced with a SFS directory, as long as you make sure the directory is already accessed R/W as mode D when LISTSERV is started. Mode Z remains reserved for LISTSERV use (dynamic minidisks, et al). ******************************************************* * Usability/Performance: Support for new file formats * ******************************************************* Release 1.7e introduces support for a number of new file formats, which can be specified on the 'F=' keyword of commands such as GET, REVIEW, etc. - VMSDUMP, the popular file transfer format for VMS systems running JNET. LISTSERV only supports sequential files. For incoming files, the variable portion of VFC files (usually a time stamp) is discarded and UNDEFINED files are not supported (but the other 3 stream types are). For outgoing files, LISTSERV only generates FIXED or VARIABLE files. This is still sufficient to conveniently store BACKUP savesets as native LISTSERV files. - XXENCODE, a version of UUENCODE modified to use a more sensible table so that the resulting messages survive ASCII to EBCDIC translation. - Two flavours of MIME base64-encoded messages. Users with a MIME-capable user interface can use 'F=MIME' to order text files which contain characters that do not survive ASCII to EBCDIC translation, or lines longer than the gateways in the way will support. LISTSERV translates the text to ASCII, adds CRLF sequences, and sends the result as a base64-encoded message of type 'text/appl'. 'F=MIME/APPL' can be used as a replacement for UUENCODE to order binary stream files (content type 'application/octet-stream'). It is important to note that this has exactly the same functionality as UUENCODE: the input file is read, ignoring file attributes and record boundaries, and turned into a stream of bytes, which are then encoded and sent. In other words, this is not a suitable method for transferring text files on binary VM or VMS files. In addition, release 1.7e no longer uses the IBM-supplied system commands to generate files in Netdata and other formats - with the exception of CARD DUMP format, for which a modified version of CARD MODULE is used. This change improves robustness, performance, and considerably simplifies functions such as AFD and GET which no longer need to copy files and swap the work disk around in order to have the right information in the header or to avoid unwanted side effects of the system commands. The new code is about 3-4 times faster for both Netdata and DISK DUMP formats. *************************************************************** * Usability: Support for mail with lines longer than 80 bytes * *************************************************************** LISTSERV internals and inter-server communication methods have been changed to support mail with lines longer than 80 bytes, in preparation for a new version of the VM mailer with support for Netdata format. There is no way to make use of that support with the current version of the mailer (2.08) and it is perfectly normal that lines continue to be split at column 80 after the installation of 1.7e. These changes introduce a number of problems when communication has to go through servers not running 1.7e. Three major components of LISTSERV are impacted: 1. Inter-server commands: older servers (including LISTEARN) are able to process the new commands without problem. However, in certain cases, they may truncate the command if they, in turn, need to forward the request to other servers. In practice this only affects unlikely cases such as e-mail addresses longer than 80 bytes (the command itself is never truncated, but list of addresses passed in DD's might be truncated at column 80). 2. Peered lists: a problem will occur if a message with lines longer than 80 bytes is posted to a peered list and some of the peers are not running version 1.7e. The message would be truncated as per the following table: +---------+-------------------+-----------------+ | Version | Header truncated | Body truncated | +---------+-------------------+-----------------+ | 1.7e | Never (*) | Never (*) | +---------+-------------------+-----------------+ | 1.7d | Folded at 80 | Truncated at 80 | +---------+-------------------+-----------------+ | Other | Truncated at 255, | Truncated at 80 | | | then folded at 80 | | +---------+-------------------+-----------------+ (*) If the local mailer does not support lines longer than 80 bytes, the headers and body will have to be folded at column 80. The truncation of the message text is obviously undesirable, but will only happen if there were lines longer than 80 bytes in the first place; in other words, it is not an issue until mailer 2.09 is released. From then on peered lists will require that all servers run release 1.7e. A temporary keyword, "Long-lines= No", is provided to assist migration in cases where one of the peers absolutely cannot install 1.7e in time (note that peered LISTSERV-LISTEARN lists are not formally supported, but can benefit from that migration aid while it is being provided). When "Long-lines= No" is specified, LISTSERV folds text lines longer than 80 bytes to bypass this problem at the expense of usability. The truncation of header lines at column 255 is potententially a very serious problems. Long header lines may be present regardless of the version of the mailer being used, as LISTSERV must internally unfold RFC822 fields in order to process them and does not re-fold them until necessary (and since inter-server jobs support lines of up to 64k bytes, it never turns out to be necessary). Although header lines longer than 255 are rare, this new mechanism can create problems with peered lists unless all the peers run 1.7d or 1.7e. Because there is no problem with the previous version and older versions are no longer supported, no migration aid was provided. 3. DISTRIBUTE: LISTSERV now uses Netdata format for DISTRIBUTE jobs, so that long mail lines (up to 64k and thus well beyond the 1000 bytes SMTP standard) can be supported. While older servers are perfectly capable of processing such jobs, the same truncation problems as in point 2 exist. Fortunately, version 1.7e is being released before the new VM mailer and it should be possible to remove the few non-LISTEARN sites which are still running 1.7d from the backbone when the mailer is released, thus solving the problem for non-LISTEARN sites. The structure of the LISTSERV backbone in Europe is such that countries running LISTSERV will seldom be affected by truncation due to LISTEARN sites, so this is fortunately a private LISTEARN problem which will have to be solved separately. IMPORTANT: The use of Netdata for DISTRIBUTE will cause significant performance problems to sites running a version older than 1.7d. There will be a noticeable impact for 1.7c sites and a factor of 2 or worse degradation for 1.7b and older. The impact of this change on 1.7d and 1.7e sites is negligible, and, for 1.7e, it is more than compensated for by other performance changes to DISTRIBUTE. *************************************************************** * Usability: Simpler administative procedures for list owners * *************************************************************** The 'Subject:' field conventions for administrative messages introduced with version 1.7d have proved to be very popular with list owners, and are being generalized with 1.7e. Furthermore, messages which request the list owner to confirm a certain action (such as adding a user to a list, or changing his distribution options) are now specially formatted such that simply forwarding the message back to LISTSERV will cause the command to be automatically executed without having to be retyped. Note that the message must be forwarded, and not replied to. While there are user interfaces whose reply function will cause the command to be properly executed, in the majority of cases this will not work (even with REPLY TEXT). Please be careful not to suggest that users use REPLY, even if it happens to work for you. ************************************************** * Usability: Welcome message for new subscribers * ************************************************** Release 1.7e no longer requires a change to the list MAILFORM in order to provide a "welcome message" for new subscribers. The main problems with the MAILFORM solution is that postmaster intervention is required every time the message has to be updated, for security reasons, and that it is very easy to make mistakes as MAILFORM files are actually programs with (a majority of) imbedded text. In particular, it is very easy for novice list owners to forget to double the quotes. Once a subscription has been accepted, LISTSERV will look for a LISTSERV (not CMS) file called 'listname WELCOME'. This is a plain text file which can be easily created and maintained by the list owner if the postmaster has created a filelist for the list. The first line, if it starts with 'Subject:', defines the subject field of the welcome message; otherwise LISTSERV uses a pre-defined subject and uses the first line as part of the message text. Again, contrary to the MAILFORM's, the welcome messages are LISTSERV files; simply placing a 'listname WELCOME' file on LISTSERV's A-disk will not accomplish anything. Since this is a plain text file, there is no need to double quotes or include formatting commands for text which must not be reformatted. This, of course, means that MAILFORM variable substitution sequences are not available. ********************************************************* * Usability: LB64.C - a tool for non-BITNET list owners * ********************************************************* LB64 is a new tool which can be used by non-BITNET list owners to send files or commands (and in particular PUT requests) to LISTSERV without having to worry about lines longer than 80 bytes or characters which do not get translated properly by the gateways the message goes through. LB64 takes advantage of a server-to-server protocol which was added with release 1.7e but was not designed to solve the problem of PUT requests from the Internet with lines longer than 80 bytes (this problem will disappear when the VM mailer supports Netdata format). In other words, LB64 does not claim to be anything but a useful side-effect of an unrelated LISTSERV development; if you dislike it, you are free not to use it. The program is written in C and is designed for unix systems, where its output can be conveniently piped to a 'mail listserv@hostname' command. It can however be used on any system that can emulate unix text files, be it ASCII or EBCDIC. For more information, issue a 'GET LB64.C' command to LISTSERV and read the comments in the program.