Description of the changes for release 1.7c of LISTSERV ------------------------------------------------------- April 20th, 1992 ************** * Highlights * ************** Version 1.7c introduces the following major enhancements: - Support for CMS release 8, and improved support for CMS release 7 and the Shared File System (SFS). Due to its generally observed lack of robustness and stability, CMS release 6 is still not formally supported; problems that cannot be reproduced under CMS release 7 will not be addressed. - Significant parts of the LISTSERV supervisor have been rewritten in PREXX, providing a stable basis for future developments while improving robustness and performance. A typical backbone server with medium load can expect to save 20% on CPU time (TCPU) and to observe a reduction of I/O operations by about 50%. - The maintenance of the "list of lists" (GLOBLIST FILE), a rather frequent and previously very expensive process in terms of CPU time, has been converted to PREXX. The resulting eight to tenfold reduction in CPU cost removes a potential obstacle to joining the backbone for small systems. - Source code will be maintained using UPDATE, whenever practical, from release 1.7d onwards (that is, release 1.7c source files have been sequenced to allow future changes to be made with UPDATE). - Relief for CMS release 6 sites suffering from "Interpreter failure" errors and unable/unwilling to apply the IBM APAR correcting the problem. Changes made in release 1.7c chanced to remove the sequence of instructions that usually triggered the error. **************** * Availability * **************** Release 1.7 is available free of charge to sites which qualify under one (or more) of the categories listed below: a. Members of CREN or one of the CREN Cooperating Networks, excluding: 1. Members of the European Academic and Research Network (EARN). 2. Members whose computing facilities are located in an EEC country. b. Members of EARN, CREN or a CREN Cooperating Network whose computing facilities are located in a nordic country (Denmark, Finland, Iceland, Norway, Sweden). c. Members of EARN whose computing facilities are not located in an EEC country and which have formally applied (in writing) for a license, and complied with the appropriate EARN directives. International organizations are not considered as belonging to the EEC, regardless of where their computing facilities are physically located. Release 1.6 licenses from EEC countries remain valid for release 1.6, but will not be carried over to release 1.7. ******************************* * Supported operating systems * ******************************* Release 1.7 introduces the following changes to the list of supported operating systems: - (1.7a) Withdrawal of support for CMS release 3. - (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA. - (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes. - (1.7b) Support for the Shared File System, with a few restrictions concerning the setup of the A and D-disks and the default filepool setting. See the corresponding section of the 1.7b release notes for more information. - (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes. Support for other versions of CP and CMS is unchanged from 1.6e. ************************** * External Compatibility * ************************** Release 1.7c is generally compatible with 1.7b, from the perspective of the end-user (including list managers and maintainers), and with the exception of the changes listed below. Release 1.7c introduces major changes to the LISTSERV internals, making compatibility with locally developed extensions or local modifications problematic. These changes are briefly described in the next section, and only affect sites which made alterations or additions to the LISTSERV code. Changes which affect all LISTSERV sites are highlighted below; note that more details are available, when appropriate, from the longer descriptive section of these release notes. 1. The obsolete NOTIFY command has been removed. This historic facility was related to the process of setting up peered lists in early versions of LISTSERV, and has not been required for years. A possible use of NOTIFY was to prevent LISTSERV from ever sending mail to certain local servers; this can also be accomplished with the TRAPOUT system variable. 2. The default for the "Mail-via=" list header keyword has been changed from "Direct" to "DISTRIBUTE". This change was mostly prompted by the fact that most list owners expected LISTSERV to optimize the use of network bandwidth by default. In addition, changes to the LISTSERV supervisor make it more efficient to use DISTRIBUTE even for lists with only local recipients. 3. The job processor no longer accepts multiple JOB cards per spool file. LISTSERV never made use of this facility, which raised a number of delicate data integrity and recovery issues that the former job processor did not address properly. Release 1.7c is to be installed directly on top of the base 1.7b code, and includes all the fixes available from LISTSERV@SEARN for release 1.7b (ie all the "17Bnnnt FIX" files from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits", which always have to be installed manually. ************************** * Internal Compatibility * ************************** Major changes have been made to the way LISTSERV deals with incoming reader files and processes jobs, and in particular DD statements. GLOBALV groups LSVPEND, LSVRDR and DDNAMES are no longer used for this purpose; any local code using variables in these groups will have to be modified to use the new LSVCD1 and LSVDD1 calls. For more information, see the new LISTSERV programming guide described later on. The following REXX files have been converted to PREXX with release 1.7c: LSVCMD LSVJCMD LSVKEYWD LSVLTELL LSVNMAT LSVNOTIF LSVPEND LSVRDR LSVRSCSP LSVSRVID LSVSTP LSVTELL The following MODULE files have become obsolete with release 1.7c: LSVWRF80 ************************** * Compatibility Warnings * ************************** This new section describes planned changes which may affect compatibility (both external and internal). The release in which the change is planned to be introduced is indicated, with the letter 'x' denoting an unknown release of the specified major version (not necessarily posterior to the last release explicitly mentioned). Note that conversion of more REXX files to PREXX is assumed to take place with each new release and is not indicated here. - (1.7d) LSVCARDR MODULE and LSVDDFID MODULE will be removed. You should remove calls to these modules from local applications as soon as possible. - (1.7d) The GETFID command will be removed. - (1.7d) The internal format of FILELIST/FILEID and attendant files such as AFDLIST FILE will change dramatically. Applications which alter them directly will no longer work. - (1.7e) The internal format of LIST and attendant files (CACHE, STATS, et al) will change dramatically. Applications which alter them directly will no longer work. - (1.7e) Support for "Mail-via= Direct" will be removed, causing all mail distribution to take place via DISTRIBUTE. - (1.7x) LSVDIAD4, LSVFRD1, LSVFWR1, RXLSVUHF and LSVPUNCH will be eventually removed. You should avoid using them. ******************************** * Minor fixes and enhancements * ******************************** This section has been removed for this release, due to the conversion of the LISTSERV supervisor to PREXX, which required numerous changes to most REXX files. Comparing a PREXX program with the old REXX file it replaces is next to impossible, while using DIFF on REXX files which have been changed in many places to interface with the new supervisor requires too much time. The new UPDATE-based maintenance scheme should make this task easier in future releases for the PREXX and assembler code. One "minor fix" worth mentioning and easily remembered is the inclusion of a more recent version of the PDUTIL utility, which LISTSERV uses to implement the UUENCODE format. Many thanks to John Fisher for having first allowed the use of PDUTIL by LISTSERV and gone to the trouble of adjusting it to fit the particular needs of LISTSERV, and then having further taken the time to remove a record length restriction in a program he had stopped using himself, for the benefit of LISTSERV users. John's address is [log in to unmask], in case you want to thank him yourself and also in the unlikely case you should encounter any problem with the new PDUTIL program (make sure to copy [log in to unmask] in that case if the problem is of a general nature). ********************************************************* * Serviceability/Performance: GLOBLIST FILE maintenance * ********************************************************* The "list of lists update" command processor has been partly rewritten in PREXX, yielding an improvement of a factor of 8-10 in terms of CPU time for normal X-LUPD jobs, exclusive of any time required to update the LISTS database (for sites which also run it). This is an important change because these jobs used to require about 30 seconds of CPU time on small machines, and a backbone server has to process many such jobs in the course of a normal day. Since this load is volume-independent, its impact was most noticeable on smaller machines, although, even on large machines processing thousands of DISTRIBUTE jobs a day, a lot of time could elapse before LISTSERV would be allowed to use up the 4-5 seconds of CPU time a X-LUPD job required. In addition, the new code will automatically delete obsolete entries belonging to sites which have stopped running LISTSERV or left the network, as soon as a PEERS NAMES update reflecting that change is received. This, again, does not apply to the LISTS database, which is not affected by this change at all. The LISTS database is already suffering from "index cancer" for other reasons, causing it to list certain entries more than once, or then not at all. This is due to a disagreement between the database manager and the driver of the LISTS database regarding what should be done to recover from certain error conditions; the more interactive searches your server processes for this database, the more likely this is to happen. While it would be possible to fix this problem, this would require a significant programming effort if a reasonable level of performance is to be kept. It has been decided not to address this problem at this time, as the LISTS database is in need of a major redesign for other reasons. The next paragraph contains some technical information about the present implementation of this database, which you may want to skip if you are not familiar with LISTSERV databases and internals. The LISTS driver presently uses one CMS file per list in the database. This clearly makes maintenance of list entries easy, as small files are just created, replaced or erased, as appropriate. Originally, with about 500 files, this system proved to perform reasonably well, at least on large enough systems with reasonably fast disk units (*not* 9335's) and after a small change to the database manager, to avoid keeping too many files open. When we reached about 1500 files, it became clear that sequential searches would be slow no matter what type of machine the database ran on, but it was hoped that newer versions of VM with minidisk caching and 3990 controllers would solve this problem. There are presently over 3000 CMS files and, while a large system with 3990-3 controllers and MDC could in principle comfortably handle such a load, in practice the operating system tends to allocate these caching resources sparingly and to reclaim them pretty fast - usually in less time than the usual "think pause", not to mention network delays. So, even though an idle mainframe was shown to perform a full sequential search in only 5 seconds and the next one in just 3, it is clear that this design will not scale down to anything smaller than a mainframe system, and something must be done about it. The LISTS database will be eventually redesigned and rewritten from scratch, with the problems mentioned above addressed by design rather than correctively. ****************************************************************** * Performance/Operational warning: DD manager rewritten in PREXX * ****************************************************************** One of the parts of the LISTSERV supervisor which was rewritten in PREXX is the DD manager - the code that handles DD job cards or, in more concrete terms, the actual data that DISTRIBUTE jobs carry. This change resulted in averaged I/O savings on the order of 50% for a typical medium-load backbone server, mostly within DISTRIBUTE. While this is obviously a good thing for the hardware, it also implies that LISTSERV is likely to require less wall-clock time to perform the same amount of DISTRIBUTE data duplication. This in turn can significantly alter the balance between LISTSERV and the VM mailer, which previously was generally able to process incoming files faster than LISTSERV could generate them, although if you looked at the respective reader queues with a short enough monitoring interval, LISTSERV would occasionally send files faster than the mailer could process them. With the mailer still I/O bound for processing incoming files and LISTSERV requiring (almost) only spool I/O, the situation may change dramatically if your system is I/O bound but not checkpoint-bound. This "balance" is relevant only when you have large backlogs, ie in "crisis" situations; during normal system operation, you will not notice any difference because there are only a handful of queued files and it does not matter where they spend most of their short waiting time. However, during a crisis you may want to monitor the mailer's reader queue carefully, and adjust priorities as appropriate (or place LISTSERV offline) to prevent them from overflowing. This behaviour will become even more apparent with release 1.7d, which will save a minimum of one extra disk I/O operation per incoming file and also require less CPU time to pre-format incoming jobs. This warning, however, will not be repeated in the 1.7d release notes. ************************************************* * Documentation: LISTFAQ MEMO and LISTPROG MEMO * ************************************************* Two new short manuals are being made available with release 1.7c. The first one, LISTFAQ MEMO, is available to any user via 'INFO FAQ' and addresses Frequently Asked Questions. LISTPROG MEMO, on the other hand, is available only from LISTSERV@SEARN and is restricted to LISTSERV maintainers. It contains limited information about LISTSERV internals and advice for people who wish to make changes to the software or to develop local extensions (new commands, extensive changes to the supplied exits, and so on). The remainder of this section concerns only EARN users. Various discussions that took place during the Istanbul EARN meetings in November 1991 led to the decision that EARN should take a more active role in providing end-user documentation, among other things for LISTEARN. This decision is the outcome of an old requirement from the user community, which the President of EARN had first promised to address in the users' meeting held in 1988 in Cesme, Turkey. Quoting from BOD19 92 (EARN Users Report to the Board of Directors - a public document): >On March 16-17, 1992, the entire EARN staff met at the EARN Office to >discuss the present state of EARN documentation and future plans. The >areas for which we need documentation were determined, priorities were >assigned to them, and individual staff members were assigned >responsibility for the various areas. The modules of documentation will >be available from a Listserv filelist and will be combined into large >documents, both electronic and hardcopy. If you are an EARN user, it is up to you to make it clear to EARN management whether you want documentation about LISTEARN (which would be mostly usable for LISTSERV as well) or about X.400/X.500/FTAM pilot services, RFC822 addressing, or whatever else you feel your institution's membership money would be best spent on. Bear in mind that documentation is already available from other sources for some of the topics mentioned above, while there are things which no other organization is likely to ever be able or want to document. ************************************************************* * Serviceability/Problem analysis: Enhanced error reporting * ************************************************************* An error reporting facility has been added to the LISTSERV supervisor and supporting PREXX library code. Errors generated by new supervisor and library routines can be analyzed automatically by LISTSERV and displayed in a more user-friendly form. For instance, the equivalent of "Error 2012 from LSVFWR1 XYZ TEST A1 1279823" would be: ------------------------------------------------------------------------- 19 Apr 1992 22:14:02 >>> Error X'00380063' writing file "XYZ TEST A1" <<< 19 Apr 1992 22:14:02 -> Severity: Error 19 Apr 1992 22:14:02 -> Facility: DMSERDBF - DMSEBLKW entry point 19 Apr 1992 22:14:02 -> Description: Disk or directory is R/O 19 Apr 1992 22:14:02 -> I/O mode: Block write ------------------------------------------------------------------------- When LISTSERV does not know the exact description, it can usually give a hint about where to find more information: ------------------------------------------------------------------------- 19 Apr 1992 22:35:14 >>> Error X'001002A3' writing file "X DBINDEX A" <<< 19 Apr 1992 22:35:14 -> Severity: Error 19 Apr 1992 22:35:14 -> Facility: FSWRITE 19 Apr 1992 22:35:14 -> Description: Unspecified error (84) - try HELP MACRO FSWRITE 19 Apr 1992 22:35:14 -> I/O mode: Record write ------------------------------------------------------------------------- Some system calls have hundreds of possible error codes, many of which cannot be described on a single line. To save disk space, LISTSERV has only been taught about the most common ones - those that are potentially meaningful to users of other operating systems, who do not know anything about VM internals. The breakup by return code and facility remains available in any case. Note that you can still get the "Error 2013" messages from existing REXX code; the new messages are generated only by the new PREXX code. For now, you will only find them in console logs, as they happen to be signalled only by routines which do not interact with command issuers. In future versions, you will start seeing these messages in "error traceback" sections of error notifications to end users. ************************************************************ * Serviceability: Change in source code maintenance scheme * ************************************************************ The remainder of these release notes describes a software maintenance changes which are of interest only to LISTSERV maintainers. If you are not in charge of running LISTSERV at your site, you may want to stop reading now. In order to improve ease of service, LISTSERV source code is now going to be mostly maintained via UPDATE. Before reacting to the surprising presence of the word 'mostly' in the previous sentence, with all the confusion that it implies, please pause a few seconds to consider the implications of the generalized use of UPDATE (in a manner similar to that of CP or CMS) on a system with an ever-growing amount of source code and well over a hundred recipients of source code distributions, sent over the network and not on a tape cartridge. Release 1.7c includes 23827 lines of source code, not counting "foreign" utilities such as CARD or PDUTIL, and still there remain 30114 lines of REXX code which, when (eventually) converted, are very unlikely to yield less than 100000 lines of extra source code. Due to lack of the kind of industrial-strength manpower that a straightforward "complete rewrite" would require, the conversion to PREXX is a staged one, with changes made to both sides little by little until a particular function can be entirely migrated to PREXX and the REXX program erased. In other words, some source files are going to take as many hits as your average CMS6 source file. On the average, each line of code is touched a bit less than once after being written in the first place, and might have to be released before the process is complete; we are talking about updates that would have about the same size as the original file. Of course, every time a source file is changed and a new UPDATE file is generated, the size of the data that has to be shipped with the next release is going to increase, not to mention the size of the "complete distribution" shipment for new sites and sites which "missed an update"; we may also run out of sequencing space, and program development tends to get seriously inconvenient when the amount of update files gets out of hand. Eventually we will have a serious problem, which will be solved the usual way (./ S) and make everyone unhappy. Since UPDATE does have very real merits but the traditional approach is not too good in the context of a network-based distribution system and a development profile in which new source files grow up little by little, rather than suddenly showing up in a PUT tape, a hybrid system has been chosen. New source files will start as unserialized files (RECFM V when applicable), and grow at their natural rate, undergoing the staged changes mentioned above, until they reach a reasonable level of maturity and are converted to serialized files. While this makes them a nuisance to modify, this is because they really ARE a nuisance to modify - significant changes are already planned, and any modification you might make would stand good chances of needing careful adjustments with the next release. Old source files whose removal is already planned will be left unserialized, as a serialized file uses up more bandwidth (the sequence field cannot be compressed into a run of blanks) and no change will be made unless a critically important problem shows up. All other existing source files will be serialized and maintained via UPDATE. So far, 7 ASSEMBLE files out of 34 are unserialized, and 2 PASCAL files out of 24 (not counting the sample program LSWTST). The amount of unserialized ASSEMBLE files should decrease regularly, while there is no telling how many RECFM V PASCAL files a given release will include. In order to make life simpler, macros are still updated by replacement: as before, you will only get LSVLIB MACLIB, and not the many individual MACRO and COPY files. You are not expected to need to make changes to them (there is no XYZBLOK to which one simply *has* to add an extra doubleword for an MVS-imported accounting package), and requiring all recipients of a source distribution to run VMFMAC is not very reasonable; not even IBM requires you to do that for their products, for reasons you will immediately understand if you try it once. In addition, most LISTSERV macros are small enough to be easily modified by replacement: just rewrite it the way you want it to be (presumably with "preferred" CMS macros), and place it in a MACLIB with a higher search precedence. REXX files are considered to be object files, which furthermore are being slowly phased out, and will continue to be updated by replacement. Converting them to a format compatible with EXECUPDT would require a major amount of editing work, which is simply not justified in view of their planned removal. Changes are expected to be implemented as individual UPDATE files during development, but these files will be concatenated and compressed when a new release is issued, to ensure that at most one UPDATE file per source file is sent for each new release. Exceptions may be made for changes which have not been carefully tested or which one might expect people to want to remove temporarily. Comments in the AUX files will list the original update "titles" to make troubleshooting easier. SID codes will not be used because they are a serious nuisance in PASCAL (where at least 4 characters must be used for comment separators, and where normal instructions do not generally end around column 30). In addition, they are a non-negligible waste of bandwidth in our context. Finally, this change requires a reorganization of the source shipments. Instead of a single ASMnnt shipment containing replacement files for all existing source code, three types of shipments will be available: - ISRCnnt: initial source shipment sent to new LISTSERV sites, and containing basically the same files as the old ASMnnt decks. - SRCnnt: incremental shipment of replacement source files for the release in question. SRC17C will contain all source files, but SRC17D will only contain new files and any unserialized file which may have changed (or been serialized) since SRC17C. - AUXnnt: non-incremental shipment of replacement AUX and UPD files, plus the current control files and LSVLIB MACLIB. AUX17C will be mostly empty. The reason for using non-incremental AUX shipments is that it makes maintenance easier at the receiving end, and that most AUX files have to be reshipped anyway. You can load the latest AUX shipment and be sure that you have the latest, complete set of AUX and update files, plus the current MACLIB. If you are missing the source file they apply to, you will find out soon enough.