Description of the changes for release 1.7c of LISTSERV
-------------------------------------------------------
April 20th, 1992
**************
* Highlights *
**************
Version 1.7c introduces the following major enhancements:
- Support for CMS release 8, and improved support for CMS release 7 and
the Shared File System (SFS). Due to its generally observed lack of
robustness and stability, CMS release 6 is still not formally
supported; problems that cannot be reproduced under CMS release 7 will
not be addressed.
- Significant parts of the LISTSERV supervisor have been rewritten in
PREXX, providing a stable basis for future developments while improving
robustness and performance. A typical backbone server with medium load
can expect to save 20% on CPU time (TCPU) and to observe a reduction of
I/O operations by about 50%.
- The maintenance of the "list of lists" (GLOBLIST FILE), a rather
frequent and previously very expensive process in terms of CPU time,
has been converted to PREXX. The resulting eight to tenfold reduction
in CPU cost removes a potential obstacle to joining the backbone for
small systems.
- Source code will be maintained using UPDATE, whenever practical, from
release 1.7d onwards (that is, release 1.7c source files have been
sequenced to allow future changes to be made with UPDATE).
- Relief for CMS release 6 sites suffering from "Interpreter failure"
errors and unable/unwilling to apply the IBM APAR correcting the
problem. Changes made in release 1.7c chanced to remove the sequence of
instructions that usually triggered the error.
****************
* Availability *
****************
Release 1.7 is available free of charge to sites which qualify under one
(or more) of the categories listed below:
a. Members of CREN or one of the CREN Cooperating Networks, excluding:
1. Members of the European Academic and Research Network (EARN).
2. Members whose computing facilities are located in an EEC country.
b. Members of EARN, CREN or a CREN Cooperating Network whose computing
facilities are located in a nordic country (Denmark, Finland, Iceland,
Norway, Sweden).
c. Members of EARN whose computing facilities are not located in an EEC
country and which have formally applied (in writing) for a license,
and complied with the appropriate EARN directives. International
organizations are not considered as belonging to the EEC, regardless
of where their computing facilities are physically located.
Release 1.6 licenses from EEC countries remain valid for release 1.6, but
will not be carried over to release 1.7.
*******************************
* Supported operating systems *
*******************************
Release 1.7 introduces the following changes to the list of supported
operating systems:
- (1.7a) Withdrawal of support for CMS release 3.
- (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA.
- (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes.
- (1.7b) Support for the Shared File System, with a few restrictions
concerning the setup of the A and D-disks and the default filepool
setting. See the corresponding section of the 1.7b release notes for
more information.
- (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes.
Support for other versions of CP and CMS is unchanged from 1.6e.
**************************
* External Compatibility *
**************************
Release 1.7c is generally compatible with 1.7b, from the perspective of
the end-user (including list managers and maintainers), and with the
exception of the changes listed below. Release 1.7c introduces major
changes to the LISTSERV internals, making compatibility with locally
developed extensions or local modifications problematic. These changes
are briefly described in the next section, and only affect sites which
made alterations or additions to the LISTSERV code. Changes which affect
all LISTSERV sites are highlighted below; note that more details are
available, when appropriate, from the longer descriptive section of these
release notes.
1. The obsolete NOTIFY command has been removed. This historic facility
was related to the process of setting up peered lists in early
versions of LISTSERV, and has not been required for years. A possible
use of NOTIFY was to prevent LISTSERV from ever sending mail to
certain local servers; this can also be accomplished with the TRAPOUT
system variable.
2. The default for the "Mail-via=" list header keyword has been changed
from "Direct" to "DISTRIBUTE". This change was mostly prompted by the
fact that most list owners expected LISTSERV to optimize the use of
network bandwidth by default. In addition, changes to the LISTSERV
supervisor make it more efficient to use DISTRIBUTE even for lists
with only local recipients.
3. The job processor no longer accepts multiple JOB cards per spool
file. LISTSERV never made use of this facility, which raised a number
of delicate data integrity and recovery issues that the former job
processor did not address properly.
Release 1.7c is to be installed directly on top of the base 1.7b code,
and includes all the fixes available from LISTSERV@SEARN for release 1.7b
(ie all the "17Bnnnt FIX" files from filelist "LFIXES"), with the
exception of those containing replacement "recommended user exits", which
always have to be installed manually.
**************************
* Internal Compatibility *
**************************
Major changes have been made to the way LISTSERV deals with incoming
reader files and processes jobs, and in particular DD statements. GLOBALV
groups LSVPEND, LSVRDR and DDNAMES are no longer used for this purpose;
any local code using variables in these groups will have to be modified
to use the new LSVCD1 and LSVDD1 calls. For more information, see the new
LISTSERV programming guide described later on.
The following REXX files have been converted to PREXX with release 1.7c:
LSVCMD LSVJCMD LSVKEYWD LSVLTELL LSVNMAT LSVNOTIF
LSVPEND LSVRDR LSVRSCSP LSVSRVID LSVSTP LSVTELL
The following MODULE files have become obsolete with release 1.7c:
LSVWRF80
**************************
* Compatibility Warnings *
**************************
This new section describes planned changes which may affect compatibility
(both external and internal). The release in which the change is planned
to be introduced is indicated, with the letter 'x' denoting an unknown
release of the specified major version (not necessarily posterior to the
last release explicitly mentioned). Note that conversion of more REXX
files to PREXX is assumed to take place with each new release and is not
indicated here.
- (1.7d) LSVCARDR MODULE and LSVDDFID MODULE will be removed. You should
remove calls to these modules from local applications as soon as
possible.
- (1.7d) The GETFID command will be removed.
- (1.7d) The internal format of FILELIST/FILEID and attendant files such
as AFDLIST FILE will change dramatically. Applications which alter them
directly will no longer work.
- (1.7e) The internal format of LIST and attendant files (CACHE, STATS,
et al) will change dramatically. Applications which alter them directly
will no longer work.
- (1.7e) Support for "Mail-via= Direct" will be removed, causing all mail
distribution to take place via DISTRIBUTE.
- (1.7x) LSVDIAD4, LSVFRD1, LSVFWR1, RXLSVUHF and LSVPUNCH will be
eventually removed. You should avoid using them.
********************************
* Minor fixes and enhancements *
********************************
This section has been removed for this release, due to the conversion of
the LISTSERV supervisor to PREXX, which required numerous changes to most
REXX files. Comparing a PREXX program with the old REXX file it replaces
is next to impossible, while using DIFF on REXX files which have been
changed in many places to interface with the new supervisor requires too
much time. The new UPDATE-based maintenance scheme should make this task
easier in future releases for the PREXX and assembler code.
One "minor fix" worth mentioning and easily remembered is the inclusion
of a more recent version of the PDUTIL utility, which LISTSERV uses to
implement the UUENCODE format. Many thanks to John Fisher for having
first allowed the use of PDUTIL by LISTSERV and gone to the trouble of
adjusting it to fit the particular needs of LISTSERV, and then having
further taken the time to remove a record length restriction in a program
he had stopped using himself, for the benefit of LISTSERV users. John's
address is [log in to unmask], in case you want to thank him yourself
and also in the unlikely case you should encounter any problem with the
new PDUTIL program (make sure to copy [log in to unmask] in that
case if the problem is of a general nature).
*********************************************************
* Serviceability/Performance: GLOBLIST FILE maintenance *
*********************************************************
The "list of lists update" command processor has been partly rewritten in
PREXX, yielding an improvement of a factor of 8-10 in terms of CPU time
for normal X-LUPD jobs, exclusive of any time required to update the
LISTS database (for sites which also run it). This is an important change
because these jobs used to require about 30 seconds of CPU time on small
machines, and a backbone server has to process many such jobs in the
course of a normal day. Since this load is volume-independent, its impact
was most noticeable on smaller machines, although, even on large machines
processing thousands of DISTRIBUTE jobs a day, a lot of time could elapse
before LISTSERV would be allowed to use up the 4-5 seconds of CPU time a
X-LUPD job required.
In addition, the new code will automatically delete obsolete entries
belonging to sites which have stopped running LISTSERV or left the
network, as soon as a PEERS NAMES update reflecting that change is
received. This, again, does not apply to the LISTS database, which is not
affected by this change at all.
The LISTS database is already suffering from "index cancer" for other
reasons, causing it to list certain entries more than once, or then not
at all. This is due to a disagreement between the database manager and
the driver of the LISTS database regarding what should be done to recover
from certain error conditions; the more interactive searches your server
processes for this database, the more likely this is to happen. While it
would be possible to fix this problem, this would require a significant
programming effort if a reasonable level of performance is to be kept. It
has been decided not to address this problem at this time, as the LISTS
database is in need of a major redesign for other reasons. The next
paragraph contains some technical information about the present
implementation of this database, which you may want to skip if you are
not familiar with LISTSERV databases and internals.
The LISTS driver presently uses one CMS file per list in the database.
This clearly makes maintenance of list entries easy, as small files are
just created, replaced or erased, as appropriate. Originally, with about
500 files, this system proved to perform reasonably well, at least on
large enough systems with reasonably fast disk units (*not* 9335's) and
after a small change to the database manager, to avoid keeping too many
files open. When we reached about 1500 files, it became clear that
sequential searches would be slow no matter what type of machine the
database ran on, but it was hoped that newer versions of VM with minidisk
caching and 3990 controllers would solve this problem. There are
presently over 3000 CMS files and, while a large system with 3990-3
controllers and MDC could in principle comfortably handle such a load, in
practice the operating system tends to allocate these caching resources
sparingly and to reclaim them pretty fast - usually in less time than the
usual "think pause", not to mention network delays. So, even though an
idle mainframe was shown to perform a full sequential search in only 5
seconds and the next one in just 3, it is clear that this design will not
scale down to anything smaller than a mainframe system, and something
must be done about it. The LISTS database will be eventually redesigned
and rewritten from scratch, with the problems mentioned above addressed
by design rather than correctively.
******************************************************************
* Performance/Operational warning: DD manager rewritten in PREXX *
******************************************************************
One of the parts of the LISTSERV supervisor which was rewritten in PREXX
is the DD manager - the code that handles DD job cards or, in more
concrete terms, the actual data that DISTRIBUTE jobs carry. This change
resulted in averaged I/O savings on the order of 50% for a typical
medium-load backbone server, mostly within DISTRIBUTE. While this is
obviously a good thing for the hardware, it also implies that LISTSERV is
likely to require less wall-clock time to perform the same amount of
DISTRIBUTE data duplication. This in turn can significantly alter the
balance between LISTSERV and the VM mailer, which previously was
generally able to process incoming files faster than LISTSERV could
generate them, although if you looked at the respective reader queues
with a short enough monitoring interval, LISTSERV would occasionally send
files faster than the mailer could process them.
With the mailer still I/O bound for processing incoming files and
LISTSERV requiring (almost) only spool I/O, the situation may change
dramatically if your system is I/O bound but not checkpoint-bound. This
"balance" is relevant only when you have large backlogs, ie in "crisis"
situations; during normal system operation, you will not notice any
difference because there are only a handful of queued files and it does
not matter where they spend most of their short waiting time. However,
during a crisis you may want to monitor the mailer's reader queue
carefully, and adjust priorities as appropriate (or place LISTSERV
offline) to prevent them from overflowing.
This behaviour will become even more apparent with release 1.7d, which
will save a minimum of one extra disk I/O operation per incoming file and
also require less CPU time to pre-format incoming jobs. This warning,
however, will not be repeated in the 1.7d release notes.
*************************************************
* Documentation: LISTFAQ MEMO and LISTPROG MEMO *
*************************************************
Two new short manuals are being made available with release 1.7c. The
first one, LISTFAQ MEMO, is available to any user via 'INFO FAQ' and
addresses Frequently Asked Questions. LISTPROG MEMO, on the other hand,
is available only from LISTSERV@SEARN and is restricted to LISTSERV
maintainers. It contains limited information about LISTSERV internals and
advice for people who wish to make changes to the software or to develop
local extensions (new commands, extensive changes to the supplied exits,
and so on).
The remainder of this section concerns only EARN users. Various
discussions that took place during the Istanbul EARN meetings in November
1991 led to the decision that EARN should take a more active role in
providing end-user documentation, among other things for LISTEARN. This
decision is the outcome of an old requirement from the user community,
which the President of EARN had first promised to address in the users'
meeting held in 1988 in Cesme, Turkey. Quoting from BOD19 92 (EARN Users
Report to the Board of Directors - a public document):
>On March 16-17, 1992, the entire EARN staff met at the EARN Office to
>discuss the present state of EARN documentation and future plans. The
>areas for which we need documentation were determined, priorities were
>assigned to them, and individual staff members were assigned
>responsibility for the various areas. The modules of documentation will
>be available from a Listserv filelist and will be combined into large
>documents, both electronic and hardcopy.
If you are an EARN user, it is up to you to make it clear to EARN
management whether you want documentation about LISTEARN (which would be
mostly usable for LISTSERV as well) or about X.400/X.500/FTAM pilot
services, RFC822 addressing, or whatever else you feel your institution's
membership money would be best spent on. Bear in mind that documentation
is already available from other sources for some of the topics mentioned
above, while there are things which no other organization is likely to
ever be able or want to document.
*************************************************************
* Serviceability/Problem analysis: Enhanced error reporting *
*************************************************************
An error reporting facility has been added to the LISTSERV supervisor and
supporting PREXX library code. Errors generated by new supervisor and
library routines can be analyzed automatically by LISTSERV and displayed
in a more user-friendly form. For instance, the equivalent of "Error 2012
from LSVFWR1 XYZ TEST A1 1279823" would be:
-------------------------------------------------------------------------
19 Apr 1992 22:14:02 >>> Error X'00380063' writing file "XYZ TEST A1" <<<
19 Apr 1992 22:14:02 -> Severity: Error
19 Apr 1992 22:14:02 -> Facility: DMSERDBF - DMSEBLKW entry point
19 Apr 1992 22:14:02 -> Description: Disk or directory is R/O
19 Apr 1992 22:14:02 -> I/O mode: Block write
-------------------------------------------------------------------------
When LISTSERV does not know the exact description, it can usually give a
hint about where to find more information:
-------------------------------------------------------------------------
19 Apr 1992 22:35:14 >>> Error X'001002A3' writing file "X DBINDEX A" <<<
19 Apr 1992 22:35:14 -> Severity: Error
19 Apr 1992 22:35:14 -> Facility: FSWRITE
19 Apr 1992 22:35:14 -> Description: Unspecified error (84) - try HELP
MACRO FSWRITE
19 Apr 1992 22:35:14 -> I/O mode: Record write
-------------------------------------------------------------------------
Some system calls have hundreds of possible error codes, many of which
cannot be described on a single line. To save disk space, LISTSERV has
only been taught about the most common ones - those that are potentially
meaningful to users of other operating systems, who do not know anything
about VM internals. The breakup by return code and facility remains
available in any case.
Note that you can still get the "Error 2013" messages from existing REXX
code; the new messages are generated only by the new PREXX code. For now,
you will only find them in console logs, as they happen to be signalled
only by routines which do not interact with command issuers. In future
versions, you will start seeing these messages in "error traceback"
sections of error notifications to end users.
************************************************************
* Serviceability: Change in source code maintenance scheme *
************************************************************
The remainder of these release notes describes a software maintenance
changes which are of interest only to LISTSERV maintainers. If you are
not in charge of running LISTSERV at your site, you may want to stop
reading now.
In order to improve ease of service, LISTSERV source code is now going to
be mostly maintained via UPDATE. Before reacting to the surprising
presence of the word 'mostly' in the previous sentence, with all the
confusion that it implies, please pause a few seconds to consider the
implications of the generalized use of UPDATE (in a manner similar to
that of CP or CMS) on a system with an ever-growing amount of source code
and well over a hundred recipients of source code distributions, sent
over the network and not on a tape cartridge. Release 1.7c includes 23827
lines of source code, not counting "foreign" utilities such as CARD or
PDUTIL, and still there remain 30114 lines of REXX code which, when
(eventually) converted, are very unlikely to yield less than 100000 lines
of extra source code. Due to lack of the kind of industrial-strength
manpower that a straightforward "complete rewrite" would require, the
conversion to PREXX is a staged one, with changes made to both sides
little by little until a particular function can be entirely migrated to
PREXX and the REXX program erased. In other words, some source files are
going to take as many hits as your average CMS6 source file. On the
average, each line of code is touched a bit less than once after being
written in the first place, and might have to be released before the
process is complete; we are talking about updates that would have about
the same size as the original file.
Of course, every time a source file is changed and a new UPDATE file is
generated, the size of the data that has to be shipped with the next
release is going to increase, not to mention the size of the "complete
distribution" shipment for new sites and sites which "missed an update";
we may also run out of sequencing space, and program development tends to
get seriously inconvenient when the amount of update files gets out of
hand. Eventually we will have a serious problem, which will be solved the
usual way (./ S) and make everyone unhappy.
Since UPDATE does have very real merits but the traditional approach is
not too good in the context of a network-based distribution system and a
development profile in which new source files grow up little by little,
rather than suddenly showing up in a PUT tape, a hybrid system has been
chosen. New source files will start as unserialized files (RECFM V when
applicable), and grow at their natural rate, undergoing the staged
changes mentioned above, until they reach a reasonable level of maturity
and are converted to serialized files. While this makes them a nuisance
to modify, this is because they really ARE a nuisance to modify -
significant changes are already planned, and any modification you might
make would stand good chances of needing careful adjustments with the
next release.
Old source files whose removal is already planned will be left
unserialized, as a serialized file uses up more bandwidth (the sequence
field cannot be compressed into a run of blanks) and no change will be
made unless a critically important problem shows up. All other existing
source files will be serialized and maintained via UPDATE. So far, 7
ASSEMBLE files out of 34 are unserialized, and 2 PASCAL files out of 24
(not counting the sample program LSWTST). The amount of unserialized
ASSEMBLE files should decrease regularly, while there is no telling how
many RECFM V PASCAL files a given release will include.
In order to make life simpler, macros are still updated by replacement:
as before, you will only get LSVLIB MACLIB, and not the many individual
MACRO and COPY files. You are not expected to need to make changes to
them (there is no XYZBLOK to which one simply *has* to add an extra
doubleword for an MVS-imported accounting package), and requiring all
recipients of a source distribution to run VMFMAC is not very reasonable;
not even IBM requires you to do that for their products, for reasons you
will immediately understand if you try it once. In addition, most
LISTSERV macros are small enough to be easily modified by replacement:
just rewrite it the way you want it to be (presumably with "preferred"
CMS macros), and place it in a MACLIB with a higher search precedence.
REXX files are considered to be object files, which furthermore are being
slowly phased out, and will continue to be updated by replacement.
Converting them to a format compatible with EXECUPDT would require a
major amount of editing work, which is simply not justified in view of
their planned removal.
Changes are expected to be implemented as individual UPDATE files during
development, but these files will be concatenated and compressed when a
new release is issued, to ensure that at most one UPDATE file per source
file is sent for each new release. Exceptions may be made for changes
which have not been carefully tested or which one might expect people to
want to remove temporarily. Comments in the AUX files will list the
original update "titles" to make troubleshooting easier. SID codes will
not be used because they are a serious nuisance in PASCAL (where at least
4 characters must be used for comment separators, and where normal
instructions do not generally end around column 30). In addition, they
are a non-negligible waste of bandwidth in our context.
Finally, this change requires a reorganization of the source shipments.
Instead of a single ASMnnt shipment containing replacement files for all
existing source code, three types of shipments will be available:
- ISRCnnt: initial source shipment sent to new LISTSERV sites, and
containing basically the same files as the old ASMnnt decks.
- SRCnnt: incremental shipment of replacement source files for the
release in question. SRC17C will contain all source files, but SRC17D
will only contain new files and any unserialized file which may have
changed (or been serialized) since SRC17C.
- AUXnnt: non-incremental shipment of replacement AUX and UPD files, plus
the current control files and LSVLIB MACLIB. AUX17C will be mostly
empty.
The reason for using non-incremental AUX shipments is that it makes
maintenance easier at the receiving end, and that most AUX files have to
be reshipped anyway. You can load the latest AUX shipment and be sure
that you have the latest, complete set of AUX and update files, plus the
current MACLIB. If you are missing the source file they apply to, you
will find out soon enough.
|
