Description of the changes for release 1.7d of LISTSERV
-------------------------------------------------------
September 14th, 1992
**************
* Highlights *
**************
Version 1.7d introduces the following major enhancements:
- Significant parts of the LISTSERV supervisor have been rewritten in
PREXX, providing a stable basis for future developments while improving
robustness and performance. The main performance improvements for this
release include the LIST GLOBAL, REVIEW and NODESGEN commands,
generation of internal tables such as PEERS and DOMAIN NAMESUM, parsing
of RFC822 headers, domain-style address resolution, maintenance of
large lists and processing of incoming job files in Netdata format.
- Support for the ':oldnode.'/':newnode.' tags of BITEARN NODES, which
make it possible to automatically alter list and file subscriptions for
nodes that leave BITNET or change their nodeid.
- Automatic conversion of Netdata, Disk Dump or Card Dump DISTRIBUTE jobs
to plain text when the recipient is only reachable via mail.
- Improved subject in notifications sent by LISTSERV to list owners
simplify list maintenance.
- Relief for sites suffering from problems with the XEDIT SORT command
and/or XEDIT failures when building the PEERS NAMESUM file, and
unable/unwilling to apply the IBM APAR correcting the problem. Release
1.7d no longer uses XEDIT for sorting internal tables.
- The REXX code has been updated in preparation for an incompatible
change to the REXX interpreter which will probably be introduced with
CMS release 9 (no official statement yet). This does not mean that 1.7d
supports CMS9 - this will not be possible until CMS9 becomes available.
The only thing that can be said is than older versions of LISTSERV will
definitely not work under CMS9 (if the REXX change is introduced then).
****************
* Availability *
****************
Release 1.7 is available free of charge to sites which qualify under one
(or more) of the categories listed below:
a. Members of CREN or one of the CREN Cooperating Networks, excluding:
1. Members of the European Academic and Research Network (EARN).
2. Members whose computing facilities are located in an EEC country.
b. Members of EARN, CREN or a CREN Cooperating Network whose computing
facilities are located in a nordic country (Denmark, Finland, Iceland,
Norway, Sweden).
c. Members of EARN whose computing facilities are not located in an EEC
country and which have formally applied (in writing) for a license,
and complied with the appropriate EARN directives. International
organizations are not considered as belonging to the EEC, regardless
of where their computing facilities are physically located.
Release 1.6 licenses from EEC countries remain valid for release 1.6, but
will not be carried over to release 1.7.
*******************************
* Supported operating systems *
*******************************
Release 1.7 introduces the following changes to the list of supported
operating systems:
- (1.7a) Withdrawal of support for CMS release 3.
- (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA.
- (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes.
- (1.7b) Support for the Shared File System, with a few restrictions
concerning the setup of the A and D-disks and the default filepool
setting. See the corresponding section of the 1.7b release notes for
more information.
- (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes.
Support for other versions of CP and CMS is unchanged from 1.6e.
**************************
* External Compatibility *
**************************
Release 1.7d is generally compatible with 1.7c, from the perspective of
the end-user (including list managers and maintainers), and with the
exception of the changes listed below. Release 1.7d introduces major
changes to the LISTSERV internals, making compatibility with locally
developed extensions or local modifications problematic. These changes
are briefly described in the next section, and only affect sites which
made alterations or additions to the LISTSERV code. Changes which affect
all LISTSERV sites are highlighted below; note that more details are
available, when appropriate, from the longer descriptive section of these
release notes.
1. The obsolete GETFID command has been removed, as announced in the
1.7c release notes. This command had been added for the benefit of a
software package which was never completed or distributed.
2. The DISTOFF configuration variable is no longer available.
3. The "Formcheck=" list header keyword is no longer honoured; all lists
now operate as per "Formcheck= No". This keyword, which in retrospect
was an incentive for BITNET nodes to install mail user interfaces
compatible with RFC822 or at least IBM NOTE, is now causing a lot of
confusion to little avail.
4. The resource usage summary produced by the DISTRIBUTE command when
used with the DEBUG option has been removed. Its purpose was to
separate the resource usage of the DISTRIBUTE process itself from
that of the job processor. The job processor overhead has become
negligible, whereas the REXX exit overhead, not accounted for in the
DISTRIBUTE resource summary, has become a significant factor.
5. When sending a non-mail file to a list with "Files= Yes" and "Send=
Editor", LISTSERV will no longer forward the file to the editors, as
this turned out to generate a lot of confusion, especially when the
editor was on a non-BITNET machine. Instead, LISTSERV informs the
file originator that he should submit the file to the editors
himself.
6. The COUNTRY option of the REVIEW command produces slightly different
output (see the section on the REVIEW command below).
Release 1.7d is to be installed directly on top of the base 1.7c code,
and includes all the fixes available from LISTSERV@SEARN for release 1.7c
(ie all the "17Cnnnt FIX" files from filelist "LFIXES"), with the
exception of those containing replacement "recommended user exits", which
always have to be installed manually.
**************************
* Internal Compatibility *
**************************
Major changes have been made to LISTSERV internals. The most important
ones are briefly described below; refer to the LSTSRV-L archives for more
information.
1. LIST files are no longer in sort order, and may contain blank lines.
2. BITEARN INTNJE and BITEARN NODESUM2 have been replaced with BITEARN
NODESUM3. Standalone utilities such as RXLSVNNI will continue to
operate outside the LISTSERV environment, provided they have access
to the new file and to the new LSWNS3 MODULE.
3. SIGNUP FILE and PWLIST FILE have been replaced with SIGNUP2 FILE,
which must be manipulated exclusively through the LSVU@N function.
Please note that /WHOIS and /WHERE are not supported by the author of
LISTSERV. Contact their respective authors for information.
4. LSVBESTP EXEC has been replaced with a LSVBSP PREXX entry point which
is only partly compatible (LSVBESTP used to blindly INTERPRET one of
its arguments).
5. LSVRECV now supports CARD format and generates RECFM V disk files for
PUNCH format input files.
6. Authors of local command packages should note that LISTSERV does not
guarantee that the program stack is in any particular state when the
command processor is invoked. MAKEBUF and DROPBUF should be used
wherever necessary.
The following REXX files have been converted to PREXX with release 1.7d:
LSVADM LSVALLOW LSVBESTP LSVCKB LSVDNSUM LSVFSEND
LSVGFD LSVMONAD LSVOWN LSVPRM LSVRECV LSVSORT
LSVU@N LSV82I
The following MODULE files have become obsolete with release 1.7d:
LSVCARDR LSVDDFID RXLSVSPU
**************************
* Compatibility Warnings *
**************************
This new section describes planned changes which may affect compatibility
(both external and internal). The release in which the change is planned
to be introduced is indicated, with the letter 'x' denoting an unknown
release of the specified major version (not necessarily posterior to the
last release explicitly mentioned). Note that conversion of more REXX
files to PREXX is assumed to take place with each new release and is not
indicated here.
- (1.7e) The internal format of FILELIST/FILEID and attendant files such
as AFDLIST FILE will change dramatically. Applications which alter them
directly will no longer work.
- (1.7x) Support for "Mail-via= Direct" will be removed, causing all mail
distribution to take place via DISTRIBUTE.
- (1.7x) The internal format of LIST and attendant files (CACHE, STATS,
et al) will change dramatically. Applications which alter them directly
will no longer work.
- (1.7x) LSVDIAD4, LSVFRD1, LSVFWR1, RXLSVUHF and LSVPUNCH will be
eventually removed. You should avoid using them.
********************************
* Minor fixes and enhancements *
********************************
REMINDER: Manual installation is not supported for release 1.7. Automatic
installation with local password validation should be used instead. Sites
which insist on installing manually should note that shipments contain an
installation pre- and post-processing exit. If that exit is not executed,
or if it is executed in a different order or environment than during
automatic installation, serious problems may occur. In the first releases
of 1.7, this happened to only cause obsolete files not to be erased, thus
the only impact was loss of disk space. With release 1.7d, the impact is
much more serious.
This section contains a brief description of all the "small yet not
unimportant worries" that have been fixed in release 1.7d, along with the
affected releases, if known ("All" meaning "any release in which the
facility is available", ie the problem was present since the very
beginning).
Minor enhancements which do not warrant a detailed description are also
included; they are all located at the bottom of the list, and can be
identified by the word "New" in the "Affected releases" column.
Affected
releases Description of the bug, problem or enhancement
-------- ----------------------------------------------
1.7a Date field incorrect in BITEARN database for non-NJE entries.
1.7a CPU usage ratio reported by the SHOW command was incorrect
under VM/SP and HPO due to the session time counters being
reset by system accounting checkpoints (CP ACNT), which (as
seen by LISTSERV) can take place at arbitrary intervals. CPU
usage had to be calculated from session CPU and CONNECT time,
yielding a result that, while valid, did not correspond to
the time since the last LISTSERV reboot. Under VM/XA and ESA,
the result could be invalid (over 100%) because accounting
checkpoints do not reset session counters and their value can
exceed the capacity of the fixed-width QUERY TIME fields. A
more sophisticated method is being used to solve the problem
for both SP/HPO and XA/ESA systems.
All '.BITNET' suffix was not consistently removed everywhere it
should have been.
All DISTRIBUTE FILE to mail-only recipient used to set the SMTP
'MAIL FROM:' to the LISTSERV address, thus causing delivery
errors to be indirectly sent to the local postmaster, who
could do nothing about them. The return address is now set
to point to the job originator.
New The 'SHOW NODEntry' command can now display member and site
entry. It was previously limited to NJE entries only.
New The new 'SHOW NAD nodeid' command will display a list of the
addresses LISTSERV recognizes as node administrators for the
node in question.
New The node description and nodeid are now displayed when a
PRINT ADDRESS or PRINT CONTACT report is requested after
searching the BITEARN database.
New (Cosmetics) Comments in RFC822 address fields not properly
concatenated to "name" field. Warning messages could be
repeated without apparent reason if the parser had to use a
given RFC822 field for multiple purposes. Some parser
messages improved.
New MYDOMAIN configuration variable auto-configured from local
node entry in BITEARN NODES if ':internet.' tag available.
MYDOMAIN is vitally important to the mailing loop detector.
New Allow 'QUERY * FOR user' and 'SET * FOR user' for list owners
as well. Lists not owned by the invoker are of course not
searched/updated.
New Because MAILFORM files are actually programs, quotes have to
be doubled when a single quote is desired in the message (as
in "It''s a good idea to..."). Failure to do so usually
caused a REXX error, which was reported to the postmaster and
to LISTSERV coordination. Postmasters usually assumed it was
a bug in LISTSERV and took no action. Mispaired quotes will
no longer cause a REXX error; instead, a warning will be
inserted in the text indicating that unpaired quotes have
been detected and suggesting to contact the list owner. Note
that in some cases the error cannot be detected. This is a
temporary solution, pending the replacement of MAILFORM's by
a more robust (but less flexible) system.
New To make life easier for list owners and moderators using
computer systems which only accept lower-case addresses, the
"Owner=" and "Editor=" keywords now accept lower-case
usernames provided that the username is enclosed in double
quotation marks and stands on a line by its own. Example:
Owner= "eric"@SEARN.SUNET.SE works
Owner= "[log in to unmask]" does NOT work
Owner= joe@xyzvm,"jack"@xyz.edu does NOT work
The requirement to use the first listed form only is a
temporary restriction pending some additional changes to the
LISTSERV internals. Be careful when updating the headers of
peered lists: this syntax only works with 1.7d. Note that
LISTSERV never supported blanks in user names, quoted or not.
New Entries for deleted nodes are now removed from SIGNUP2 file
during the NODESGEN process, to save disk space.
New The default values for FILEMAXL and MAILMAXL, which dated
back to 1986, have been doubled. Since 1.7c, a backbone
server which does not issue warnings about lack of storage at
startup should be able to process 20,000 lines jobs.
****************************************************************
* Performance/Reliability: Changes to BITEARN NODES processing *
****************************************************************
The BITEARN INTNJE and BITEARN NODESUM2 files have been replaced with a
new file, BITEARN NODESUM3, designed to easily accomodate additional
tables as the need arises. Four new tables have already been
incorportated into the file.
Information about administrative networks (CREN, EARN, CAREN, etc) is now
extracted directly from BITEARN NODES rather than COUNTRY FILE, which it
was difficult to keep up to date.
Information about BITNET mailers is also extracted directly from BITEARN
NODES. LISTSERV no longer needs access to XMAILER NAMES, and
synchronization problems are avoided while improving lookup performance.
The NODESGEN process has been converted to PREXX to facilitate the
development of functions requiring changes to the network tables through
a performance improvement of a factor of about 5 in CPU time and better
overall performance on I/O and paging constrained systems. The whole
process now takes less than 3 seconds of CPU time on a 3090-J, about half
of which is due to post-processing code in REXX (filelist refresh, etc).
***********************************************************
* Usability: Support for the ':oldnode.'/':newnode.' tags *
***********************************************************
The use of a more efficient language for the NODESGEN process has made it
possible to add support for the ':oldnode.' and ':newnode.' BITEARN NODES
tags, whose exact syntax is described in the NEWTAGS DESCRIPT file
available from NETSERV.
IMPORTANT: LISTSERV implements a superset of the definitions described in
the november 1991 edition of NEWTAGS DESCRIPT. The document is being
updated to extend the syntax of the tags, but, at the time of this
writing, the new version of NEWTAGS DESCRIPT was still being proofread.
You can already make use of the new syntax, as both UPDATE and VERNODES
have already been changed to recognize it; it is only the formal
description which has been delayed.
LISTSERV uses the information in these tags to update subscriptions in
mailing lists, AFD/FUI lists, and the user registration file (SIGNUP2
FILE), which contains full names, passwords and user preferences. In
addition, nodes with a ':newnode.' tag are saved in the BITEARN NODESUM3
table, and any command from a user at the old node, or subscription
request for such a user by a list owner, is automatically translated to
the new address. This prevents new entries with the old node name from
being entered in mailing lists or other subscription files; this is
particularly important when the new address is an Internet hostname, as
the node will not have the benefit of a ':oldnode.' tag in its new entry
to take care of any left-over subscription, since it will never have a
new entry.
*********************************************************************
* Usability: Enhanced subject on administrative mail to list owners *
*********************************************************************
In order to make it easier for list owners to process large amounts of
administrative mail from LISTSERV, the 'Subject:' field of these messages
has been modified as follows:
<1.7c> Delivery error notice sent to list XYZ-L
<1.7d> XYZ-L: error report from sun3.abc.edu
<1.7c> New subscription to list XYZ-L
<1.7d> XYZ-L: [log in to unmask] joined the list
<1.7c> User signing off the XYZ-L list
<1.7d> XYZ-L: [log in to unmask] left the list
This makes it possible for experienced list owners to process most
messages without having to actually read the message text, without making
any difference to inexperienced owners.
*************************************************************
* Usability/Performance: Enhancements to the REVIEW command *
*************************************************************
The REVIEW command has been partly rewritten in PREXX and enhanced to
allow the user to indicate how he wants the subscribers list to be
sorted. This is accomplished through a new option, 'BY', which must be
followed by one of the following field names (minimum abbreviations are
shown in upper case, as usual):
- 'Name': sort by subscriber name. LISTSERV uses some heuristics in an
attempt to separate the first name and last name from any possible
organization name, title or affiliation, and sorts the list by last
name, then first name. There are of course cases where this does not
have the expected effect.
- 'NODEid' or 'Hostname': sort by nodeid/hostname (whatever is to the
right of the '@' sign in the subscriber's address - the two options are
synonyms).
- 'Userid/USERName': sort by username (the part to the left of the '@'
sign).
- 'Country/ies': sort by country of origin and produce a statistics
summary listing the various countries encountered and the number of
subscribers in each country. Note that the list is sorted by country
name (Switzerland), and not by country code (CH). Sorting by country
causes comments with the country names to be inserted in the list (see
example below), since it would otherwise be difficult to associate
country names with e-mail addresses. Because of this difficulty, you
may not use the country name as a secondary sort field; that is,
'REVIEW XYZ-L (BY (COUNTRY NAME)' is valid, but 'BY (NAME COUNTRY)'
would be rejected.
---------- Example of comments inserted for the COUNTRY option ----------
* Netherlands:
SOND0008@HASARA11 peter vons
U001222@HEARN Jos Wennmacker
EVERS@HUTRUU53 Evert Jan Evers / Univ. Utrecht, NL
* Spain:
ZCOCBF01@EBCESCA1 Cristina Blanxer
* Sweden:
ERIC@SEARN Eric Thomas
-------------------------------------------------------------------------
The default sort fields are 'BY (HOSTNAME USERNAME)', for compatibility.
Unless requested otherwise, entries with the same primary field are then
sorted by hostname/username to ensure results are predictable (ie that
reviewing the same list twice with the same options always gives the same
results).
Finally, the old COUNTRY options (as in 'REVIEW XYZ-L (COUNTRY') is now a
synonym for 'BY COUNTRY'. There is no way to get the country statistics
report without also ordering the subscribers list by country name.
***************************************************************
* Usability: Changed 'Reply-To:' field on administrative mail *
***************************************************************
Most admnistrative messages sent by LISTSERV to end users now have a
'Reply-To:' field pointing to 'LISTNAME-request@LOCALNODE', an address
which LISTSERV expects to point to the list owners (see next paragraph).
This prevents hasty users from inadvertently replying to the LISTSERV
userid, only to discover that the long letter they had not saved as been
processed as a set of commands and discarded.
In order for the address in question to work as advertised, you must have
the automatic owner-/-request support installed in your mailer. If you
are running version 2.08 of the VM mailer, you can get the necessary
changes from LISTSERV@SEARN; please refer to the description posted to
the LSTSRV-L archives on 26 Nov 1991 for more details. If you are running
version 2.09 or higher, it should be possible for you to activate this
function simply by setting a configuration variable; the syntax is
unknown at the time of this writing, as 2.09 was not yet available.
*****************************************************************
* Performance/Reliability: Internal incoming spool file decoder *
*****************************************************************
In order to improve performance when processing incoming spool files in
Netdata format, which might become the norm rather than the exception due
to planned changes to the VM mailer, and to decrease the dependence on
unexpected changes to standard VM commands, LISTSERV now has its own
internal decoder for incoming spool files. This decoder accepts the three
major formats (Netdata, DISK and CARD), plus of course "raw" PUNCH files.
It is considerably faster than native commands, except for CARD format,
but even then performance is better since the data does not need to be
staged through a disk file.
Noteworthy points:
1. You may now submit jobs to LISTSERV in CARD format. Previous versions
did not support CARD for input files.
2. With CARD and DISK formats, only the first file in the deck is
extracted and processed. As mentioned in previous release notes,
LISTSERV does not accept multiple jobs per spool file because it would
not be able to properly checkpoint them. Jobs are generally
transferred to the postmaster for manual examination and retry in case
of severe failure, and this requires each job to reside in a separate
file.
3. LISTSERV does not support the pre-VM/SP (1970's) version of the DISK
format. While there is no VM/370 system on BITNET, some very old
emulator software might still use this ancient format.
*************************************************************************
* Usability: Automatic conversion of DISTRIBUTE to mail-only recipients *
*************************************************************************
The addition of the internal spool file decoder has made it possible for
LISTSERV to automatically convert DISTRIBUTE FILE jobs to plain text when
the recipient is only reachable via mail.
Scenario: a user sends a file to a list without a mail envelope (ie via
SENDFILE, SEND/FILE or equivalent), because the file is large or because
he does not want it logged to the archives. The typical example is a
large postscript file which may need to be revised and resent a few
times. Unfortunately, some of the recipients do not have BITNET access
and can be reached only via mail.
Former behaviour: LISTSERV encapsulates the NJE file in an RFC822
envelope and delivers it to the mail-only users. If the file was sent in
an encoded format such as Netdata or DISK DUMP, the data in question will
contain binary characters which will not survive delivery. In any case,
the recipient is unlikely to have software that can decode such files.
New behaviour: LISTSERV examines the first record of the NJE file and
realizes that it contains binary characters. It runs the file through its
internal decoder and gets a successful return code, confirming that the
file was indeed in a supported network format (see note on VMSDUMP
below). LISTSERV sends the decoded text to the recipients, along with the
following message:
----------------------- Message sent to recipient -----------------------
Note: conversion from a BITNET transmission format not suitable for mail
delivery was locally attempted. This type of conversion may sometimes
require "choices" to be made by the conversion program, based on the
(lack of) support for various file formats on the target operating
system. The "choices" made by LISTSERV may not be the ones you expected,
since it does not know anything about the system you are using. However,
you would not have been able to use the file at all if it had not been
converted. If you have trouble using the file as you received it, please
contact the person who sent it and arrange for an alternate delivery
method.
-------------------------------------------------------------------------
Important remarks:
1. LISTSERV can only decode the formats it understands. In particular,
VMSDUMP format and some types of Netdata files originating from MVS
systems cannot be decoded. In such cases, behaviour is unchanged from
the previous versions. Note that VMSDUMP is a proprietary format for
which no specifications are available.
2. The decoded file might still be unusable if it was, for instance, an
executable file. LISTSERV attempts the conversion on the assumption
that binary characters do not survive ASCII to EBCDIC translation in
mail gateways anyway, thus there is a choice between something that
might work and something that assuredly will not work.
3. The server that converts the file is the server that performs the
final delivery to the user. Backbone servers which relay the file
onwards do not examine the delivery lists of other servers. Thus,
submitting a job to a 1.7d server is no guarantee that conversion will
take place for all recipients; it is the version of the server
delivering to the recipient that matters.
***********************************************************************
* Performance: New BITEARN DISTSUM2 file improves large distributions *
***********************************************************************
In order to provide acceptable performance, the DISTRIBUTE process uses a
cache file, formerly BITEARN DISTSUM, to save the results of its
CPU-intensive topological calculations. This file used to contain
information about both BITNET and domain-style addresses, with a ratio
depending on the type of list hosted by the server.
The conversion of the domain-style address resolver to PREXX has made it
possible to remove domain-style addresses from this file, thus saving
disk space while decreasing lookup time as the file becomes much smaller.
This was a good opportunity to change the caching algorithm and the
format of the file, to further improve performance. The new file is
called BITEARN DISTSUM2.
For a typical list with 150 subscribers, a performance improvement of
30-40% was achieved with an empty cache, while still saving about 10%
when all the entries are in the cache. A more significant improvement
should be achieved with large (1000+ subscribers) lists, but no benchmark
is available at this time. Note that this only affects the server
originating the distribution - the savings for "relay" servers are
minimal.
|
