Description of the changes for release 1.7e of LISTSERV
-------------------------------------------------------
November 11th, 1992
**************
* Highlights *
**************
Version 1.7e introduces the following major enhancements:
- Support for the popular VMSDUMP format, both for incoming and outgoing
files. BACKUP savesets can now be stored as native LISTSERV files and
conveniently retrieved in VMSDUMP format.
- Support for MIME (RFC1341) for outgoing files. Internet users may now
order text files via e-mail without having to worry about truncated
lines or incorrect ASCII to EBCDIC translation.
- LISTSERV internals, DISTRIBUTE and inter-server communication methods
were changed in preparation for release 2.09 of the VM mailer, which
will have the ability to handle lines longer than 80 bytes. Note that
it was unfortunately not possible to test this code in a realistic
setup, and further adjustments might thus be needed before LISTSERV can
actually use that feature of MAILER V2.09.
- Performance and robustness improvements through the use of private
formatting routines rather than IBM-supplied commands for the
generation of files in Netdata and other NJE delivery formats (except
CARD DUMP). This change makes it possible to use a SFS directory for
LISTSERV's work disk (D-disk).
- Additional improvements have been made to the notifications sent by
LISTSERV to list owners such that the proposed commands can be executed
without retyping by simply forwarding the notification back to
LISTSERV.
****************
* Availability *
****************
Release 1.7 is available free of charge to sites which qualify under one
(or more) of the categories listed below:
a. Members of CREN or one of the CREN Cooperating Networks, excluding:
1. Members of the European Academic and Research Network (EARN).
2. Members whose computing facilities are located in an EEC country.
b. Members of EARN, CREN or a CREN Cooperating Network whose computing
facilities are located in a nordic country (Denmark, Finland, Iceland,
Norway, Sweden).
c. Members of EARN whose computing facilities are not located in an EEC
country and which have formally applied (in writing) for a license,
and complied with the appropriate EARN directives. International
organizations are not considered as belonging to the EEC, regardless
of where their computing facilities are physically located.
Release 1.6 licenses from EEC countries remain valid for release 1.6, but
will not be carried over to release 1.7.
*******************************
* Supported operating systems *
*******************************
Release 1.7 introduces the following changes to the list of supported
operating systems:
- (1.7a) Withdrawal of support for CMS release 3.
- (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA.
- (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes.
- (1.7b) Support for the Shared File System, with a few restrictions
concerning the setup of the A and D-disks and the default filepool
setting. See the corresponding section of the 1.7b release notes for
more information.
- (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes.
- (1.7e) Support for the Shared File System for the D-disk.
Support for other versions of CP and CMS is unchanged from 1.6e.
**************************
* External Compatibility *
**************************
Release 1.7e is generally compatible with 1.7d, from the perspective of
the end-user (including list managers and maintainers), and with the
exception of the changes listed below. Release 1.7e introduces major
changes to the LISTSERV internals, making compatibility with locally
developed extensions or local modifications problematic. These changes
are briefly described in the next section, and only affect sites which
made alterations or additions to the LISTSERV code. Changes which affect
all LISTSERV sites are highlighted below; note that more details are
available, when appropriate, from the longer descriptive section of these
release notes.
1. The changes to support mail items with lines longer than 80 bytes are
not transparent for peered lists if some of the peers are running a
version which does not support the long lines. More details are
provided further down.
2. The changes to DISTRIBUTE to support mail items with lines longer
than 80 bytes can cause problems with LISTEARN and older versions of
LISTSERV. See further down for more details.
3. Release 1.7e internally uses DISTRIBUTE for all postings to mailing
lists, in order to avoid having to duplicate the code that had to be
added to support CRC verification of messages with long lines. When
"Mail-via= Direct" is specified, a new internal DISTRIBUTE option is
used to prevent the message from being forwarded to other servers,
but the message is still formatted and delivered to the mailer by
DISTRIBUTE code. As a result, the SHOW command no longer displays the
amount of postings that were processed through DISTRIBUTE; instead,
it shows the total amount of recipients.
4. LISTSERV now removes control characters from mail messages by default
and expands tabs in the process. Control characters usually have
undesirable and unexpected results, as they seldom survive the double
ASCII->EBCDIC->ASCII translation and cause the recipient's terminal
to execute sequences different from the ones meant by the message
sender - not to mention the case of ASCII control characters on
EBCDIC terminals. Furthermore, certain combinations of control
characters were found to create problems with IBM's SMTP (due to
unexpected CRLF sequences appearing in the middle of a record after
translation). Lists which absolutely require control characters
must have a "Translate= No" keyword added to their header.
Release 1.7e is to be installed directly on top of the base 1.7d code,
and includes all the fixes available from LISTSERV@SEARN for release 1.7d
(ie all the "17Dnnnt FIX" files from filelist "LFIXES"), with the
exception of those containing replacement "recommended user exits", which
always have to be installed manually.
**************************
* Internal Compatibility *
**************************
Major changes have been made to LISTSERV internals. The most important
ones are briefly described below; refer to the LSTSRV-L archives for more
information.
1. FFORMAT/FCLASS information is no longer kept in GLOBALV variables,
and must be accessed/changed via calls to LSVCD1.
2. LSVRECV no longer forces the filemode to D if a value was specified.
3. The UUENCODE and LISTSERV-Punch file formats are now provided by
built-in LISTSERV code rather than standalone MODULE's. As a result,
PDUTIL, LSVPUNCH and LSV$UUE will be automatically erased after
installation of 1.7e. If you have local applications which depend on
them, make a copy before installing 1.7e. Furthermore, the format of
SYSFF FILE has changed; if you had installed new file formats on your
server, you will need to make a few adjustments. See the comments in
SYSFF FILE for more information.
4. Authors of local command packages should note that LISTSERV does not
guarantee that the program stack is in any particular state when the
command processor is invoked. MAKEBUF and DROPBUF should be used
wherever necessary.
The following REXX files have been converted to PREXX with release 1.7e:
LSV$UUE LSVMAIL LSVSENDF MIG_*
The following MODULE files have become obsolete with release 1.7e:
LSVPUNCH PDUTIL
**************************
* Compatibility Warnings *
**************************
This new section describes planned changes which may affect compatibility
(both external and internal). The release in which the change is planned
to be introduced is indicated, with the letter 'x' denoting an unknown
release of the specified major version (not necessarily posterior to the
last release explicitly mentioned). Note that conversion of more REXX
files to PREXX is assumed to take place with each new release and is not
indicated here.
- (1.7x) The internal format of FILELIST/FILEID and attendant files such
as AFDLIST FILE will change dramatically. Applications which alter them
directly will no longer work.
- (1.7x) The internal format of LIST and attendant files (CACHE, STATS,
et al) will change dramatically. Applications which alter them directly
will no longer work.
- (1.7x) LSVDIAD4, LSVFRD1, LSVFWR1 and RXLSVUHF will be eventually
removed. You should avoid using them.
********************************
* Minor fixes and enhancements *
********************************
REMINDER: Manual installation is not supported for release 1.7, and the
code that implemented it has been removed with version 1.7e. Automatic
installation with local password validation must be used instead. Sites
which insist on installing manually should note that shipments contain an
installation pre- and post-processing exit. If that exit is not executed,
or if it is executed in a different order or environment than during
automatic installation, serious problems may occur.
This section contains a brief description of all the "small yet not
unimportant worries" that have been fixed in release 1.7e, along with the
affected releases, if known ("All" meaning "any release in which the
facility is available", ie the problem was present since the very
beginning).
Minor enhancements which do not warrant a detailed description are also
included; they are all located at the bottom of the list, and can be
identified by the word "New" in the "Affected releases" column.
Affected
releases Description of the bug, problem or enhancement
-------- ----------------------------------------------
1.7d Servers running on VM/SP or VM/HPO were occasionally unable
to purge incoming job files if a severe error occured during
processing; the job would be reprocessed about 15 minutes
later and cause multiple error notifications.
1.7d LISTSERV will no longer generate addresses of the form:
From: "(Joe Smith)" <[log in to unmask]>
These addresses were not generated intentionally and are
unnecessarily complex, but they are perfectly valid. There
exist popular mail programs which cannot handle them, but
that only means they are in error.
1.7a The recipients counts for network-wide lists produced by LIST
SUMMARY were incorrectly calculated. The totals reported by
pre-1.7e code happened to be lower than the correct values at
the time the fix was written.
All Some components of LISTSERV were unable to append to files
with more than 64k records, due to the unexpected behaviour
of FSWRITE with RECNO=0 but without FORM=E.
New Plural forms ('Owners' and 'Editors') are now supported for
access-level keywords, with the same meaning as the singular
forms.
New JOIN and LEAVE are now accepted as synonyms for SUBSCRIBE and
SIGNOFF, respectively, in order to improve compatibility with
Mailbase. Mailbase appears to be the most robust and
comprehensive unix-based list manager currently available,
with the added advantage of not confusing the users by
claiming to be a unix version of the BITNET LISTSERV. The
addition of these two synonyms will hopefully remove an
obstacle to the use of Mailbase by a larger community.
New In order to make it easier to implement "sub-lists" (a la
Mailbase) with LISTSERV, the syntax of the "Subscription="
keyword has been enhanced to support access-control
specifications, in addition to the previously supported
keywords ("Open", "Closed" and "By_Owner"). Sub-lists can be
convenient when a rather large forum is to discuss several
sub-topics in parallel and participants are to be free to
join and leave the various sub-lists as they see fit. This
change makes it possible to implement sub-lists where
membership is not open.
New A side-effect of the previous change is that "Subscription=
Public" is now a synonym for "Subscription= Open", removing
one of the most common mistakes made by new list owners.
New LISTSERV no longer needs to swap its work minidisk (192) from
mode D to mode Z and back. This makes it possible for the
work disk to be replaced with a SFS directory, as long as you
make sure the directory is already accessed R/W as mode D
when LISTSERV is started. Mode Z remains reserved for
LISTSERV use (dynamic minidisks, et al).
*******************************************************
* Usability/Performance: Support for new file formats *
*******************************************************
Release 1.7e introduces support for a number of new file formats, which
can be specified on the 'F=' keyword of commands such as GET, REVIEW,
etc.
- VMSDUMP, the popular file transfer format for VMS systems running JNET.
LISTSERV only supports sequential files. For incoming files, the
variable portion of VFC files (usually a time stamp) is discarded and
UNDEFINED files are not supported (but the other 3 stream types are).
For outgoing files, LISTSERV only generates FIXED or VARIABLE files.
This is still sufficient to conveniently store BACKUP savesets as
native LISTSERV files.
- XXENCODE, a version of UUENCODE modified to use a more sensible table
so that the resulting messages survive ASCII to EBCDIC translation.
- Two flavours of MIME base64-encoded messages. Users with a MIME-capable
user interface can use 'F=MIME' to order text files which contain
characters that do not survive ASCII to EBCDIC translation, or lines
longer than the gateways in the way will support. LISTSERV translates
the text to ASCII, adds CRLF sequences, and sends the result as a
base64-encoded message of type 'text/appl'. 'F=MIME/APPL' can be used
as a replacement for UUENCODE to order binary stream files (content
type 'application/octet-stream'). It is important to note that this has
exactly the same functionality as UUENCODE: the input file is read,
ignoring file attributes and record boundaries, and turned into a
stream of bytes, which are then encoded and sent. In other words, this
is not a suitable method for transferring text files on binary VM or
VMS files.
In addition, release 1.7e no longer uses the IBM-supplied system commands
to generate files in Netdata and other formats - with the exception of
CARD DUMP format, for which a modified version of CARD MODULE is used.
This change improves robustness, performance, and considerably simplifies
functions such as AFD and GET which no longer need to copy files and swap
the work disk around in order to have the right information in the header
or to avoid unwanted side effects of the system commands. The new code is
about 3-4 times faster for both Netdata and DISK DUMP formats.
***************************************************************
* Usability: Support for mail with lines longer than 80 bytes *
***************************************************************
LISTSERV internals and inter-server communication methods have been
changed to support mail with lines longer than 80 bytes, in preparation
for a new version of the VM mailer with support for Netdata format. There
is no way to make use of that support with the current version of the
mailer (2.08) and it is perfectly normal that lines continue to be split
at column 80 after the installation of 1.7e.
These changes introduce a number of problems when communication has to go
through servers not running 1.7e. Three major components of LISTSERV are
impacted:
1. Inter-server commands: older servers (including LISTEARN) are able to
process the new commands without problem. However, in certain cases,
they may truncate the command if they, in turn, need to forward the
request to other servers. In practice this only affects unlikely cases
such as e-mail addresses longer than 80 bytes (the command itself is
never truncated, but list of addresses passed in DD's might be
truncated at column 80).
2. Peered lists: a problem will occur if a message with lines longer than
80 bytes is posted to a peered list and some of the peers are not
running version 1.7e. The message would be truncated as per the
following table:
+---------+-------------------+-----------------+
| Version | Header truncated | Body truncated |
+---------+-------------------+-----------------+
| 1.7e | Never (*) | Never (*) |
+---------+-------------------+-----------------+
| 1.7d | Folded at 80 | Truncated at 80 |
+---------+-------------------+-----------------+
| Other | Truncated at 255, | Truncated at 80 |
| | then folded at 80 | |
+---------+-------------------+-----------------+
(*) If the local mailer does not support lines longer than 80 bytes,
the headers and body will have to be folded at column 80.
The truncation of the message text is obviously undesirable, but will
only happen if there were lines longer than 80 bytes in the first
place; in other words, it is not an issue until mailer 2.09 is
released. From then on peered lists will require that all servers run
release 1.7e. A temporary keyword, "Long-lines= No", is provided to
assist migration in cases where one of the peers absolutely cannot
install 1.7e in time (note that peered LISTSERV-LISTEARN lists are not
formally supported, but can benefit from that migration aid while it
is being provided). When "Long-lines= No" is specified, LISTSERV folds
text lines longer than 80 bytes to bypass this problem at the expense
of usability.
The truncation of header lines at column 255 is potententially a very
serious problems. Long header lines may be present regardless of the
version of the mailer being used, as LISTSERV must internally unfold
RFC822 fields in order to process them and does not re-fold them until
necessary (and since inter-server jobs support lines of up to 64k
bytes, it never turns out to be necessary). Although header lines
longer than 255 are rare, this new mechanism can create problems with
peered lists unless all the peers run 1.7d or 1.7e. Because there is
no problem with the previous version and older versions are no longer
supported, no migration aid was provided.
3. DISTRIBUTE: LISTSERV now uses Netdata format for DISTRIBUTE jobs, so
that long mail lines (up to 64k and thus well beyond the 1000 bytes
SMTP standard) can be supported. While older servers are perfectly
capable of processing such jobs, the same truncation problems as in
point 2 exist. Fortunately, version 1.7e is being released before the
new VM mailer and it should be possible to remove the few non-LISTEARN
sites which are still running 1.7d from the backbone when the mailer
is released, thus solving the problem for non-LISTEARN sites. The
structure of the LISTSERV backbone in Europe is such that countries
running LISTSERV will seldom be affected by truncation due to LISTEARN
sites, so this is fortunately a private LISTEARN problem which will
have to be solved separately.
IMPORTANT: The use of Netdata for DISTRIBUTE will cause significant
performance problems to sites running a version older than 1.7d. There
will be a noticeable impact for 1.7c sites and a factor of 2 or worse
degradation for 1.7b and older. The impact of this change on 1.7d and
1.7e sites is negligible, and, for 1.7e, it is more than compensated for
by other performance changes to DISTRIBUTE.
***************************************************************
* Usability: Simpler administative procedures for list owners *
***************************************************************
The 'Subject:' field conventions for administrative messages introduced
with version 1.7d have proved to be very popular with list owners, and
are being generalized with 1.7e. Furthermore, messages which request the
list owner to confirm a certain action (such as adding a user to a list,
or changing his distribution options) are now specially formatted such
that simply forwarding the message back to LISTSERV will cause the
command to be automatically executed without having to be retyped. Note
that the message must be forwarded, and not replied to. While there are
user interfaces whose reply function will cause the command to be
properly executed, in the majority of cases this will not work (even with
REPLY TEXT). Please be careful not to suggest that users use REPLY, even
if it happens to work for you.
**************************************************
* Usability: Welcome message for new subscribers *
**************************************************
Release 1.7e no longer requires a change to the list MAILFORM in order to
provide a "welcome message" for new subscribers. The main problems with
the MAILFORM solution is that postmaster intervention is required every
time the message has to be updated, for security reasons, and that it is
very easy to make mistakes as MAILFORM files are actually programs with
(a majority of) imbedded text. In particular, it is very easy for novice
list owners to forget to double the quotes.
Once a subscription has been accepted, LISTSERV will look for a LISTSERV
(not CMS) file called 'listname WELCOME'. This is a plain text file which
can be easily created and maintained by the list owner if the postmaster
has created a filelist for the list. The first line, if it starts with
'Subject:', defines the subject field of the welcome message; otherwise
LISTSERV uses a pre-defined subject and uses the first line as part of
the message text. Again, contrary to the MAILFORM's, the welcome messages
are LISTSERV files; simply placing a 'listname WELCOME' file on
LISTSERV's A-disk will not accomplish anything.
Since this is a plain text file, there is no need to double quotes or
include formatting commands for text which must not be reformatted. This,
of course, means that MAILFORM variable substitution sequences are not
available.
*********************************************************
* Usability: LB64.C - a tool for non-BITNET list owners *
*********************************************************
LB64 is a new tool which can be used by non-BITNET list owners to send
files or commands (and in particular PUT requests) to LISTSERV without
having to worry about lines longer than 80 bytes or characters which do
not get translated properly by the gateways the message goes through.
LB64 takes advantage of a server-to-server protocol which was added with
release 1.7e but was not designed to solve the problem of PUT requests
from the Internet with lines longer than 80 bytes (this problem will
disappear when the VM mailer supports Netdata format). In other words,
LB64 does not claim to be anything but a useful side-effect of an
unrelated LISTSERV development; if you dislike it, you are free not to
use it.
The program is written in C and is designed for unix systems, where its
output can be conveniently piped to a 'mail listserv@hostname' command.
It can however be used on any system that can emulate unix text files, be
it ASCII or EBCDIC. For more information, issue a 'GET LB64.C' command to
LISTSERV and read the comments in the program.
|
