Description of the changes for release 1.7a of LISTSERV
-------------------------------------------------------
July 8th, 1991
**************
* Highlights *
**************
Version 1.7 introduces the following major enhancements:
- Support for the "new format" BITEARN NODES file, which is scheduled to
replace the existing version by VERS9109. Availability of the "current
format" file is to be eventually discontinued (1Q92?); as any software
relying on BITEARN NODES and not supporting the "new format" will
become basically unusable at that time, this is an item of critical
importance.
- Support for the CP component of VM/SP release 7 and VM/ESA, for
XA/ESA-mode virtual machines and for the REXX compiler. Support for CMS
release 7 is a short-term goal.
- Support for the RSCS list processor can save significant amounts of CPU
and I/O resources while conserving network bandwidth.
- New network topology compiler and DISTRIBUTE path generator allow the
use of the new set of EARN-BITNET connections and other redundant
connections, thus spreading the load over several lines/sites and
increasing both availability and speed of distribution.
- Changes in internals can improve performance of DISTRIBUTE-intensive
servers by up to 40% CPU time and 50% I/O, as compared to release 1.6e
and taking advantage of the RSCS list processor.
- CRC-based checksumming of DISTRIBUTE jobs provides a level of data
transfer integrity higher than that of the native network services,
while guarding against "X-DEL storms".
- Improved, CRC-validated software update installation procedure ensures
integrity of the update being loaded and provides recovery mechanisms
in case of error during the update (disk full, etc).
- Improved SHOW command provides additional information about the network
topology and the operation of the local server.
****************
* Availability *
****************
Release 1.7 is available free of charge to sites which qualify under one
(or more) of the categories listed below:
a. Members of CREN or one of the CREN Cooperating Networks, excluding
EARN members.
b. EARN members from nordic countries.
c. EARN members from non-EEC countries which have formally applied (in
writing) for a license, and complied with the appropriate EARN
directives. International organizations are not considered as
belonging to the EEC.
Licenses for release 1.6 from non-EEC EARN members are automatically
carried over to release 1.7 - please do not mail a new license agreement.
Release 1.6 licenses from EEC countries remain valid for release 1.6, but
will not be carried over to release 1.7.
*******************************
* Supported operating systems *
*******************************
Release 1.7 introduces the following changes to the list of supported
operating systems:
- CMS release 3 is no longer supported.
- The CP components of VM/SP release 7 and VM/ESA are supported as of
release 1.7a.
- Support for CMS release 7 will be introduced once suitable development
and beta-test sites have been found. It is expected that release 1.7a
should work under CMS release 7, however formal support cannot be
provided without access to a suitable CMS7 development system.
Support for other versions of CP and CMS is unchanged from 1.6e.
*****************
* Compatibility *
*****************
Release 1.7a is generally compatible with 1.6e. The most notable changes,
in terms of compatibility, are highlighted below; note that more details
are available from the longer descriptive section of these release notes.
1. The "Extended" options of the "Stats=" keyword is now ignored.
2. Some commands and functions will produce different output when the
"new format" BITEARN NODES file is used, because the data in this
file is structured differently and, generally speaking, full
compatibility between the information in the old and new format files
has not been preserved.
3. The DISTRIBUTE function may generate completely different (but more
efficient) paths.
4. Processing of held reader files and gathering of input files from the
various list userids has been changed.
5. Some information about DISTRIBUTE jobs was removed from the SHOW
STATISTICS command; you should now use SHOW DISTRIBUTE for detailed
information about the DISTRIBUTE function.
6. The index of the LISTS database now contains a "subscribers count"
field. Note that this field will remain uninitialized for lists which
are not managed by release 1.7 servers.
7. Application developers should note the following changes in some of
the internal LISTSERV files:
a. LSVBITWR MODULE is removed.
b. BITEARN NODESUM is replaced with BITEARN NODESUM2.
c. LINKSWT FILE is replaced with LINKSWT2 FILE.
Local modifications relying on these files will have to be updated.
8. Formal support for manual installation of new releases is being
withdrawn; automatic installation with local password validation is
still the recommended procedure.
Release 1.7a is to be installed directly on top of the base 1.6e code,
and includes all the fixes available from LISTSERV@SEARN for release 1.6e
(ie all the "16Ennnt FIX" files from filelist "LFIXES"), with the
exception of those containing replacement "recommended user exits", which
always have to be installed manually.
********************************
* Minor fixes and enhancements *
********************************
This section is being omitted for release 1.7a. This is a major release,
coming up almost one year after the last minor release (1.6e); the amount
of small changes that were made is such that this section would require
500 to 1000 lines.
***************************************************
* Integrity/Serviceability: New INSTALL processor *
***************************************************
This is probably the most important item for LISTSERV maintainers, but
list owners and end users should feel free to skip it if they are not
interested.
Major changes have been made to the INSTALL processor with release 1.7,
in order to guarantee the integrity of the server code and to make
generation and distribution of software updates easier for the author.
Unfortunately, these changes could not be made totally transparent to the
LISTSERV maintainer, so please read the following carefully.
One of the most frequent causes of software integrity loss (running a
mixture of one release and another, with unpredictable results) was
corruption of the update decks by RSCS. It seems that CARD DUMP generates
patterns (at least with the LISTSERV code) that trigger integrity bugs in
some versions of RSCS (especially V1.3), and with each new release, about
3-4 servers in the same area received update files where two 400-bytes
chunks had been swapped. In order to prevent this from happening in the
future, INSTALL decks are now CRC-checked before being loaded (this is in
addition to the DISTRIBUTE checksum). In order to prevent this from
happening right now, with the very 1.7a shipment, the new INSTALL
processor is being shipped beforehand - one can hope that a very small
update deck has a much lower probability of getting corrupted than 6
large ones. This means you will have to exceptionally install two
shipments in order to load 1.7a.
The other common cause of integrity loss was a disk full condition during
the CARD LOAD operation, or then a "bug" in the CARD command (usually the
use of a version of CARD incompatible with the release of CMS). To solve
this problem, files are no longer shipped under their "real" (production)
filetype, but under a harmless "shadow" filetype. The files will be
renamed to their "production" filetype upon successful completion of the
CARD LOAD operation; otherwise, they will be deleted to recover the disk
space, and the deck will be kept on LISTSERV's reader, in HOLD status,
ready to be manually retried (via an INSTALL RELOAD command) once the
problem has been corrected. A side-effect is that LISTSERV needs more
"scratch" disk space on its 191 minidisk to load INSTALL decks than it
used to, and that you may have to increase the size of this minidisk
accordingly (and/or delete old LFIX backup files before installing).
In order to simplify the generation of update decks, an installation exit
may now be included in the deck to perform special processing at various
points of the installation procedure. This is totally transparent to the
LISTSERV maintainer in case of automatic installation (with or without
local password validation). However, this does not take place when manual
installation has been selected by the site, as the server is stopped
during the whole duration of the update and cannot perform the necessary
calls to the shipment exit. For this reason, manual installation is no
longer supported, and sites which chose that option are strongly
encouraged to switch to automatic installation with local password
validation. There is no technical reason to want to use manual
installation: if you really want control, you can always logon to
LISTSERV, type the INSTALL PASSWORD command and watch the show with your
finger on a PF-key set to 'IMM HX'. If you have local modifications that
you want to refit, you should install the new version on a test server
anyway. Still, there will be sites that continue to insist on the use of
manual installation for religious reasons; you may skip the following
paragraph if this is not your case.
If you install an update manually, it is your responsibility to ensure
that the installation exit gets called at the appropriate times and with
the appropriate parameters. If you omit these calls, the update may not
install properly. The first thing you should do is read the comments in
INSTFT FILE, in order to understand the general philosophy of the "shadow
filetype" system. You should then do the CARD LOAD and examine LSVINST
EXEC (*not* LSVINST -LSV-EXE but the "old" EXEC file) to see what it
would have done after the CARD LOAD, and do it manually. Do not assume
that this has not changed since last time. You can then re-IPL and start
LSVPROF. If you have any problem or question with this procedure, save
your time and look for the answer in the code; manual installation is no
longer supported, and questions will not be answered.
In case you had configured your server for manual installation but want
to switch to automatic, just edit LSV$INST EXEC to enable automatic
installation, specify an installation validation password (INSTPW) in
LOCAL SYSVARS, issue an INSTALL CLEANUP CODE_1.7A to your server and ask
for a second shipment of the 1.7a update files.
Finally, there are three minor but notable improvements to the INSTALL
command in release 1.7:
- LISTSERV will no longer lose track of INSTALL spool files under VM/HPO
5 or VM/XA if they are CP TRANSFERred to another account for
examination and then transferred back. However, if you RECEIVE the file
by mistake, you should not try to SENDFILE it back but contact the
author immediately.
- The new INSTALL CANCEL command can be used to order LISTSERV to cancel
installation of a particular shipment and discard the corresponding
spool files (if any).
- Informational messages regarding the progress of the installation are
now sent to a special userid rather than to LISTSERV coordination.
Error messages are still sent directly to the coordinators.
********************************************************
* Integrity: CRC-based checksumming of DISTRIBUTE jobs *
********************************************************
BITNET used to enjoy a reputation of being a "safe" network - granted, it
might take a while for your file to be delivered, but very little files
ever got lost and corrupted files were rare enough to be printed, framed
and hung somewhere in your office. Unfortunately, things have changed
dramatically in the last 6 months or so. Whether the problem is due to
buggy service levels of RSCS or other networking software, to obscure
bugs in LISTSERV/LISTEARN or to an integrity problem in the operating
system at a key site is still unknown; the fact remains that every day,
several corrupted files travel across the LISTSERV backbone. This might
be less than one file in 1000, but it is still not acceptable, especially
as it is the cause of the infamous "X-DEL storms". An X-DEL storm is
generated whenever one of the DISTRIBUTE jobs the servers use to notify
each other of expired accounts happens to be hit by the most frequent
type of file corruption: duplication (a second copy of the file is
appended at the end of the original file). Instead of instructing each
other to delete all subscriptions for a particular list of users, the
servers then instruct each other to delete subscriptions, and then
forward a deletion request to a number of other servers. In other words,
each server receives as many copy of the deletion request as there are
servers (rather than just one). Given that there are over 250 servers,
the number of resulting files is quite significant, and a single storm
can keep the network busy for a couple days.
In order to solve this problem, DISTRIBUTE jobs are being checksummed
with a CRC generator as of release 1.7a. Each release 1.7 server on the
DISTRIBUTE path will verify the checksum, and abort the job (transferring
the original spool file to the postmaster) if a mismatch is detected;
older servers (including LISTEARN's) will simply forward the CRC
indicator, treating it as a comment. The postmaster should examine
aborted jobs and either:
a. Attempt to correct the data and resubmit the job, leaving the CRC
indicator untouched. If the data was improperly corrected, the job
will fail CRC verification and be rejected again, so this is something
that can always be tried without fear of causing extra, unwanted
distributions.
b. Decide that the data cannot be corrected but is still usable and
resubmit the job, after deleting the CRC indicator. The job will then
be unconditionally accepted and a new CRC will be generated for the
remainder of the distribution path.
c. Decide that the data is neither correctable nor usable, and notify the
job originator or list owner that the job is being purged.
To resubmit the job, RECEIVE the spool file, edit it, and use SENDFILE to
resend it to LISTSERV (this will work only from one of the postmaster
accounts). The CRC indicator in the job stream can take either of the
following forms, depending on the version of the server which last
forwarded it:
//CRC DD "value"
or:
//CRC DD *
value
/*
To remove it, just delete the lines in question.
Because each and every release 1.7 server in the DISTRIBUTE path will
verify that the checksum is correct, links causing noticeable amounts of
file corruption will be quickly identified. One can hope that data
corruption detected by an unbiased piece of software will have a higher
impact on local staff than vague complaints from anonymous end-users such
as "Mail I get via your machine is often scrambled up!", and that the
faulty links/nodes will be fixed in a reasonable amount of time. It will
also ensure that "X-DEL storms" no longer have the devastating effects
mentioned above, as jobs failing the CRC check will be aborted as soon as
the corruption can be detected. Even though there will always be servers
that do not run release 1.7, the CRC code, having the "core" of the
backbone check CRC's and discard corrupted jobs should considerably
decrease the duplication factor and thus the overall impact of such
storms on the network (from N^2 to some 25^2 at most).
The CRC generator costs 580 instructions per data record; these are
mostly register-to-register instructions, which correspond to about
200-300 "average" 370 instructions. A typical DISTRIBUTE job will require
an extra 50ms of CPU time on a 9370-60, or one tenth of that on a 3090-E,
plus 2 extra disk I/O's: not negligible, but nothing to worry about.
**************************************************
* Performance: Support for the IBM REXX compiler *
**************************************************
While the REXX compiler is almost fully compatible with the interpreter,
it was in practice unusable with versions older than 1.7a due, in
particular, to the use of INTERPRET instructions in critical
(often-executed) EXEC files.
Most INTERPRET instructions in version 1.7 have been either eliminated or
moved away from critical EXEC files. It is now possible to compile most
of the EXEC files, however this should be done CAREFULLY and according to
the guidelines listed below.
- Compiled REXX programs are 4 times larger than the interpreted source,
even when the NOSOURCE option is used. This means you will be trading
CPU time for a significant increase in working set size and an increase
in disk I/O, which may not bring any performance enhancement if your
system is heavily memory constrained. This also means LISTSERV will
need more virtual memory to do the same thing, therefore:
>>> Increase virtual storage by 2M if you compile LISTSERV <<<
For release 1.7a, the EXECLOADed REXX programs take about 1.6M of
memory when compiled. Increasing virtual storage by 2M gives you an
extra 400k of storage for non-EXECLOADed programs in the process of
being executed and compiler work areas.
- LISTSERV needs access to the source files of compiled programs, under a
filetype of SEXEC, in order to perform file version consistency
verifications. These files do not need to be on the A-disk, but they
have to be somewhere in the search order. If you do not compile a
particular EXEC file, there is no need to copy it to SEXEC: the SEXEC
file is needed only when the EXEC file is compiled.
- LSVPUT EXEC and LDBASE EXEC are user interfaces - they are never
executed by LISTSERV. The copy residing on LISTSERV's A-disk should NOT
be compiled, as this would cause remote users requesting the software
from your server to get a compiled program which they may not be able
to use. There is not much to be gained from compiling either program,
but if you insist on doing that, the best solution is to place a
compiled copy on a public disk for your local users.
- A number of EXEC files will give a warning message at compilation time,
due to the use of the TRACE OFF instruction. These warnings should be
ignored, since compiled programs always operate with TRACE OFF;
ideally, the compiler should not even issue them. ANY OTHER WARNING
should be treated as a fatal compilation error and immediately reported
to the author.
- The following EXEC files cannot be compiled:
LSVBESTP LSVDBPN LSVDNSUM LSVIMAIL LSVINIT LSVINTPT LSVPRSUM
LSVUDDC LSVUDDK
- Using the NOSOURCE option when compiling will improve performance by
decreasing the working set, as well as the amount of data that needs to
be read from disk for non-EXECLOADed programs. However, it will cause
the tracebacks generated in case of REXX error to contain just line
numbers, and blanks in lieu of the failing statement. Such tracebacks
will NOT be accepted in bug reports; if you decide to compile with the
NOSOURCE option, you must "fill in" any traceback you mail the author
for problem resolution, using your copy of the source file. Sending
your source file is not an acceptable substitute, nor is stating that
you have made no local modifications to the program in question.
The new SHOW EXECLOAD command can be used to find out how much storage is
used by the EXECLOADed programs, and how often they have been accessed.
****************************************************
* Performance: Support for the RSCS list processor *
****************************************************
IMPORTANT: Make sure to read the whole description before enabling the
use of the list processor, as there are integrity considerations.
Release 1.7 introduces support for the RSCS list processor available in
RSCS V3, RSCS V2.3 and RSCS V2.2 with APAR VM30091. To enable the use of
the RSCS list processor, you must:
1. Define a "list processor" link and ensure that it is automatically
STARTed when RSCS comes up.
2. Add a definition in LOCAL SYSVARS of the following form:
LISTPROC = 'linkid'
That is, if your list processor is called '*LIST', you would code:
LISTPROC = '*LIST'
Using the list processor will save both CPU time and I/O operations - not
only from the point of view of the LISTSERV userid, but also on a
system-wide basis, as RSCS can duplicate files far more efficiently than
LISTSERV. In addition, it may save a significant amount of network
bandwidth, based on the topology in your area of the network, the amount
and placement of backbone LISTSERV's in the vicinity, and the type of
distributions your server is handling.
Unfortunately, some service levels of RSCS create serious problems when
the list processor is used. There may be bugs in the list processor code
in your RSCS, but there may also be bugs in NJE software outside your
institution which are only triggered by files with multiple dataset
headers. In particular, versions of RSCS which do not support the list
processor are prone to suffer from this sort of problem at one service
level or another. By enabling the use of the list processor on your
machine, you can cause loss of data (corrupted or rejected files) on
downstream nodes which you have no control over. The best solution is
clearly to have these sites correct the problem, but you may want to warn
your "neighbourhood" in advance if you have reasons to believe they are
running software which could be affected by the change.
Finally, the list processor does not need to reside on the RSCS virtual
machine LISTSERV is sending files to, as long as this RSCS has an
appropriate ROUTE to the RSCS hosting the list processor. For instance,
if you have a VM/HPO guest on a VM/XA system with a LISTSERV at both
levels, you could define a list processor at the VM/XA level and instruct
the VM/HPO RSCS to route this destination to the VM/XA system (eg 'ROUTE
LISTPROC TO VMXA'). You can of course define a list processor on both
RSCS machines as well - it is just a matter of performance.
**************************************************************
* Usability: Support for the "new format" BITEARN NODES file *
**************************************************************
Release 1.7a can operate indifferently with a "new format" (as per the
specifications dated 1 Feb 1991) or "old format" file posterior to
VERS9009. Full compatibility is, however, not possible, and you should
expect minor changes in behaviour when switching from old to new format,
even when both files have the same VERSyynn identifier. Some of the data
(eg NJE-level aliases) is simply not available any longer with the new
format; some (eg "people-related" information) is still there, but does
not look the same.
Performance improvements make most of the functions based on BITEARN
NODES perform much better, when using an old format file, than under
release 1.6e. With a new format file, however, performance varies
significantly with the "structure" of the data in the file - how often
and how many times LISTSERV needs to look up a different entry than the
one it was examining in order to get at the information it needs. It is
expected that most functions should still perform better than 1.6e when
using a new format file, but only marginally so. Database reports in
particular will take longer to generate, but will provide more
information. The new SHOW BITEARN and SHOW NETWORK commands can be used
to compare some of the attributes of the two formats.
**************************************************
* Functionality: New options to the SHOW command *
**************************************************
The following new options have been added to the SHOW command:
- 'SHOW ALIAS host1 <host2<...>>' shows the registered Internet address
of a BITNET node, or the BITNET nodeid corresponding to an Internet
address.
- 'SHOW BITEARN' displays technical information about the BITEARN NODES
file (old vs new format, number of records, size of the associated
tables, etc).
- 'SHOW DISTribute' displays comprehensive statistics regarding the
DISTRIBUTE activity since the last reboot.
- 'SHOW DPATHs * | node1 <node2<...>>' shows the DISTRIBUTE paths to the
specified backbone nodes, or to all nodes hosting a backbone server if
'*' is specified. Note that these paths are used only for jobs
generated by the server in question; jobs which are relayed from other
servers use the paths determined by the originating server.
- 'SHOW EXECLoad <ALL>' (privileged) displays information about
EXECLOADed files. Use the ALL option to get information at the
individual file level; omit it for just a summary.
- 'SHOW FIXes' displays a list of the fixes installed since the last
official release.
- 'SHOW LINKs host1 <host2<...>>' extracts all link specifications from
the BITEARN NODES entries of the specified hosts and displays them in a
condensed form (destination nodeid and link speed pairs).
- 'SHOW LSVFILER <ALL>' (privileged) displays information about the
LSVFILER cache. Use the ALL option to get information at the individual
file level; omit it for just a summary.
- 'SHOW NETwork' displays technical information regarding the network
topology (number of nodes, networks, countries, links, etc).
- 'SHOW NODEntry host1 <host2<...>> </tag1</tag2<...>>>' extracts the
BITEARN NODES entries for the specified hosts and displays either the
full entry, if no search pattern was specified, or the ':node.' tag and
any other tag matching one of the specified tag patterns (note that
wildcards can be used in the 'tag1' et seq patterns).
- 'SHOW PATHs source-host host1 <host2...>> <(<Weight> <NOSPeed>>' will
display a path diagram showing the shortest paths from the source node
to the listed destination nodes, along with the corresponding link
speeds. The WEIGHT option causes LSVBITFD weights (1E9/speed) to be
displayed instead, while the NOSPEED option removes any speed/weight
indication (useful when the diagram does not fit in 80 columns).
- 'SHOW RELease' and 'SHOW VERSion' are now synonyms for the 'RELEASE'
command.
Where 'node1' et seq refer to a valid NJE nodeid and 'host1' et seq refer
to either an NJE nodeid or an Internet host address. In both cases, the
nodeid/host must appear in the server's copy of BITEARN NODES.
*************************************************************
* Serviceability/Performance: New network topology compiler *
*************************************************************
Release 1.7 includes an enhanced network topology compiler that makes use
of the line speed specifications in BITEARN NODES (both "old" and "new"
formats). This provides a much more accurate view of the network than the
former topology compiler, which treated all links equally and required a
large LINKSWT FILE (70 entries as of VERS9107) to produce acceptable
routing. The release 1.7 topology compiler obsoletes LINKSWT FILE, which
is replaced with a much shorter LINKSWT2 FILE (3 entries as of VERS9107).
Maintenance of this new link weights file is expected to be much more
straightforward, as only saturated links over which extra (LISTSERV)
traffic is to be avoided have to be coded. Combined with the new
DISTRIBUTE path generator (qqv), the ability to enter new links in the
LISTSERV configuration without having to alter BITEARN NODES makes it
possible to assess the impact of major network changes in advance (before
the NODUPD files are released) and avoid the all too frequent "emergency
LINKSWT shipment" following a BITEARN NODES update.
*******************************************************
* Performance/Functionality: Improved LSVBITFD MODULE *
*******************************************************
The algorithm used by LSVBITFD has been changed to improve performance
and allow the calling application to request that the full selected path
be returned, in addition to the distance between source and target nodes
(new PATH and PATHELM options). "Regular" calls to LSVBITFD will require
around 30% less CPU time than with the release 1.6e version, while
savings of 75% can be expected when the NEAREST option is specified. Path
resolution does not involve any significant extra CPU cost.
Important notes for application developers:
A. The new version of LSVBITFD may produce unpredictable results when
used with the "old" topology generator, as the new algorithm does not
support links with a weight of zero.
B. The REVERSE option outputs the reverse distance of the path that is
shortest in the forward direction. In other words, the REVERSE option
does not affect the path determination logic.
C. Before migrating to the new version of LSVBITFD and BITEARN LINKSUM2,
you must ensure your software can handle the much larger units
(104,166 for a 9600bps link) generated by the new topology compiler.
As long as your application can handle 31-bits integers and makes no
assumption as to the meaning of the returned units (and, in
particular, does not assume that 9600/result = line speed), there will
be no compatibility problem.
***********************************************************
* Availability/Performance: New DISTRIBUTE path generator *
***********************************************************
For reasons which would be impossible to explain in a less than a hundred
lines or so, the release 1.6 DISTRIBUTE path generator suffers from a
number of serious flaws which have only become apparent in the last 6-12
months. In particular, it is unable to exploit network redundancy in a
satisfactory way; any attempt to deviate from the pre-1990 "one and only
one way to each destination" topology model results in unexpected and
usually undesirable routing. To circumvent this problem, the LINKSWT FILE
has been "nullifying" (giving an extremely high weight to) all new
intercontinental links, thus forcing all the EARN<->BITNET LISTSERV
traffic to go through a single server in France. This obviously creates
all sorts of availability problems: bottlenecks, single point of failure
causing total loss of service - in fact, this very server (or one of the
RSCS/JES leading to it) is responsible for the recent X-DEL storms.
It is again impossible to explain, in a few sentences, how the new path
generator solves these problems; suffice it to say that it is using a
completely different algorithm, which generates optimal distribution for
jobs with several recipients per destination (the majority of jobs), at a
reasonable CPU cost (in fact, sites processing a large amount of
DISTRIBUTE jobs, or jobs with large amounts of destination sites, will
actually save CPU time).
The new path generator requires no configuration change, however you are
advised to issue a 'MAIL SHOW DPATH *' command to your server and check
that the distribution paths "make sense", bearing in mind that they are
based on the line speed information in BITEARN NODES, rather than the
actual physical line speeds.
************************************************************************
* Future Development: New DISTRIBUTE path generator/':nodeweight.' tag *
************************************************************************
The new DISTRIBUTE path generator includes support for a ':nodeweight.'
tag in PEERS NAMES, which reflects the response time and/or availability
of the various backbone servers. Through the use of this tag, it is
possible to make the path generator favour a server over another, with
the aim of increasing service to the end-user at the expense of network
resources. This is a provisional facility; there is no plan to use it in
the near future, but it will be available should the need arise. It is
being documented here because it includes a tool to measure the
"efficiency" of your server for DISTRIBUTE operations, which you may find
useful for planning purposes ("How much more DISTRIBUTE load can we
handle with the new CPU?", or "Should we run the DISTRIBUTE service on
the 4341 or the 3081?"): the new SHOW BENCHMARK command, which will
measure the availability of CPU, paging and disk I/O resources and
produce a suggested ':nodeweight.' value (the lower, the better - this is
the same scale as network link weights, ie 100,000 roughly corresponds to
"the same kind of delays a 9600bps line would introduce").
Important:
1. The benchmark was designed to be run under "average" load during peak
hours and, of course, without any special priority or reserved pages.
2. This command may take up to 30 seconds or so on saturated systems.
Since it burns a fair amount of resources, it is restricted to
postmasters and LISTSERV coordination.
3. The results are not to be used for any other purpose than evaluating
the fitness of your machine for DISTRIBUTE operations.
Small, saturated systems like CEARN or SEARN have suggested node weights
ranging from 150,000 to 400,000. A typical VM/XA 3090 is around 20,000.
This just reflects the fact that large systems, regardless of the load
they have to bear, have no problem coping with LISTSERV's paging and disk
I/O requirements (CPU time might be a problem, though, and so can CKPT
contention). A 9370 with 8M of real memory and two (slow) disk units, on
the other hand, will cause significant delays for paging and disk I/O if
there are other active users, regardless of how much CPU time is
available. For DISTRIBUTE activity, it is generally better to run
LISTSERV at low priority on a large system than at normal priority on a
small, dedicated system.
*********************************************************
* Performance: Improvements to the DISTRIBUTE processor *
*********************************************************
A number of performance improvements have been made to the DISTRIBUTE
processor with release 1.7, in order to reduce startup time (around 1M
370 instructions were saved), working set (30-40% reduction) and
resolution of domain-style addresses (factor of 2-3 in the average).
Coupled with the LSVBITFD performance improvements and the use of the
RSCS list processor, this can significantly decrease the cost of enabling
the DISTRIBUTE function at your installation. If system resources
consumption is the reason you opted not to join the backbone, you may
want to reconsider your decision. Backbone sites processing over 500 jobs
a day under normal conditions are encouraged to compare LISTSERV resource
utilization before and after the installation of 1.7a and mail their
figures to the LSTSRV-L list.
*******************************************************************
* Performance: Improvements to LSVIUCV and reader-queues handling *
*******************************************************************
A number of performance improvements have been made to LSVIUCV in order
to decrease CPU utilization. This will affect LISTSERV's behaviour in the
following ways:
1. A server which has been placed offline will consume much less CPU time
than with previous versions.
2. LISTSERV may not react instantly to sudden changes in the amount of
spool files in the system, and may take a while to place itself
offline. Adjust the OFFLINEDLY variable (whose description can be
found in LISTSERV SYSVARS) if this turns out to be a problem at your
site.
3. The DELAY configuration variable is superseded by the new POLLDLY
variable (described in LISTSERV SYSVARS) and ignored.
4. LISTSERV will fetch files a few times an hour from the reader of the
list userids even when its own reader queue is not empty. This happens
even in offline mode, and ensures that these files take a "fair"
position in the queue and are not left in the list userids for hours
(or even days) if the server is constantly busy.
5. There will, as before, be a delay between the arrival of a file in the
reader of a mailing list and its distribution. However, sending a
message to LISTSERV or hitting ENTER twice on the LISTSERV console
will no longer cause the file to be processed immediately. Issue a NOP
command to LISTSERV if it is critical to have the file processed right
away.
6. LISTSERV will no longer act as a spool system stress-test program when
large amounts of jobs are postponed for execution outside of prime
time. However, LISTSERV is no longer as tolerant as it used to be with
respect to bugs in the CP spooling functions. In particular, if a file
is CP TRANSFERred back to LISTSERV and CP fails to reset the SFBSEEN
bit, it may take quite a while for LISTSERV to notice the arrival of
the file.
*****************************************************************
* Performance: Maximum amount of BSMTP 'RCPT TO:' per mail file *
*****************************************************************
In order to avoid "paralyzing" the local MAILER and/or SMTP servers for
long periods of time, LISTSERV will no longer generate BSMTP mail files
with more than 500 'RCPT TO:' specifications. You can change this value
by defining a MAXBSMTP variable in LOCAL SYSVARS with the desired value.
*******************************************************
* Usability: Support for XA/ESA-mode virtual machines *
*******************************************************
All components of LISTSERV (including the LDBASE interface) can now
operate in an XA or ESA virtual machine in toleration mode. Partial
exploitation is possible with versions of CMS in which the REXX
interpreter is XA-exploitive. Some of the assembler components can be
made XA-exploitive when re-assembled with 'SYSPARM(XA)'; please note,
however, that the resulting MODULE files will not operate on versions of
CMS older than 5.5.
The recommended virtual machine mode for LISTSERV is what the majority of
your local users/applications are using. There is strictly no benefit to
be expected from running LISTSERV in an XA virtual machine when most
users run in 370 machines: shared segment paging will increase, and CMS
SVC 20x overhead is higher in an XA virtual machine by up to 40%.
****************************************************
* Performance: "Extended" statistics now withdrawn *
****************************************************
In the early days of LISTSERV, before the inception of DISTRIBUTE, list
distribution optimization was achieved through peered lists, whose
placement had to be manually determined. One of the tools that LISTSERV
provided for this purpose was the "Extended" option of the "Stats=" list
header keyword, which allowed the list owner to learn about the load
(links traversed x kilobytes) generated by each of the peers. When
DISTRIBUTE was introduced, it became clear that there was no way this
"extended" information could be provided at a reasonable programming and
system resources cost; you could still activate it, but if the list was
DISTRIBUTEd you would get only zeroes in the "Network load" column. This
was never really a problem, because the use of DISTRIBUTE ensured optimal
distribution of the postings, ie there was no way you could further
improve the distribution with peered lists. The "extended" information
was no longer needed to allocate new peers.
Extended statistics did work for non-DISTRIBUTEd lists, though, and some
list owners did not want to use DISTRIBUTE because it required too much
CPU time (this was the considerably slower DIST1 processor).
Unfortunately, extended statistics cost a significant amount of CPU time
to maintain, and work only with BITNET-style addresses (ie JOE@UOFXYZ as
opposed to [log in to unmask]); domain-style recipients are not accounted
for, which means that the figures produced by the STAT command will be
incorrect if there is a significant proportion of domain-style
recipients. Since most mailing lists now have a majority of domain-style
addresses, extended statistics have basically become a way to burn CPU
time in order to produce meaningless figures. Unfortunately, list owners
are often unaware of this and turn on the option just to "get as much
information as possible". It was therefore decided to ignore this option
in release 1.7 in order to save CPU time and prevent people from
attempting to draw conclusions from inaccurate figures.
******************************************************
* Usability: Support for RFC822-compliant time zones *
******************************************************
LISTSERV will now attempt to generate RFC822-compliant time zones in the
'Date:' field of mail messages it generates. This is accomplished as
follows:
1. If a TZONE variable is defined in LOCAL SYSVARS, its contents are used
as is (and assumed to be valid). It is strongly recommended to use the
universal +/-nnnn format to specify this time zone, rather than the
more cryptic (to the uninitiated) time zone codes ('PDT', 'N', etc).
2. If the time zone returned by CP QUERY TIME starts with a + or - sign,
two zeroes are appended to form the time zone (eg '+02' becomes
'+0200').
3. If that time zone is one of those specified in RFC822, it is converted
to +/-nnnn format both for the convenience of non-US users and to
avoid ambiguities, as the "military" time zones listed in RFC822 were
incorrectly specified. Thus, 'D' or 'EDT' are automatically turned
into '-0400'.
4. If that time zone is something else, or the much-misused GMT, LISTSERV
will extract the GMT offset via a diagnose X'00' and convert it to
+/-nnnn format. This will produce incorrect results if the DMKSYS GMT
offset is incorrect, but at least it will be RFC822-compliant and
expose the problem to the system administrator.
Unless a non-numeric TZONE variable is defined by the postmaster,
LISTSERV will always generate a time zone in +/-nnnn format.
********************************************************
* Usability: List maintenance from non-BITNET accounts *
********************************************************
IMPORTANT: Read the instructions carefully. If you rush into a test of
the new facility, you may lose information about your subscribers.
List maintenance from a non-BITNET account is made easier with release
1.7, as you no longer need to switch to a BITNET account or request
assistance from the local postmaster to edit the list header. This is
accomplished by allowing the list header to be modified independently of
the subscribers list, using the procedure described below:
1. Send a 'GET listname (HEADER' command to LISTSERV. The list header
will be returned and the list will be locked. NEVER SKIP THIS STEP!
2. If you happened to have an up-to-date copy of the list header in a
local file, do NOT make use of it and proceed to step 3; just go to
step 1. Similarly, if you had forgotten the HEADER option by mistake,
do NOT delete the subscribers list from the file you got and proceed
with step 3. Instead, issue an 'UNLOCK listname' command and repeat
step 1.
3. Edit the list header, fill in the 'PW=' keyword and send (or mail) the
resulting file back to LISTSERV.
It is very important for you to remember that pre-1.7a servers do not
have the capability to update just the list header. This procedure will
fail with such servers, because they will not recognize the HEADER option
of the GET command in step 1. However, they would gladly accept a PUT
command with just the list header - and delete all the subscribers from
the list. This is the reason why it is so important to always use the
HEADER option, especially if you maintain several lists on several
servers. If you do delete your subscribers list by mistake, you should
immediately issue a 'GET listname (OLD' to recover the old copy.
In addition, if you send the file back via mail, you should REMOVE THE
SIGNATURE LINES before sending it. If you forget to do this, you will
probably cause a syntax error and your command will be rejected. However,
if your signature file happens to look like a valid subscribers list
(which is admittedly unlikely), you would cause the existing list to be
overwritten, as the old "PUT header + list" function is still supported
for compatibility reasons.
Finally, this new procedure will obviously save both CPU time and network
bandwidth for large lists; there is no reason to use the "old" procedure
if you do not want to update the subscribers list as well.
|
