Description of the changes for release 1.7a of LISTSERV ------------------------------------------------------- July 8th, 1991 ************** * Highlights * ************** Version 1.7 introduces the following major enhancements: - Support for the "new format" BITEARN NODES file, which is scheduled to replace the existing version by VERS9109. Availability of the "current format" file is to be eventually discontinued (1Q92?); as any software relying on BITEARN NODES and not supporting the "new format" will become basically unusable at that time, this is an item of critical importance. - Support for the CP component of VM/SP release 7 and VM/ESA, for XA/ESA-mode virtual machines and for the REXX compiler. Support for CMS release 7 is a short-term goal. - Support for the RSCS list processor can save significant amounts of CPU and I/O resources while conserving network bandwidth. - New network topology compiler and DISTRIBUTE path generator allow the use of the new set of EARN-BITNET connections and other redundant connections, thus spreading the load over several lines/sites and increasing both availability and speed of distribution. - Changes in internals can improve performance of DISTRIBUTE-intensive servers by up to 40% CPU time and 50% I/O, as compared to release 1.6e and taking advantage of the RSCS list processor. - CRC-based checksumming of DISTRIBUTE jobs provides a level of data transfer integrity higher than that of the native network services, while guarding against "X-DEL storms". - Improved, CRC-validated software update installation procedure ensures integrity of the update being loaded and provides recovery mechanisms in case of error during the update (disk full, etc). - Improved SHOW command provides additional information about the network topology and the operation of the local server. **************** * Availability * **************** Release 1.7 is available free of charge to sites which qualify under one (or more) of the categories listed below: a. Members of CREN or one of the CREN Cooperating Networks, excluding EARN members. b. EARN members from nordic countries. c. EARN members from non-EEC countries which have formally applied (in writing) for a license, and complied with the appropriate EARN directives. International organizations are not considered as belonging to the EEC. Licenses for release 1.6 from non-EEC EARN members are automatically carried over to release 1.7 - please do not mail a new license agreement. Release 1.6 licenses from EEC countries remain valid for release 1.6, but will not be carried over to release 1.7. ******************************* * Supported operating systems * ******************************* Release 1.7 introduces the following changes to the list of supported operating systems: - CMS release 3 is no longer supported. - The CP components of VM/SP release 7 and VM/ESA are supported as of release 1.7a. - Support for CMS release 7 will be introduced once suitable development and beta-test sites have been found. It is expected that release 1.7a should work under CMS release 7, however formal support cannot be provided without access to a suitable CMS7 development system. Support for other versions of CP and CMS is unchanged from 1.6e. ***************** * Compatibility * ***************** Release 1.7a is generally compatible with 1.6e. The most notable changes, in terms of compatibility, are highlighted below; note that more details are available from the longer descriptive section of these release notes. 1. The "Extended" options of the "Stats=" keyword is now ignored. 2. Some commands and functions will produce different output when the "new format" BITEARN NODES file is used, because the data in this file is structured differently and, generally speaking, full compatibility between the information in the old and new format files has not been preserved. 3. The DISTRIBUTE function may generate completely different (but more efficient) paths. 4. Processing of held reader files and gathering of input files from the various list userids has been changed. 5. Some information about DISTRIBUTE jobs was removed from the SHOW STATISTICS command; you should now use SHOW DISTRIBUTE for detailed information about the DISTRIBUTE function. 6. The index of the LISTS database now contains a "subscribers count" field. Note that this field will remain uninitialized for lists which are not managed by release 1.7 servers. 7. Application developers should note the following changes in some of the internal LISTSERV files: a. LSVBITWR MODULE is removed. b. BITEARN NODESUM is replaced with BITEARN NODESUM2. c. LINKSWT FILE is replaced with LINKSWT2 FILE. Local modifications relying on these files will have to be updated. 8. Formal support for manual installation of new releases is being withdrawn; automatic installation with local password validation is still the recommended procedure. Release 1.7a is to be installed directly on top of the base 1.6e code, and includes all the fixes available from LISTSERV@SEARN for release 1.6e (ie all the "16Ennnt FIX" files from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits", which always have to be installed manually. ******************************** * Minor fixes and enhancements * ******************************** This section is being omitted for release 1.7a. This is a major release, coming up almost one year after the last minor release (1.6e); the amount of small changes that were made is such that this section would require 500 to 1000 lines. *************************************************** * Integrity/Serviceability: New INSTALL processor * *************************************************** This is probably the most important item for LISTSERV maintainers, but list owners and end users should feel free to skip it if they are not interested. Major changes have been made to the INSTALL processor with release 1.7, in order to guarantee the integrity of the server code and to make generation and distribution of software updates easier for the author. Unfortunately, these changes could not be made totally transparent to the LISTSERV maintainer, so please read the following carefully. One of the most frequent causes of software integrity loss (running a mixture of one release and another, with unpredictable results) was corruption of the update decks by RSCS. It seems that CARD DUMP generates patterns (at least with the LISTSERV code) that trigger integrity bugs in some versions of RSCS (especially V1.3), and with each new release, about 3-4 servers in the same area received update files where two 400-bytes chunks had been swapped. In order to prevent this from happening in the future, INSTALL decks are now CRC-checked before being loaded (this is in addition to the DISTRIBUTE checksum). In order to prevent this from happening right now, with the very 1.7a shipment, the new INSTALL processor is being shipped beforehand - one can hope that a very small update deck has a much lower probability of getting corrupted than 6 large ones. This means you will have to exceptionally install two shipments in order to load 1.7a. The other common cause of integrity loss was a disk full condition during the CARD LOAD operation, or then a "bug" in the CARD command (usually the use of a version of CARD incompatible with the release of CMS). To solve this problem, files are no longer shipped under their "real" (production) filetype, but under a harmless "shadow" filetype. The files will be renamed to their "production" filetype upon successful completion of the CARD LOAD operation; otherwise, they will be deleted to recover the disk space, and the deck will be kept on LISTSERV's reader, in HOLD status, ready to be manually retried (via an INSTALL RELOAD command) once the problem has been corrected. A side-effect is that LISTSERV needs more "scratch" disk space on its 191 minidisk to load INSTALL decks than it used to, and that you may have to increase the size of this minidisk accordingly (and/or delete old LFIX backup files before installing). In order to simplify the generation of update decks, an installation exit may now be included in the deck to perform special processing at various points of the installation procedure. This is totally transparent to the LISTSERV maintainer in case of automatic installation (with or without local password validation). However, this does not take place when manual installation has been selected by the site, as the server is stopped during the whole duration of the update and cannot perform the necessary calls to the shipment exit. For this reason, manual installation is no longer supported, and sites which chose that option are strongly encouraged to switch to automatic installation with local password validation. There is no technical reason to want to use manual installation: if you really want control, you can always logon to LISTSERV, type the INSTALL PASSWORD command and watch the show with your finger on a PF-key set to 'IMM HX'. If you have local modifications that you want to refit, you should install the new version on a test server anyway. Still, there will be sites that continue to insist on the use of manual installation for religious reasons; you may skip the following paragraph if this is not your case. If you install an update manually, it is your responsibility to ensure that the installation exit gets called at the appropriate times and with the appropriate parameters. If you omit these calls, the update may not install properly. The first thing you should do is read the comments in INSTFT FILE, in order to understand the general philosophy of the "shadow filetype" system. You should then do the CARD LOAD and examine LSVINST EXEC (*not* LSVINST -LSV-EXE but the "old" EXEC file) to see what it would have done after the CARD LOAD, and do it manually. Do not assume that this has not changed since last time. You can then re-IPL and start LSVPROF. If you have any problem or question with this procedure, save your time and look for the answer in the code; manual installation is no longer supported, and questions will not be answered. In case you had configured your server for manual installation but want to switch to automatic, just edit LSV$INST EXEC to enable automatic installation, specify an installation validation password (INSTPW) in LOCAL SYSVARS, issue an INSTALL CLEANUP CODE_1.7A to your server and ask for a second shipment of the 1.7a update files. Finally, there are three minor but notable improvements to the INSTALL command in release 1.7: - LISTSERV will no longer lose track of INSTALL spool files under VM/HPO 5 or VM/XA if they are CP TRANSFERred to another account for examination and then transferred back. However, if you RECEIVE the file by mistake, you should not try to SENDFILE it back but contact the author immediately. - The new INSTALL CANCEL command can be used to order LISTSERV to cancel installation of a particular shipment and discard the corresponding spool files (if any). - Informational messages regarding the progress of the installation are now sent to a special userid rather than to LISTSERV coordination. Error messages are still sent directly to the coordinators. ******************************************************** * Integrity: CRC-based checksumming of DISTRIBUTE jobs * ******************************************************** BITNET used to enjoy a reputation of being a "safe" network - granted, it might take a while for your file to be delivered, but very little files ever got lost and corrupted files were rare enough to be printed, framed and hung somewhere in your office. Unfortunately, things have changed dramatically in the last 6 months or so. Whether the problem is due to buggy service levels of RSCS or other networking software, to obscure bugs in LISTSERV/LISTEARN or to an integrity problem in the operating system at a key site is still unknown; the fact remains that every day, several corrupted files travel across the LISTSERV backbone. This might be less than one file in 1000, but it is still not acceptable, especially as it is the cause of the infamous "X-DEL storms". An X-DEL storm is generated whenever one of the DISTRIBUTE jobs the servers use to notify each other of expired accounts happens to be hit by the most frequent type of file corruption: duplication (a second copy of the file is appended at the end of the original file). Instead of instructing each other to delete all subscriptions for a particular list of users, the servers then instruct each other to delete subscriptions, and then forward a deletion request to a number of other servers. In other words, each server receives as many copy of the deletion request as there are servers (rather than just one). Given that there are over 250 servers, the number of resulting files is quite significant, and a single storm can keep the network busy for a couple days. In order to solve this problem, DISTRIBUTE jobs are being checksummed with a CRC generator as of release 1.7a. Each release 1.7 server on the DISTRIBUTE path will verify the checksum, and abort the job (transferring the original spool file to the postmaster) if a mismatch is detected; older servers (including LISTEARN's) will simply forward the CRC indicator, treating it as a comment. The postmaster should examine aborted jobs and either: a. Attempt to correct the data and resubmit the job, leaving the CRC indicator untouched. If the data was improperly corrected, the job will fail CRC verification and be rejected again, so this is something that can always be tried without fear of causing extra, unwanted distributions. b. Decide that the data cannot be corrected but is still usable and resubmit the job, after deleting the CRC indicator. The job will then be unconditionally accepted and a new CRC will be generated for the remainder of the distribution path. c. Decide that the data is neither correctable nor usable, and notify the job originator or list owner that the job is being purged. To resubmit the job, RECEIVE the spool file, edit it, and use SENDFILE to resend it to LISTSERV (this will work only from one of the postmaster accounts). The CRC indicator in the job stream can take either of the following forms, depending on the version of the server which last forwarded it: //CRC DD "value" or: //CRC DD * value /* To remove it, just delete the lines in question. Because each and every release 1.7 server in the DISTRIBUTE path will verify that the checksum is correct, links causing noticeable amounts of file corruption will be quickly identified. One can hope that data corruption detected by an unbiased piece of software will have a higher impact on local staff than vague complaints from anonymous end-users such as "Mail I get via your machine is often scrambled up!", and that the faulty links/nodes will be fixed in a reasonable amount of time. It will also ensure that "X-DEL storms" no longer have the devastating effects mentioned above, as jobs failing the CRC check will be aborted as soon as the corruption can be detected. Even though there will always be servers that do not run release 1.7, the CRC code, having the "core" of the backbone check CRC's and discard corrupted jobs should considerably decrease the duplication factor and thus the overall impact of such storms on the network (from N^2 to some 25^2 at most). The CRC generator costs 580 instructions per data record; these are mostly register-to-register instructions, which correspond to about 200-300 "average" 370 instructions. A typical DISTRIBUTE job will require an extra 50ms of CPU time on a 9370-60, or one tenth of that on a 3090-E, plus 2 extra disk I/O's: not negligible, but nothing to worry about. ************************************************** * Performance: Support for the IBM REXX compiler * ************************************************** While the REXX compiler is almost fully compatible with the interpreter, it was in practice unusable with versions older than 1.7a due, in particular, to the use of INTERPRET instructions in critical (often-executed) EXEC files. Most INTERPRET instructions in version 1.7 have been either eliminated or moved away from critical EXEC files. It is now possible to compile most of the EXEC files, however this should be done CAREFULLY and according to the guidelines listed below. - Compiled REXX programs are 4 times larger than the interpreted source, even when the NOSOURCE option is used. This means you will be trading CPU time for a significant increase in working set size and an increase in disk I/O, which may not bring any performance enhancement if your system is heavily memory constrained. This also means LISTSERV will need more virtual memory to do the same thing, therefore: >>> Increase virtual storage by 2M if you compile LISTSERV <<< For release 1.7a, the EXECLOADed REXX programs take about 1.6M of memory when compiled. Increasing virtual storage by 2M gives you an extra 400k of storage for non-EXECLOADed programs in the process of being executed and compiler work areas. - LISTSERV needs access to the source files of compiled programs, under a filetype of SEXEC, in order to perform file version consistency verifications. These files do not need to be on the A-disk, but they have to be somewhere in the search order. If you do not compile a particular EXEC file, there is no need to copy it to SEXEC: the SEXEC file is needed only when the EXEC file is compiled. - LSVPUT EXEC and LDBASE EXEC are user interfaces - they are never executed by LISTSERV. The copy residing on LISTSERV's A-disk should NOT be compiled, as this would cause remote users requesting the software from your server to get a compiled program which they may not be able to use. There is not much to be gained from compiling either program, but if you insist on doing that, the best solution is to place a compiled copy on a public disk for your local users. - A number of EXEC files will give a warning message at compilation time, due to the use of the TRACE OFF instruction. These warnings should be ignored, since compiled programs always operate with TRACE OFF; ideally, the compiler should not even issue them. ANY OTHER WARNING should be treated as a fatal compilation error and immediately reported to the author. - The following EXEC files cannot be compiled: LSVBESTP LSVDBPN LSVDNSUM LSVIMAIL LSVINIT LSVINTPT LSVPRSUM LSVUDDC LSVUDDK - Using the NOSOURCE option when compiling will improve performance by decreasing the working set, as well as the amount of data that needs to be read from disk for non-EXECLOADed programs. However, it will cause the tracebacks generated in case of REXX error to contain just line numbers, and blanks in lieu of the failing statement. Such tracebacks will NOT be accepted in bug reports; if you decide to compile with the NOSOURCE option, you must "fill in" any traceback you mail the author for problem resolution, using your copy of the source file. Sending your source file is not an acceptable substitute, nor is stating that you have made no local modifications to the program in question. The new SHOW EXECLOAD command can be used to find out how much storage is used by the EXECLOADed programs, and how often they have been accessed. **************************************************** * Performance: Support for the RSCS list processor * **************************************************** IMPORTANT: Make sure to read the whole description before enabling the use of the list processor, as there are integrity considerations. Release 1.7 introduces support for the RSCS list processor available in RSCS V3, RSCS V2.3 and RSCS V2.2 with APAR VM30091. To enable the use of the RSCS list processor, you must: 1. Define a "list processor" link and ensure that it is automatically STARTed when RSCS comes up. 2. Add a definition in LOCAL SYSVARS of the following form: LISTPROC = 'linkid' That is, if your list processor is called '*LIST', you would code: LISTPROC = '*LIST' Using the list processor will save both CPU time and I/O operations - not only from the point of view of the LISTSERV userid, but also on a system-wide basis, as RSCS can duplicate files far more efficiently than LISTSERV. In addition, it may save a significant amount of network bandwidth, based on the topology in your area of the network, the amount and placement of backbone LISTSERV's in the vicinity, and the type of distributions your server is handling. Unfortunately, some service levels of RSCS create serious problems when the list processor is used. There may be bugs in the list processor code in your RSCS, but there may also be bugs in NJE software outside your institution which are only triggered by files with multiple dataset headers. In particular, versions of RSCS which do not support the list processor are prone to suffer from this sort of problem at one service level or another. By enabling the use of the list processor on your machine, you can cause loss of data (corrupted or rejected files) on downstream nodes which you have no control over. The best solution is clearly to have these sites correct the problem, but you may want to warn your "neighbourhood" in advance if you have reasons to believe they are running software which could be affected by the change. Finally, the list processor does not need to reside on the RSCS virtual machine LISTSERV is sending files to, as long as this RSCS has an appropriate ROUTE to the RSCS hosting the list processor. For instance, if you have a VM/HPO guest on a VM/XA system with a LISTSERV at both levels, you could define a list processor at the VM/XA level and instruct the VM/HPO RSCS to route this destination to the VM/XA system (eg 'ROUTE LISTPROC TO VMXA'). You can of course define a list processor on both RSCS machines as well - it is just a matter of performance. ************************************************************** * Usability: Support for the "new format" BITEARN NODES file * ************************************************************** Release 1.7a can operate indifferently with a "new format" (as per the specifications dated 1 Feb 1991) or "old format" file posterior to VERS9009. Full compatibility is, however, not possible, and you should expect minor changes in behaviour when switching from old to new format, even when both files have the same VERSyynn identifier. Some of the data (eg NJE-level aliases) is simply not available any longer with the new format; some (eg "people-related" information) is still there, but does not look the same. Performance improvements make most of the functions based on BITEARN NODES perform much better, when using an old format file, than under release 1.6e. With a new format file, however, performance varies significantly with the "structure" of the data in the file - how often and how many times LISTSERV needs to look up a different entry than the one it was examining in order to get at the information it needs. It is expected that most functions should still perform better than 1.6e when using a new format file, but only marginally so. Database reports in particular will take longer to generate, but will provide more information. The new SHOW BITEARN and SHOW NETWORK commands can be used to compare some of the attributes of the two formats. ************************************************** * Functionality: New options to the SHOW command * ************************************************** The following new options have been added to the SHOW command: - 'SHOW ALIAS host1 <host2<...>>' shows the registered Internet address of a BITNET node, or the BITNET nodeid corresponding to an Internet address. - 'SHOW BITEARN' displays technical information about the BITEARN NODES file (old vs new format, number of records, size of the associated tables, etc). - 'SHOW DISTribute' displays comprehensive statistics regarding the DISTRIBUTE activity since the last reboot. - 'SHOW DPATHs * | node1 <node2<...>>' shows the DISTRIBUTE paths to the specified backbone nodes, or to all nodes hosting a backbone server if '*' is specified. Note that these paths are used only for jobs generated by the server in question; jobs which are relayed from other servers use the paths determined by the originating server. - 'SHOW EXECLoad <ALL>' (privileged) displays information about EXECLOADed files. Use the ALL option to get information at the individual file level; omit it for just a summary. - 'SHOW FIXes' displays a list of the fixes installed since the last official release. - 'SHOW LINKs host1 <host2<...>>' extracts all link specifications from the BITEARN NODES entries of the specified hosts and displays them in a condensed form (destination nodeid and link speed pairs). - 'SHOW LSVFILER <ALL>' (privileged) displays information about the LSVFILER cache. Use the ALL option to get information at the individual file level; omit it for just a summary. - 'SHOW NETwork' displays technical information regarding the network topology (number of nodes, networks, countries, links, etc). - 'SHOW NODEntry host1 <host2<...>> </tag1</tag2<...>>>' extracts the BITEARN NODES entries for the specified hosts and displays either the full entry, if no search pattern was specified, or the ':node.' tag and any other tag matching one of the specified tag patterns (note that wildcards can be used in the 'tag1' et seq patterns). - 'SHOW PATHs source-host host1 <host2...>> <(<Weight> <NOSPeed>>' will display a path diagram showing the shortest paths from the source node to the listed destination nodes, along with the corresponding link speeds. The WEIGHT option causes LSVBITFD weights (1E9/speed) to be displayed instead, while the NOSPEED option removes any speed/weight indication (useful when the diagram does not fit in 80 columns). - 'SHOW RELease' and 'SHOW VERSion' are now synonyms for the 'RELEASE' command. Where 'node1' et seq refer to a valid NJE nodeid and 'host1' et seq refer to either an NJE nodeid or an Internet host address. In both cases, the nodeid/host must appear in the server's copy of BITEARN NODES. ************************************************************* * Serviceability/Performance: New network topology compiler * ************************************************************* Release 1.7 includes an enhanced network topology compiler that makes use of the line speed specifications in BITEARN NODES (both "old" and "new" formats). This provides a much more accurate view of the network than the former topology compiler, which treated all links equally and required a large LINKSWT FILE (70 entries as of VERS9107) to produce acceptable routing. The release 1.7 topology compiler obsoletes LINKSWT FILE, which is replaced with a much shorter LINKSWT2 FILE (3 entries as of VERS9107). Maintenance of this new link weights file is expected to be much more straightforward, as only saturated links over which extra (LISTSERV) traffic is to be avoided have to be coded. Combined with the new DISTRIBUTE path generator (qqv), the ability to enter new links in the LISTSERV configuration without having to alter BITEARN NODES makes it possible to assess the impact of major network changes in advance (before the NODUPD files are released) and avoid the all too frequent "emergency LINKSWT shipment" following a BITEARN NODES update. ******************************************************* * Performance/Functionality: Improved LSVBITFD MODULE * ******************************************************* The algorithm used by LSVBITFD has been changed to improve performance and allow the calling application to request that the full selected path be returned, in addition to the distance between source and target nodes (new PATH and PATHELM options). "Regular" calls to LSVBITFD will require around 30% less CPU time than with the release 1.6e version, while savings of 75% can be expected when the NEAREST option is specified. Path resolution does not involve any significant extra CPU cost. Important notes for application developers: A. The new version of LSVBITFD may produce unpredictable results when used with the "old" topology generator, as the new algorithm does not support links with a weight of zero. B. The REVERSE option outputs the reverse distance of the path that is shortest in the forward direction. In other words, the REVERSE option does not affect the path determination logic. C. Before migrating to the new version of LSVBITFD and BITEARN LINKSUM2, you must ensure your software can handle the much larger units (104,166 for a 9600bps link) generated by the new topology compiler. As long as your application can handle 31-bits integers and makes no assumption as to the meaning of the returned units (and, in particular, does not assume that 9600/result = line speed), there will be no compatibility problem. *********************************************************** * Availability/Performance: New DISTRIBUTE path generator * *********************************************************** For reasons which would be impossible to explain in a less than a hundred lines or so, the release 1.6 DISTRIBUTE path generator suffers from a number of serious flaws which have only become apparent in the last 6-12 months. In particular, it is unable to exploit network redundancy in a satisfactory way; any attempt to deviate from the pre-1990 "one and only one way to each destination" topology model results in unexpected and usually undesirable routing. To circumvent this problem, the LINKSWT FILE has been "nullifying" (giving an extremely high weight to) all new intercontinental links, thus forcing all the EARN<->BITNET LISTSERV traffic to go through a single server in France. This obviously creates all sorts of availability problems: bottlenecks, single point of failure causing total loss of service - in fact, this very server (or one of the RSCS/JES leading to it) is responsible for the recent X-DEL storms. It is again impossible to explain, in a few sentences, how the new path generator solves these problems; suffice it to say that it is using a completely different algorithm, which generates optimal distribution for jobs with several recipients per destination (the majority of jobs), at a reasonable CPU cost (in fact, sites processing a large amount of DISTRIBUTE jobs, or jobs with large amounts of destination sites, will actually save CPU time). The new path generator requires no configuration change, however you are advised to issue a 'MAIL SHOW DPATH *' command to your server and check that the distribution paths "make sense", bearing in mind that they are based on the line speed information in BITEARN NODES, rather than the actual physical line speeds. ************************************************************************ * Future Development: New DISTRIBUTE path generator/':nodeweight.' tag * ************************************************************************ The new DISTRIBUTE path generator includes support for a ':nodeweight.' tag in PEERS NAMES, which reflects the response time and/or availability of the various backbone servers. Through the use of this tag, it is possible to make the path generator favour a server over another, with the aim of increasing service to the end-user at the expense of network resources. This is a provisional facility; there is no plan to use it in the near future, but it will be available should the need arise. It is being documented here because it includes a tool to measure the "efficiency" of your server for DISTRIBUTE operations, which you may find useful for planning purposes ("How much more DISTRIBUTE load can we handle with the new CPU?", or "Should we run the DISTRIBUTE service on the 4341 or the 3081?"): the new SHOW BENCHMARK command, which will measure the availability of CPU, paging and disk I/O resources and produce a suggested ':nodeweight.' value (the lower, the better - this is the same scale as network link weights, ie 100,000 roughly corresponds to "the same kind of delays a 9600bps line would introduce"). Important: 1. The benchmark was designed to be run under "average" load during peak hours and, of course, without any special priority or reserved pages. 2. This command may take up to 30 seconds or so on saturated systems. Since it burns a fair amount of resources, it is restricted to postmasters and LISTSERV coordination. 3. The results are not to be used for any other purpose than evaluating the fitness of your machine for DISTRIBUTE operations. Small, saturated systems like CEARN or SEARN have suggested node weights ranging from 150,000 to 400,000. A typical VM/XA 3090 is around 20,000. This just reflects the fact that large systems, regardless of the load they have to bear, have no problem coping with LISTSERV's paging and disk I/O requirements (CPU time might be a problem, though, and so can CKPT contention). A 9370 with 8M of real memory and two (slow) disk units, on the other hand, will cause significant delays for paging and disk I/O if there are other active users, regardless of how much CPU time is available. For DISTRIBUTE activity, it is generally better to run LISTSERV at low priority on a large system than at normal priority on a small, dedicated system. ********************************************************* * Performance: Improvements to the DISTRIBUTE processor * ********************************************************* A number of performance improvements have been made to the DISTRIBUTE processor with release 1.7, in order to reduce startup time (around 1M 370 instructions were saved), working set (30-40% reduction) and resolution of domain-style addresses (factor of 2-3 in the average). Coupled with the LSVBITFD performance improvements and the use of the RSCS list processor, this can significantly decrease the cost of enabling the DISTRIBUTE function at your installation. If system resources consumption is the reason you opted not to join the backbone, you may want to reconsider your decision. Backbone sites processing over 500 jobs a day under normal conditions are encouraged to compare LISTSERV resource utilization before and after the installation of 1.7a and mail their figures to the LSTSRV-L list. ******************************************************************* * Performance: Improvements to LSVIUCV and reader-queues handling * ******************************************************************* A number of performance improvements have been made to LSVIUCV in order to decrease CPU utilization. This will affect LISTSERV's behaviour in the following ways: 1. A server which has been placed offline will consume much less CPU time than with previous versions. 2. LISTSERV may not react instantly to sudden changes in the amount of spool files in the system, and may take a while to place itself offline. Adjust the OFFLINEDLY variable (whose description can be found in LISTSERV SYSVARS) if this turns out to be a problem at your site. 3. The DELAY configuration variable is superseded by the new POLLDLY variable (described in LISTSERV SYSVARS) and ignored. 4. LISTSERV will fetch files a few times an hour from the reader of the list userids even when its own reader queue is not empty. This happens even in offline mode, and ensures that these files take a "fair" position in the queue and are not left in the list userids for hours (or even days) if the server is constantly busy. 5. There will, as before, be a delay between the arrival of a file in the reader of a mailing list and its distribution. However, sending a message to LISTSERV or hitting ENTER twice on the LISTSERV console will no longer cause the file to be processed immediately. Issue a NOP command to LISTSERV if it is critical to have the file processed right away. 6. LISTSERV will no longer act as a spool system stress-test program when large amounts of jobs are postponed for execution outside of prime time. However, LISTSERV is no longer as tolerant as it used to be with respect to bugs in the CP spooling functions. In particular, if a file is CP TRANSFERred back to LISTSERV and CP fails to reset the SFBSEEN bit, it may take quite a while for LISTSERV to notice the arrival of the file. ***************************************************************** * Performance: Maximum amount of BSMTP 'RCPT TO:' per mail file * ***************************************************************** In order to avoid "paralyzing" the local MAILER and/or SMTP servers for long periods of time, LISTSERV will no longer generate BSMTP mail files with more than 500 'RCPT TO:' specifications. You can change this value by defining a MAXBSMTP variable in LOCAL SYSVARS with the desired value. ******************************************************* * Usability: Support for XA/ESA-mode virtual machines * ******************************************************* All components of LISTSERV (including the LDBASE interface) can now operate in an XA or ESA virtual machine in toleration mode. Partial exploitation is possible with versions of CMS in which the REXX interpreter is XA-exploitive. Some of the assembler components can be made XA-exploitive when re-assembled with 'SYSPARM(XA)'; please note, however, that the resulting MODULE files will not operate on versions of CMS older than 5.5. The recommended virtual machine mode for LISTSERV is what the majority of your local users/applications are using. There is strictly no benefit to be expected from running LISTSERV in an XA virtual machine when most users run in 370 machines: shared segment paging will increase, and CMS SVC 20x overhead is higher in an XA virtual machine by up to 40%. **************************************************** * Performance: "Extended" statistics now withdrawn * **************************************************** In the early days of LISTSERV, before the inception of DISTRIBUTE, list distribution optimization was achieved through peered lists, whose placement had to be manually determined. One of the tools that LISTSERV provided for this purpose was the "Extended" option of the "Stats=" list header keyword, which allowed the list owner to learn about the load (links traversed x kilobytes) generated by each of the peers. When DISTRIBUTE was introduced, it became clear that there was no way this "extended" information could be provided at a reasonable programming and system resources cost; you could still activate it, but if the list was DISTRIBUTEd you would get only zeroes in the "Network load" column. This was never really a problem, because the use of DISTRIBUTE ensured optimal distribution of the postings, ie there was no way you could further improve the distribution with peered lists. The "extended" information was no longer needed to allocate new peers. Extended statistics did work for non-DISTRIBUTEd lists, though, and some list owners did not want to use DISTRIBUTE because it required too much CPU time (this was the considerably slower DIST1 processor). Unfortunately, extended statistics cost a significant amount of CPU time to maintain, and work only with BITNET-style addresses (ie JOE@UOFXYZ as opposed to [log in to unmask]); domain-style recipients are not accounted for, which means that the figures produced by the STAT command will be incorrect if there is a significant proportion of domain-style recipients. Since most mailing lists now have a majority of domain-style addresses, extended statistics have basically become a way to burn CPU time in order to produce meaningless figures. Unfortunately, list owners are often unaware of this and turn on the option just to "get as much information as possible". It was therefore decided to ignore this option in release 1.7 in order to save CPU time and prevent people from attempting to draw conclusions from inaccurate figures. ****************************************************** * Usability: Support for RFC822-compliant time zones * ****************************************************** LISTSERV will now attempt to generate RFC822-compliant time zones in the 'Date:' field of mail messages it generates. This is accomplished as follows: 1. If a TZONE variable is defined in LOCAL SYSVARS, its contents are used as is (and assumed to be valid). It is strongly recommended to use the universal +/-nnnn format to specify this time zone, rather than the more cryptic (to the uninitiated) time zone codes ('PDT', 'N', etc). 2. If the time zone returned by CP QUERY TIME starts with a + or - sign, two zeroes are appended to form the time zone (eg '+02' becomes '+0200'). 3. If that time zone is one of those specified in RFC822, it is converted to +/-nnnn format both for the convenience of non-US users and to avoid ambiguities, as the "military" time zones listed in RFC822 were incorrectly specified. Thus, 'D' or 'EDT' are automatically turned into '-0400'. 4. If that time zone is something else, or the much-misused GMT, LISTSERV will extract the GMT offset via a diagnose X'00' and convert it to +/-nnnn format. This will produce incorrect results if the DMKSYS GMT offset is incorrect, but at least it will be RFC822-compliant and expose the problem to the system administrator. Unless a non-numeric TZONE variable is defined by the postmaster, LISTSERV will always generate a time zone in +/-nnnn format. ******************************************************** * Usability: List maintenance from non-BITNET accounts * ******************************************************** IMPORTANT: Read the instructions carefully. If you rush into a test of the new facility, you may lose information about your subscribers. List maintenance from a non-BITNET account is made easier with release 1.7, as you no longer need to switch to a BITNET account or request assistance from the local postmaster to edit the list header. This is accomplished by allowing the list header to be modified independently of the subscribers list, using the procedure described below: 1. Send a 'GET listname (HEADER' command to LISTSERV. The list header will be returned and the list will be locked. NEVER SKIP THIS STEP! 2. If you happened to have an up-to-date copy of the list header in a local file, do NOT make use of it and proceed to step 3; just go to step 1. Similarly, if you had forgotten the HEADER option by mistake, do NOT delete the subscribers list from the file you got and proceed with step 3. Instead, issue an 'UNLOCK listname' command and repeat step 1. 3. Edit the list header, fill in the 'PW=' keyword and send (or mail) the resulting file back to LISTSERV. It is very important for you to remember that pre-1.7a servers do not have the capability to update just the list header. This procedure will fail with such servers, because they will not recognize the HEADER option of the GET command in step 1. However, they would gladly accept a PUT command with just the list header - and delete all the subscribers from the list. This is the reason why it is so important to always use the HEADER option, especially if you maintain several lists on several servers. If you do delete your subscribers list by mistake, you should immediately issue a 'GET listname (OLD' to recover the old copy. In addition, if you send the file back via mail, you should REMOVE THE SIGNATURE LINES before sending it. If you forget to do this, you will probably cause a syntax error and your command will be rejected. However, if your signature file happens to look like a valid subscribers list (which is admittedly unlikely), you would cause the existing list to be overwritten, as the old "PUT header + list" function is still supported for compatibility reasons. Finally, this new procedure will obviously save both CPU time and network bandwidth for large lists; there is no reason to use the "old" procedure if you do not want to update the subscribers list as well.