Description of the changes for release 1.7b of LISTSERV ------------------------------------------------------- November 25th, 1991 ************** * Highlights * ************** Version 1.7b introduces the following major enhancements: - Support for CMS release 7 in both 370 and XA/ESA/XC modes. Support for CMS release 8 will be provided shortly after its general availability. Due to its generally observed lack of robustness and stability, CMS release 6 is still not formally supported; problems that cannot be reproduced under CMS release 7 will not be addressed. - Support for the Shared File System, with the restrictions listed under the heading "Support for the CMS Shared File System". - Introduction of the "PREXX library" - a development tool allowing REXX functions to be rewritten in PASCAL for better performance, with measured savings on the order of a factor of 10 to 15, depending on the nature of the function being rewritten. - Introduction of the 'IETFHDR' header style to allow BITNET users to become familiar with Internet-style mailing lists conventions, on which the Internet version of LISTSERV will be based. Read important warnings and disclaimer under the heading "IETF-style mail headers". **************** * Availability * **************** Release 1.7 is available free of charge to sites which qualify under one (or more) of the categories listed below: a. Members of CREN or one of the CREN Cooperating Networks, excluding EARN members. b. EARN members from nordic countries. c. EARN members from non-EEC countries which have formally applied (in writing) for a license, and complied with the appropriate EARN directives. International organizations are not considered as belonging to the EEC. Licenses for release 1.6 from non-EEC EARN members are automatically carried over to release 1.7 - please do not mail a new license agreement. Release 1.6 licenses from EEC countries remain valid for release 1.6, but will not be carried over to release 1.7. ******************************* * Supported operating systems * ******************************* Release 1.7 introduces the following changes to the list of supported operating systems: - (1.7a) Withdrawal of support for CMS release 3. - (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA. - (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes. - (1.7b) Support for the Shared File System, with a few restrictions concerning the setup of the A and D-disks and the default filepool setting. See the corresponding section for more information. - It is expected that version 1.7b will work under CMS release 8, however formal support cannot be provided at the time of its release, as CMS8 is not yet GA. Support for other versions of CP and CMS is unchanged from 1.6e. ***************** * Compatibility * ***************** Release 1.7b is generally compatible with 1.7a. The most notable changes, in terms of compatibility, are highlighted below; note that more details are available, when appropriate, from the longer descriptive section of these release notes. 1. Support for the "old" format of BITEARN NODES is no longer available, as BITEARN NODES has been distributed in the "new" format since VERS9109 (SEP91). Sites with a current copy of BITEARN NODES are not affected; sites with an outdated copy must install a current copy before upgrading to release 1.7b. 2. The CARD component no longer supports CDF minidisks. Note that LISTSERV itself never supported CDF minidisks: this warning is included for the benefit of sites which use the LISTSERV version of CARD for their users. 3. IETF-style headers will not operate properly in a peered-list setup if some of the peers are running a version older than 1.7b. 4. Application developers should note the changes in internal LISTSERV files described under the heading "Renamed REXX files". 5. New source code and MODULE files will now be called LSWxxxxxx rather than LSVxxxxxx. Maintenance programs issuing commands such as 'LISTFILE LSV* MODULE (STACK' will need to be modified accordingly. Release 1.7b is to be installed directly on top of the base 1.7a code, and includes all the fixes available from LISTSERV@SEARN for release 1.7a (ie all the "17Annnt FIX" files from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits", which always have to be installed manually. ******************************** * Minor fixes and enhancements * ******************************** This section contains a brief description of all the "small yet not unimportant worries" that have been fixed in release 1.7b, along with the affected releases, if known ("All" meaning "any release in which the facility is available", ie the problem was present since the very beginning). Minor enhancements which do not warrant a detailed description are also included; they are all located at the bottom of the list, and can be identified by the word "New" in the "Affected releases" column. Finally, changes which might introduce compatibility problems are flagged with one asterisk for minor problems (change in a message or in the output of some commands originally designed for human readers rather than programs, but which some people might still have decided to feed to programs), or two asterisks for more serious problems, for which a detailed description will always be provided further on. Affected releases Description of the bug, problem or enhancement -------- ---------------------------------------------- 1.7a REXX error in LSVLTELL during SHOW NODE command via mail. Fixed. 1.7a LISTSERV could generate 'Date:' fields with a time zone of 'TZONE'. Fixed. All Files with a fileid of the form 'llll LOGaaaa' where 'llll' is the name of an existing list and 'aaaa' is not numeric could be mistaken for archive files by the database driver. The code now checks that the four characters following 'LOG' are numeric but does not attempt to check that they represent a valid month. All Comments at the end of the 'Date:' field in incoming mail could cause the date to be flagged as invalid. Such comments are now accepted. All Miscellaneous bug fixes to the VMS version of LDBASE from Mark Strawcutter <MJSTRAW@IUP> have been incorporated. The main problem was that LDBASE could fail to quit properly in a cluster environment. All Loop detector did not handle improperly folded RFC822 fields in body of delivery error notice. Fixed. All The MAIL file format (ie 'F=MAIL') was not supported for AFD. LISTSERV now properly handles AFD subscriptions with F=MAIL; a DISTRIBUTE MAIL job is generated in response to such requests. All The REPRO subscription option was not handled properly when the subscription address (in the LIST file) and the sending address ('From:' field) were different, but equivalent as per the ':internet.' tag. New In order to avoid causing downstream servers to run out of virtual memory or disk space when processing distributions to very large lists, LISTSERV will now ensure that outgoing DISTRIBUTE jobs do not exceed a certain number of recipients, as specified by the MAXDISTN system variable (default value: 1000). That is, multiple jobs with the same data will be generated, whenever necessary, to keep the individual recipients counts below MAXDISTN. New The POSTMAST and POSTMASTER userids are now accepted as valid node administrator sources. Note however that requests from these addresses will not work with servers running an older version. New Performance of the GET command has been improved for files stored under their real fileid. In particular, requests for copies of BITEARN NODES will take less CPU time and will not require an increase to the size of the 192 minidisk. New New loop-detector option, "Loopcheck= NoOrigin", permits the list owner to disable the check for "known mailer origins" such as MAILER, POSTMASTER, ROOT, UUCP, et al. Mail whose 'From:' field is the address of the local mailer is still trapped, but wildcard checks on the mail origin are disabled. ***************************************************** * Usability: Support for the CMS Shared File System * ***************************************************** WARNING: read the restrictions and usage notes carefully before migrating ANY minidisk to the SFS! This is not a trivial exercise, due to the inherently different structure of SFS and minidisks. Starting with release 1.7b, LISTSERV provides support for the SFS under CMS release 7 or higher, subject to the 3 following restrictions: 1. The 'ACCESS' command, when issued without any parameter, must cause LISTSERV's A-disk to be re-ACCESSed properly. This is non-trivial due to the unintuitive behaviour of CMS when both a 191 minidisk and a default filepool are present: the minidisk is ignored and CMS accesses the top directory at mode A. Thus, if you decide not to migrate the A-disk to the SFS, you must ensure that no default filepool is defined. On the other hand, if you want to migrate the A-disk to the SFS, a default filepool must have been defined *and* the A-disk must be the top directory in that filepool. 2. The D-disk must not be moved to the SFS; it must remain a minidisk at virtual address 192. While this restriction might be removed in a future version, you should bear in mind that this is the place where LISTSERV places all its temporary work files: having it as a minidisk does have a number of advantages. 3. Care must be taken to ensure that LISTSERV is guaranteed a reasonable amount of free space on its A-disk, regardless of how much space has been used up on other disks (list archives, file-server data, etc). As of the CMS7 version of SFS, this means that the A-disk must either be a minidisk or reside in a dedicated file pool, with no possible interference from the other disks. Provided these restrictions are met, you can freely mix SFS directories and minidisks as you see fit: simply supply a fully-qualified SFS directory name (eg 'SERVERS:LISTSERV.PUBLIC', and not just '.PUBLIC') wherever LISTSERV would accept a minidisk address (not filemode). Thus: "Notebook= Yes,L,Monthly,Public" becomes: "Notebook= Yes,X1/SERVERS:LISTSERV.PUBLIC,Monthly,Public" Again, a SFS directory name is not an acceptable substitute for a filemode: you must use a syntax form where minidisk *addresses* are accepted, and replace the virtual address with the directory name. In order to reduce the potential for misconfigurations resulting in potential disruption of production service, you are advised to use one of the following recommended configurations, especially if you feel you are not proficient enough with the SFS yet to fully understand the implications of migrating minidisks to SFS directories. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Recommended SFS configuration 1: SFS for shared files only <<<< <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< For a site using a mixture of SFS and minidisks, the most logical and most manageable configuration is to use the SFS for "shared" files (ie list archives, file-server data) only. Keeping a 191 minidisk ensures the A-disk will not fill up unexpectedly while writing to a notebook file, and should provide better performance than a SFS directory - especially as the top directory is always a file control directory with higher file-open overhead. On the other hand, using the SFS for list archives and the like gives a much better level of control over who can access what archive files, while users don't have to worry about re-ACCESSing the public minidisk to gain access to new postings. In that configuration, the 191 minidisk is unchanged, there is NO default file pool, R/O files (BITEARN NODES et al) still come from a R/O link to a public minidisk, and selected minidisks are migrated to SFS directories (directory control directories are recommended unless you want to use access control lists at the file level). While doing so, make sure to change: - PROFILE EXEC, if you were ACCESSing "static" minidisks from there. - The MDISK./CHECKMDISK entries in LOCAL SYSVARS: fully-qualified directory names replace minidisk addresses. Note that, since the SFS does not presently allow you to define quotas for individual directories, the concept of utilization ratio has no meaning at the directory level. The values used when checking the warning and logout thresholds are those related to the file pool. In other words, if you use a single file pool for all your directories and LISTSERV has used up 61% of its authorized quota in this file pool, all directories will be said to be 61% full. - The "Notebook=" keyword in all affected lists. - All FILEID files referring to that minidisk. WARNING: if you manually set up a default file pool for your convenience after logging on to the server to edit files, do not forget to undefine it before starting up LISTSERV! You may use the new (privileged) SHOW STORAGE command, after starting up LISTSERV, to check the utilization ratios of all accessed minidisks and SFS directories. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Recommended SFS configuration 2: SFS for all files <<<< <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Some sites have a strategic commitment to SFS and may want to migrate all their services to SFS, unless there is a very good reason for not doing so. In such a configuration, the only minidisk that would remain is the D-disk (192), which presently cannot be moved to SFS. The checklist for this configuration is the following: 1. Enroll LISTSERV in a first file pool, which should be made the default file pool via with the IPL directory statement, and move all files from the 191 to the top directory on that file pool. 2. Enroll LISTSERV in a second file pool and migrate all the other minidisks (except the 192) to directories in this file pool (for this step, the instructions are the same as for the previous recommended configuration). 3. Log LISTSERV off and log it in from a terminal. Once the IPL procedure completes, check that the default file pool is that of the A-disk, and issue an 'ACCESS' command with no parameter to make sure the right directory gets accessed again. Check that the 192 has been brought up as filemode D (if not, add an 'ACCESS 192 D' command to PROFILE EXEC). 4. If the R/O configuration files (BITEARN NODES et al) are in a SFS directory control directory, LISTSERV will never notice that they have been updated, due to the nature of the SFS. You are thus strongly advised to: a. Cause LISTSERV to reboot once a day, by adding an entry to WAKEPARM FILE as follows: ALL 23:00:00 00/00/00 CP MSG * STOP REIPL b. Re-ACCESS the directories in question every hour (for instance), by adding a WAKEPARM FILE as follows: ALL &01:00 00:00:00 CMS ACCESS whatever **************************************************** * Performance: Introduction of the "PREXX library" * **************************************************** One of the major changes in release 1.7a was the introduction of support for the IBM REXX compiler. "Live" tests carried on production servers have since then indicated that compiling LISTSERV results in CPU savings on the order of 50%, while the amount of virtual storage required by the code is considerably increased (just the EXECLOAD'ed files add up to 1.3M of virtual storage, as compared to about 300k). While this is obviously better than nothing, the REXX compiler is clearly not the panacea that will solve the "LISTSERV CPU bill" problem - especially as smaller systems may have a hard time accomodating the increase in virtual storage that it requires. The "PREXX library" is a set of VS PASCAL callable subroutines that will make it possible to convert LISTSERV's REXX code to PASCAL. These subroutines include usable and efficient string manipulation, high-performance interface with the file system, and a REXX interface making it possible for PASCAL procedures to be called as REXX functions, transparently to the caller. In order to avoid conflicts with locally-provided RXxxxFN function packages, a different mechanism (via an explicit bootstrap) has been used: RXUSERFN and RXLOCFN remain available for local use. IMPORTANT: The PREXX library is not supported for local use. It was designed for the sole purpose of converting LISTSERV code to PASCAL. Problems occuring in different environments will not be addressed, unless they would also affect PREXX-based LISTSERV code. Questions related to the use of the PREXX library in a non-LISTSERV environment will not be answered: this is a purely internal development tool. With release 1.7b, two of the most frequently called REXX functions, LSVNADDR and LSV822TO, have been converted to PASCAL; in addition, part of the list archive database driver (LSVDBNB) has been rewritten in PASCAL, under the name LSVDN1. The two CPU-intensive functions require 10 to 15 times less CPU time than their REXX counterparts (counting the seizable SVC 202 overhead during the calling sequence in the REXX interpreter), while the I/O intensive database driver now generates a database index in about half the CPU time an 'EXECIO * DISKR (SKIP' on the input file would require. The new (privileged) command SHOW PREXX can be used to obtain statistics on PREXX functions similar to those formerly supplied by CMS EXECMAP/SHOW EXECLOAD. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Important note to developers: renamed REXX files <<<< <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Application developers should take note of the fact that numerous key REXX files had to be renamed as a consequence of the introduction of PREXX. Since PREXX functions are not written in REXX, they have to be declared to the REXX interpreter as "external functions" - in much the same way as assembler REXX functions such as LSVFIF. Due to the interface chosen by the developers of the CMS REXX interpreter for that purpose, only the first 6 characters of the function name are significant. For instance, it is not possible to write a REXX function called CMSFLAME EXEC, as any reference to CMSFLAME() in a REXX program will cause the (IBM-supplied) external function CMSFLAG() to be called instead (with no error or warning message). Since it is not possible to bypass this mechanism, LISTSERV REXX files whose first 6 characters are not unique must be renamed before they can be converted to PREXX. In order to simplify maintenance, this conversion is being done with release 1.7b for all REXX files whose conversion is likely to take place in the foreseeable future. Files whose first 6 characters are not unique have thus been split into three categories: 1. Functions which it is reasonable for an application developer to call. For these files, a "shell" REXX file has been provided under the old name, in order to preserve compatibility of existing locally-written applications. The "shell" merely calls the new routine, passing on the result if appropriate. Still, you are strongly advised to update local code to refer to the new names, both for performance reasons and because the functions will unavoidably be discussed under their new names on mailing lists, given enough time for the old name to be forgotten. There is no deadline for this conversion, though, as the "shells" will remain on LISTSERV's A-disk until you erase them manually. A list of these shells can be found in RXSHELLS FILE (which will be updated if more shells are created). 2. Functions which it is not reasonable for an application developer to call directly. These files have been simply renamed without leaving any "shell" behind. 3. Functions which will not be renamed because there is no plan to ever convert them to PREXX: LSVINST* EXEC and LSVUDD* EXEC. IMPORTANT: "Shells" are provided only for compatibility reasons. They are not a "stable programming interface". You are encouraged to erase them if you have no local software relying on the old name; in addition, new LISTSERV installations will NOT be sent copies of shell files. Here is a list of the function name "clashes" and the way they have been addressed: LSVBESTP Unchanged LSVBESTE -> LSVBSE (Shell supplied) LSVCHK Unchanged LSVCHKBN -> LSVCKB (Added after 1.7a was released, so no shell) LSVCKPRV Unchanged LSVCKPRM -> LSVPRM (Shell supplied) LSVCKPUN -> LSV00D LSVCMD Unchanged LSVCMDFW -> LSVCFW LSVCMD1 -> LSVCM1 LSVEXPIR Unchanged LSVEXPLD -> LSVXPL LSVFILID Unchanged LSVFILEV -> LSVFLV LSVGET Unchanged LSVGETFD -> LSVGFD LSVHELD Unchanged LSVHELP -> LSVHLP LSVNAD -> LSVADM (Shell supplied) LSVNADDR -> LSVADR (Shell supplied; LSVADR converted to PREXX) LSVSTOP -> LSVSTP LSVSTORE -> LSVSTL LSVUPFID -> LSVUFI LSVUPFL -> LSVUFL LSV822IN -> LSV82I (Original preserved) LSV822TO -> LSV82T (Original preserved; LSV82T converted to PREXX) Note: for the LSV822* functions, the original REXX files have been preserved because these files are made available from TOOLS FILELIST as public domain RFC822 "building blocks" for application developers. The PREXX counterparts are not in the public domain and are not plug-compatible (the PREXX library requires explicit initialization). As a result of this conversion, the following EXEC files will no longer be present on LISTSERV's A-disk after the installation of release 1.7b: LSVCHKBN LSVCKPUN LSVCMDFW LSVCMD1 LSVEXPLD LSVFILEV LSVGETFD LSVHELP LSVSTOP LSVSTORE LSVUPFID LSVUPFL ************************************** * Usability: IETF-style mail headers * ************************************** WARNING: There are a number of important warnings in this section. Read them carefully. In particular, IETF-style headers must NOT be used by LISTSERV-to-usenet gateways. DISCLAIMER: All references to the discussions of the IETF "listserv working group" reflect the author's personal appraisal of the debate on the working group's list, as of late august 1991 (shortly before the code being described was written). At this time, the group did not expect to produce a RFC (other than a possible informational one) in the immediate future, and the first non-informational RFC was not expected to address the topic of mail headers. This topic has, however, been thoroughly debated on the list, and it is this debate that the author's comments are based upon. The working group archives are available via anonymous FTP from FTP.UTDALLAS.EDU, should you want to form you own opinion on the matters at hand. Up to and including version 1.7a, LISTSERV provided two flavours of RFC822 headers: SHORT, with just the "essential" fields, and FULL, with all fields (including those usually deemed "superfluous", such as 'Received:'). For both types of header, two delivery mechanisms were supported: "regular" delivery, with the recipient's address listed in the RFC822 'To:' field, and "BSMTP" delivery, with a generic 'To:' field not pointing to the recipient. This resulted in four possible header options: SHORTHDR, SHORTBSMTP, FULLHDR and FULLBSMTP. The only difference between the HDR and BSMTP options is the contents of the 'To:' field, which for the present discussion is immaterial: we only need to consider the SHORT vs FULL attribute of the header. Due to their different culture and habits, Internet people are usually not satisfied with either option. Most Internet users (or at least most of the vocal ones) read mail from workstations and use sophisticated user interfaces which hide "uninteresting" header fields automatically; they are thus not interested in any removal of "superfluous" tags at the sending point (and contend that the bandwidth saved is immaterial, given T1-speed access). They are furthermore used to SMTP alias lists, which are basically a feature of Internet mail systems making it possible to have the mailer replace a given mailbox with a list of mailboxes on the fly; in other words, by sending mail to [log in to unmask] your message is automatically sent to a number of people as it reaches the host Y.EDU, by causing the mail system to internally expand the recipients list without touching anything else. In particular, the mail header is unaltered and still reads 'To: [log in to unmask] - the 'To:' field is where you should look for the list name. The IETF's "standard listserv" RFC's will be based on this type of headers, and disallow the LISTSERV-style ones. In order to allow BITNET users to become familiar with Internet-style headers and to decrease the rate of incoming "suggestions" for future development of LISTSERV from Internet users forced to subscribe to BITNET-based mailing lists for professional reasons, a third type of mail header is being introduced: 'IETFHDR' (which always uses BSMTP delivery). That is, one would send a 'SET XYZ-L IETFHDR' command to LISTSERV to switch to IETF-style headers. For a regular, non-peered list, the IETFHDR is the header LISTSERV received from the mail system with exactly three changes: 1. A 'Received:' line is added on top of the header. 2. A 'Message-ID:' field is added, if none was present in the incoming message. 3. The 'Sender:' tag is set to the value of the "Sender=" list header keyword, if one is supplied, or to 'OWNER-listname@hostname' (in lower case). For instance, a posting sent to the list [log in to unmask] would have a default 'Sender:' tag value of [log in to unmask] It is important to understand that these headers are not being provided in answer to requirements from non-DP-literate users. They are being provided to answer requirements from Internet specialists; it is "their" headers, not "ours". They are not meant to be user-friendly and are not open to negotiations. They will not spawn sub-options with various bells and whistles added. They are here to allow LISTSERV to emulate an SMTP redistribution list if desired, and nothing more. Note that this does not mean that LISTSERV will be converted to become compliant with the upcoming "IETF listserv" standard, nor that setting a list to use IETFHDR for all subscribers will make it IETF-compliant. It is the author's opinion that the "IETF listserv" standard will not make it possible for LISTSERV to be compliant without becoming totally incompatible and ceasing to address the needs of some categories of BITNET users. In particular, the very fact of allowing users to request, no matter how explicitly, that headers be delivered in a format other than IETFHDR would be a violation of the future standard, assuming no change in the direction of the working group from what the author experienced during his participation. Since the "IETF listserv" is going to be a fully-specified standard and one of the goals of the "IETF listserv" working group (again, assuming no change of direction) is to see to it that it is implemented in a portable way, it seems more appropriate to have whatever new software the Internet comes up with replace LISTSERV, rather than striving to turn LISTSERV into a poor emulation of the latest fashion. Indeed, it would seem unlikely for more than a small minority of sites to want to run such a service on an expensive mainframe if it can be provided as easily on cheaper hardware. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> IMPORTANT WARNINGS regarding the IETFHDR option <<<<<<<<<<<< <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< A. LISTSERV does not alter the header beyond the three points mentioned above: insertion of a 'Received:' line, generation of a 'Message-ID:' if not already present, and generation of a 'Sender:' field. This means LISTSERV cannot guarantee that the header is syntactically or semantically correct, beyond the fields it generates itself. Because of this, THE IETFHDR OPTION MUST BE NOT USED FOR SUBSCRIBING PROGRAMS WHICH REACT ON THE CONTENTS OF THE HEADER. This means you must not use this option for a usenet gateway, but you can subscribe a "logger" program which just appends everything it receives to a disk file. B. LISTSERV does not (and cannot) take control of the "owner-listname" address, which is normally too long to be a valid VM userid. It is the responsibility of the list owner to ensure that an alias for this address is set in the mailer at the site hosting the list if he wants to receive delivery errors sent to that address. C. In a peered-list environment, the IETFHDR option will operate properly only if ALL peers are running release 1.7b or higher. If this is not the case, some postings sent to IETFHDR subscribers may appear with a FULLBSMTP-like header; this is not a bug, but a consequence of the simple fact that the original header has not been preserved by the peer to which the posting was sent. D. If the posting received by LISTSERV comes in a format other than RFC822 (eg PROFS), LISTSERV will generate a new header out of the information at hand. Such a header may look "suspiciously short", so it was felt necessary to mention this possibility. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> End of important warnings <<<<<<<<<<<<<<<<<<<<<<< <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Here is an IETFHDR-style header example, built from an INFO-VAX contribution using a test list (L201@SEARN) for formatting the headers: ------------------------------------------------------------------------- Received: by SEARN (Mailer R2.05) id 0023; Fri, 18 Oct 91 15:23:22 +0100 Received: from SEARN.BITNET by SEARN.SUNET.SE (LISTSERV release 1.7a) with NJE id 0005 for [log in to unmask]; Fri, 18 Oct 1991 15:22:52 +0100 Received: from DEARN by HEARN.BITNET (Mailer R2.07) with BSMTP id 4454; Thu, 17 Oct 91 14:28:43 CET Received: by DEARN (Mailer R2.08) id 4111; Thu, 17 Oct 91 14:27:44 CET Received: from vtvm2.cc.vt.edu by DEARN (Mailer R2.08) with BSMTP id 5972; Thu, 17 Oct 91 11:53:31 CET Received: by VTVM2 (Mailer R2.08 R208002) id 6841; Thu, 17 Oct 91 06:54:17 EDT Received: from VMA.CC.CMU.EDU by vtvm2.cc.vt.edu (Mailer R2.08 R208002) with BSMTP id 6831; Thu, 17 Oct 91 06:52:10 EDT Received: by CMUCCVMA (Mailer R2.04) id 2335; Thu, 17 Oct 91 06:48:43 EDT Received: from UGA.CC.UGA.EDU by VMA.CC.CMU.EDU (Mailer R2.04) with BSMTP id 2331; Thu, 17 Oct 91 06:46:33 EDT Received: by UGA (Mailer R2.07) id 1757; Thu, 17 Oct 91 06:49:07 EDT Received: from CUNYVM.BITNET by UGA.CC.UGA.EDU (Mailer R2.07) with BSMTP id 1753; Thu, 17 Oct 91 06:48:35 EDT Received: from CUNYVM by CUNYVM.BITNET (Mailer R2.08) with BSMTP id 1159; Thu, 17 Oct 91 06:48:35 EDT Received: from CRVAX.SRI.COM by CUNYVM.CUNY.EDU (IBM VM SMTP R1.2.2MX) with TCP; Thu, 17 Oct 91 06:48:33 EDT Received: From DANPOST.UNI-C.DK by CRVAX.SRI.COM with TCP; Thu, 17 OCT 91 03:01:39 PDT Received: from vms2.uni-c.dk by danpost.uni-c.dk (5.65/1.34) id AA05308; Thu, 17 Oct 91 10:01:27 GMT X-Envelope-To: [log in to unmask], agate!spool.mu.edu!uwm.edu!zaphod.mps.ohio-state .edu!wupost!micro-heart-of-gold [log in to unmask] X-Vms-To: UNIL63::IN %"agate!spool.mu.edu!uwm.edu!zaphod.mps.ohio-state.edu!wupost!micro-h eart-of-gold.mit.edu!news.bbn.com!star-trek!cmiksis @UCBVAX.BERKELEY.EDU" X-Vms-Cc: IN::"[log in to unmask]" Message-ID: <[log in to unmask]> Newsgroups: bit.listserv.info-vax Date: Thu, 17 Oct 1991 11:01:00 +0200 Sender: [log in to unmask] Comments: Warning -- original Sender: tag was INFO-VAX@DEARN From: <deleted for privacy reason> Subject: Re: Would should a descriptors length be? To: L201@SEARN (message text removed) ------------------------------------------------------------------------- Note that the contents of the list archives are always kept in a special version of the SHORT header style (without 'To:' field). This is the only format the database driver can handle, and with the IETFHDR format there would be no guarantee that all the information required by the driver is present and valid. If you want public archives in IETFHDR format, just subscribe a "logger" program on your local anonymous FTP workstation to the list with the IETFHDR option. *********************************************************************** * Clarification: File requests from NJE nodes with Internet addresses * *********************************************************************** This item does not describe a change but merely documents more clearly the behaviour of LISTSERV, when processing file requests (such as 'GET SOME FILE') sent via mail with the Internet address of a node which also has BITNET access and has properly registered a ':internet.' tag in its BITEARN NODES entry. Contrary to what experienced users expect, the file is usually sent back via mail, and not via the preferred file format registered in BITEARN NODES. This is not because LISTSERV fails to recognize that the request is coming from a node with BITNET access, nor because it fails to read the preferred file format for that node; it is a design decision. The requirement here is that non-technical users, who only know how to use mail and have little or no knowledge of BITNET or the operating system they are using, should not be forced to learn about non-mail commands they have no need for. Thus, if such a user mails a GET request for a plain text file to a list server, he expects the file to be deposited in his mailbox, in the same format as the rest of his mail. Sending the file in (say) Netdata format just because this user happens to be connected to BITNET and the node can handle Netdata would make life much more difficult for him: he would get the impression that the file never arrived, and have to call user support to find out how to copy it to his "normal mailbox". On the other hand, a BITNET-literate user would surely know how to extract a text file from a mail message. Note that, if LISTSERV determines that the requested file will not be usable if sent via mail, it will bypass LISTSERV-Punch encoding (which requires special processing from the user anyway) and use RSCS to deliver the file. The rationale here is that the level of local support for (say) Netdata is likely to be much higher than for LISTSERV-Punch: in most cases, there will be an operating-system command to convert Netdata format, whereas LISTSERV-Punch usually requires the user to obtain and compile his own conversion program. In addition, LISTSERV-Punch does not always preserve binaries. If the file request comes via mail but with a .BITNET address, the result will NOT be sent via mail. The assumption here is that BITNET sites with Internet access configure their mail system to use their Internet hostname by default in outgoing headers. Thus, if the user is writing from a site with Internet access, he knows about the Internet/BITNET duality, knows enough about the mail system to change the default hostname, and presumably had a good reason to want to do that: such a user is most probably BITNET-literate. The case of a user writing from a site without Internet access can be debated endlessly, as such sites may indeed have users that only know about mail. The behaviour was kept the same whether or not the site has Internet connectivity for three reasons: 1. It is what LISTSERV always did for these users, and there has been no change on the user's side. On the other hand, in the case of Internet sites using their Internet hostname in mail headers, there HAS been a change on the user's side (even though it may not have been the user's decision). 2. It is easier to explain to non-technical users. 3. The behaviour does not change unexpectedly if a ':internet.' tag is added (and does not vary according to the version of BITEARN NODES the various servers are using). A BITNET-literate user wishing to order a file via mail (because the link to the server is down) and get it in (say) Netdata format can do so by sending the request as a non-mail file, by using a .BITNET address in the 'From:' field, or by explicitly selecting the file format via the 'F=' keyword (eg 'GET SOME FILE F=NETDATA'). One easy solution is to write a procedure, with the same syntax as the command you use to send interactive messages, that places the message in a one-line file and sends it to the recipient. Thus, after trying a request via (assuming VM/CMS) 'TELL LISTSERV AT XYZ', you would just retrieve your command, replace 'TELL' with (say) 'TN', and hit ENTER again. ********************************* * Clarification: AFD/FUI design * ********************************* This item does not describe a change but merely documents more clearly the design objectives of the AFD/FUI functions and, in particular, their behaviour when presented with Internet addresses. The purpose of the AFD function is to take advantage of BITNET's unsolicited file transfer function and make it easy for users to subscribe, mainly but not exclusively, to software packages. In order to present a more uniform interface, Internet addresses are also supported. Unfortunately, the design objectives do not match the expectations of a number of list owners and LISTSERV maintainers. Based on the design objectives, it is not reasonable for a mail-only Internet user to subscribe to an object file via AFD. If he does so, he will be sent a file via mail but that file will probably not be usable due to character code translation and the like. This may be unfortunate, but it is consistent with the fact that the same thing happens if a GET command is used to order the same file. In other words, this is not a problem with the AFD function. It is common practice for list owners to set up a file archive associated with their list; this file archive may contain basically any type of material in support of the activities of the list. In many cases, this archive will only contain plain text files (minutes, papers, etc). These files might be quite large, and it may happen that a subset of the list membership is not interested in receiving these files. It may thus be desirable to refrain from posting a copy of the files to the list, and merely announce their availability (via the GET command) for interested parties. Such a setup makes it natural for interested users to subscribe to the files via AFD, so that they do not have to use GET every time. From this point it may be tempting for the list owner to consider the AFD subscriptions as an extension of his mailing list, and to attempt to manage it in the same way as the list - sending administrative requests on behalf of its members. These requests are, however, rejected by LISTSERV due to lack of suitable privileges. One of the cornerstones of the design of AFD is that the user maintains his AFD list by himself, and that subscribing to a file is a voluntary act. His node administrator is, of course, allowed to make changes on his behalf, which is consistent with the general LISTSERV policy of allowing registered administrators to act on behalf of their users. The file owner, however, is not allowed to subscribe people to one of his files just because he owns it, nor to delete existing AFD subscriptions. The potential for abuse and the integrity exposure is too high to allow such functions to be performed by just anyone who owns a file. Since AFD subscriptions are usually locked with a user-chosen password, toying with RFC822 headers does not work either. The bottom line is that AFD is not a mailing list for documents, and should not be used for that purpose. If your "business need" is to have a list of people interested in getting a copy of the voluminous documents you are posting, AND you require the ability to manage it yourself like a mailing list, then what you need is, simply, another mailing list, to which you can post new documents as they become available. You can even AFD the list address to the files with F=MAIL to automate this process. *************************************************** * Serviceability: Changes in RFC822 quoting rules * *************************************************** RFC822 requires names to be quoted in fields such as 'From:' and 'To:' whenever they contain one or more characters from the SPECIALS or CTL sets. SPECIALS, being an enumerated list of characters (angle brackets, period, comma, etc) is unambiguously defined on EBCDIC machines. CTL on the other hand represents all ASCII control characters, and the corresponding EBCDIC meaning is open to interpretation. While most IBM systems programmers consider "EBCDIC control characters" to refer to the X'00'-X'3F' range, the VM mailer uses a more exotic definition whereby basically anything but A-Z and 0-9 is a control character. This causes problems with "national characters" in most languages - LISTSERV does not quote them, since it does not consider national characters to be EBCDIC control codes, and the VM mailer refuses to process the message, since it has a different definition. Even though the author disagrees with the definition the VM mailer is using and takes objection at the ethnocentric behaviour it results in, LISTSERV has been modified to force quoting in all cases where it is not unambiguously obvious that the name contains no special or "control" character. The list of characters not causing quoting has been double-checked against version 2.05 of the VM mailer. ***************************************************************** * Serviceability/Performance: Improvements to the SERVE command * ***************************************************************** The effects of a SERVE OFF command from the LISTSERV postmaster are now permanent - only a SERVE ON command by a postmaster will restore access to the user. In other words, it is no longer possible for a user who has been banned by a LISTSERV postmaster to restore his access himself, using someone else's account. Apart from the obvious benefits when dealing with malicious users, it ensures that disabled gateways are not inadvertently re-enabled by well-meaning end-user who do not understand all the implications their attempt. In addition, in order to reduce the size of the SERVE data in LASTING GLOBALV, decrease virtual storage utilization and improve overall performance, only data effecting a change in SERVE status (from OFF to ON or vice-versa) are written to permanent GLOBALV variables. In other words, the counters are reset every time LISTSERV is rebooted, although locked users remain served out until their access is manually restored. Finally, LSVCLGBV has been modified to remove any LASTING GLOBALV record containing a SERVE counter not causing access to be denied. Note that the savings in virtual storage will not become apparent until you have rebooted LISTSERV (after running LSVCLGBV). ************************************************************************ * Serviceability: "Welcome message" bounces now returned to list owner * ************************************************************************ LISTSERV-generated list administration messages (including the "Welcome" message sent to new subscribers) now include a special 'X-LSV-ListID:' header tag that makes it possible for LISTSERV to forward delivery errors caused by these messages to the list owner, rather than the postmaster, PROVIDED that the mail system generating the error message includes a copy of the original header in the delivery error (which is usually the case). Since some mail system do not even mention the failing address (eg "Unknown xyzmail error 133 processing your mail dated **unknown**"), it is not possible to guarantee that this will always happen. Note that setting the mail origin to that of the list owner is not practical, as the wording of these administrative messages is not suitable for direct person-to-person (rather than computer-to-person) mail, and the owner probably does not want to receive the "Mail sent to..." messages from VM mailers. ********************************************************** * Serviceability: SHOW STORAGE and "REXX traceback trap" * ********************************************************** Recent bugs in CMS (a storage cancer bug in CMS 5.x and a REXX interpreter bug in CMS 6) have caused a large amount of REXX errors to be reported to LISTSERV coordination recently. These automatic reports from the server are not very useful, because they only indicate that a REXX error took place while processing a particular command; they do not include any indication regarding the nature of the error (since the error does not happen within the program detecting and reporting it but as the result of a CALL statement, the REXX SIGL and ERR variables are not meaningful in this context). Due to the amount of error reports received, it is not possible to individually ask LISTSERV maintainers to supply console logs for each and every error. But in order to assist the site in solving this problem, LISTSERV coordination needs to know whether this is a "Machine storage exhausted", "Interpreter failure" or other error and, in case of storage exhaustion, how much storage is actually available to LISTSERV. The "REXX traceback trap" addresses the first problem by continuously recording console messages in a wrap-around buffer. When a REXX error occurs, the REXX interpreter traceback can be extracted from that buffer and included in the automatic error report (there is, regretfully but unsurprisingly, no standard REXX or CMS function to obtain a copy of such tracebacks). As a side-effect, it is now possible for LISTSERV to return the last 20 lines of trapped console output to the issuer of a CMS command (ie 'TELL LISTSERV CMS LISTFILE LSVX* EXEC' would return up to 20 lines of LISTFILE output). Note however that LISTSERV does not trap messages issued through certain console I/O services (the list of which furthermore varies according to the release of CMS you are using); there is thus no guarantee that all console output will be returned, and you should check the console log if in doubt. Again, the one and only purpose of this "traceback trap" is to trap error messages generated by the REXX interpreter, and anything else is just a fortunate side-effect. The new (privileged) SHOW STORAGE command displays virtual storage statistics including total amount of actually available storage and various figures related to storage fragmentation. This makes it possible to detect storage cancers by issuing the command at regular intervals. Note that it is normal for available storage to decrease for a while after restarting LISTSERV, as network activity cause files eligible for caching to be actually read and cached. This also made it possible to improve the boot-time warning about available storage, since the amount of actually available storage (rather than the virtual machine size) can be queried.