Description of the changes for release 1.8a of LISTSERV ------------------------------------------------------- Copyright L-Soft international, 1993 December 7th, 1993 ************** * Highlights * ************** Version 1.8a introduces the following major enhancements: - Significant performance improvements, and in particular order of magnitude improvements in the management of large lists, make it possible to successfully operate lists with thousands of subscribers on the smallest machines. This and other I/O-reduction changes, such as the introduction of a SIGNUP FILE cache, should make it possible to run LISTSERV on systems which by mainframe standards can be considered I/O-constrained, such as the P/370 or a high-end personal computer running S/370 emulation software. While at this point L-Soft international is not in a position to guarantee that any of these solutions will actually support LISTSERV, we plan to seriously investigate the P/370 as soon as TCP/IP support is added to the base offering, and review current S/370 emulators for unix and/or MS-DOS, subject again to the availability of native TCP/IP support to the emulated VM/SP image. - New template processor that lets list owners customize administrative messages without intervention by the LISTSERV maintainer. Formerly this was not possible without a LISTSERV modification that would jeopardize the security of all users on the LISTSERV host, even when the list owner could be trusted. Customers who decided to trade this risk for added functionality can now enjoy direct support for cutomized messages without exposing their users in any way. - Administrative messages have been rewritten to be more easily understood by non-technical users, or users whose native language is not English. As a result of the rationalization of command parsing code made possible by the conversion of most commands to a compiled language, error and informational messages have been improved and made more consistent to reduce end-user confusion. Error messages now give hints to help the user compose a syntactically correct command. - Support for list "exits" covering the major list management commands was introduced, providing additional flexibility in tailoring the behaviour of LISTSERV to specific local requirements. In particular, this makes it possible to create lists from which users are not allowed to sign off. - Support for delivery of large documents as multipart MIME messages, through the new 'SPLIT=' command keyword, which can be combined with any of the file format keywords for electronic mail (UUENCODE, XXENCODE, MIME/APPL, etc). - In order to simplify LISTSERV maintenance, the DOMAIN NAMES file will now be updated by LISTSERV Coordination (for BITNET sites). Customers will longer need to worry about keeping this file up to date. - Support for the RFC1312/MSP "Internet TELL" protocol was introduced, making it possible for non-BITNET users to issue commands to LISTSERV interactively. **************** * Availability * **************** Release 1.8a is the first commercial version of LISTSERV. An individual or collective "service license" is required to receive this update. For information on individual licenses, write to [log in to unmask] The following collective licenses were in force at the time of this writing: - EARN sites receive free access and free upgrades under a collective agreement valid until July 28th, 1994. Contact the EARN office for more information. In general, collective licenses are yearly agreements with volume discount that are renewed implicitly, but may be terminated at 3 months' notice. The dates listed above indicate the period for which you are guaranteed to receive free access to the software, if you qualify under the agreement, and do not necessarily imply that the agreement will not be renewed. Note that, in order to receive free service through a collective license, you must both qualify under certain criteria defined in the agreement (such as membership in a particular networking organization) and sign a contract indicating your acceptance of the licensing terms and conditions. In other words, you will have to go through some administrative steps the first time; the software will not be sent automatically if you have not done the necessary paperwork, even if you were already running an older version. Disclaimer: your rights and obligations are defined in the collective agreements. The summary in these release notes is necessarily incomplete, and is provided for your information only. The wording of the formal agreements takes precedence over these release notes. ******************************* * Supported operating systems * ******************************* Release 1.8 introduces the following changes to the list of supported operating systems: - (1.8a) Support for CMS release 10. Support for other versions of CP and CMS is unchanged from 1.7f. ************************** * External Compatibility * ************************** Release 1.8a is generally compatible with 1.7f, from the perspective of the end-user (including list managers and maintainers), and with the exception of the changes listed below. Release 1.8a introduces major changes to the LISTSERV internals, making compatibility with locally developed extensions or local modifications problematic. These changes are briefly described in the next section, and only affect sites which made alterations or additions to the LISTSERV code. Changes which affect all LISTSERV sites are highlighted below; note that more details are available, when appropriate, from the longer descriptive section of these release notes. 1. Support for the obsolete CHECKDISK configuration statement has been removed. CHECKDISK was replaced with CHECKMDISK and MDISK.cuu statements in 1988, but LISTSERV still accepted the old statement for compatibility reasons. This migration aid has now been removed. 2. To reduce spooling overhead and traffic over the BITNET core, the default header type for NJE recipients was changed from SHORTHDR to SHORTBSMTP. 3. For the same reasons, the minimum abbreviations for the SHORTHDR, SHORTBSMTP, FULLHDR and FULLBSMTP options were changed so that SHORT and FULL now refer to the BSMTP variants. 4. Support for the '.IT' (imbed text file) list header command has been removed. Only one site was known to use this historic facility, so it will not be described here; contact L-Soft support for more information. 5. The DELETE command no longer supports a MIXEDCASE option. Case is now always ignored when searching a list for a target address (formerly, case was ignored by default but an exact match could be requested with the MIXEDCASE option). The justification for this change is that the DELETE command was the only one supporting a MIXEDCASE option. There was no way to request case insensitivity on commands such as SET and CONFIRM, which made it virtually useless to have this support in the DELETE command only. Since this option was hardly ever used, it was decided not to carry it forward when the DELETE command was rewritten in PASCAL, as the library list interface routines are all case insensitive. 6. The public domain utilities LSV822IN EXEC and LSV822TO EXEC have been removed from the TOOLS FILELIST, along with LSV822 HELPCMS. This is to reflect the fact that these utilities are not supported by L-Soft international and are not part of the LISTSERV code. Users may of course continue to use these utilities; they are just no longer distributed with LISTSERV. 7. Support for the RSCS list processor has been removed. The main reason for this change is the increasing number of JNET sites acting as minor or regional BITNET hubs. JNET does not properly handle LISTPROC generated files with multiple dataset headers and this is a documented restriction for which no fix is available. Due to the nature of DISTRIBUTE, it is not generally possible for the LISTSERV manager to assess the probability that a LISTPROC file generated by LISTSERV will have to cross a JNET hub; this can happen even if there is no JNET site in the immediate vicinity. Since the result is permanent loss of data (the job is delivered in two halves, both of which are unusable and rejected by the target LISTSERV), it was felt that the only way to guarantee the reliability of the DISTRIBUTE backbone was to remove support for the list processor. It should be noted that LISTSERV can duplicate spool files more efficiently than the LISTPROC link, so there is no longer any reason to use the list processor for local resource/overhead considerations. 8. As announced in the 1.7f release notes, support for netwide DELETE jobs has been removed (netwide SIGNOFF is still supported). Sites running LMail, MX V3.3 or PMDF V4.2 need not worry about submitting netwide DELETE jobs for expired accounts as their mail system takes care of that automatically. Sites running older versions of MX or PMDF should upgrade to the current version. The specifications of the LMail/LISTSERV delivery error format are available from L-Soft support. ************************************************ * Warning: do NOT use netwide SIGNOFF instead! * ************************************************ Support for netwide DELETE has been removed because this method of broadcasting information about the deletion of hundreds of of thousands of local accounts every semester to the entire network does not scale up and placed an unbearable load on the hosts providing the LISTSERV service. Do not modify your procedures to use a series of netwide SIGNOFF jobs instead, as the overhead would be even higher. Netwide SIGNOFF remains for the convenience of individual users who know for a fact that they are subscribed to mailing lists, and for emergencies. If the LISTSERV backbone is flooded with tens of thousands of "preventive" netwide SIGNOFF jobs from users who have never been subscribed to any list, the overhead will be such that there will be no alternative but to remove netwide SIGNOFF permanently. 9. As a corollary to the removal of netwide DELETE and the conversion of the DELETE command to PASCAL, support for the "DD=" syntax has been removed. This syntax was needed only to support netwide DELETE in a more efficient manner. Simply issue multiple DELETE commands if you need to remove multiple users from a list. The setup overhead with PASCAL command is much lower and there is no noticeable performance impact unless hundreds of users are to be removed. Release 1.8a is to be installed directly on top of the base 1.7f code, and includes all the fixes available from LISTSERV@SEARN for release 1.7f (ie all the "17Fnnnt FIX" files from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits", which always have to be installed manually. ************************** * Internal Compatibility * ************************** Major changes have been made to LISTSERV internals. The most important ones are briefly described below; refer to the LSTSRV-L archives for more information. 1. RXLSVUHF and the various CACHE files have been removed. User-written programs that examined list header keywords by reading the CACHE file will need to be modified to use the permanent LSVKEYWD interface. 2. Due to a change in the list caching mechanism, certain unsupported list management practices that used to work in spite of their unsupported status will no longer produce the desired results. See below for more details. 3. The DOMAIN NAMES file will now be updated by LISTSERV Coordination. If you were making changes to DOMAIN NAMES before installing it on LISTSERV's A-disk, you will have to use the LSV$DNT user exit introduced in version 1.7f to implement these changes. See below for more information. The following REXX files have become obsolete with release 1.8a and are deleted during migration: LSVADD LSVCFIRM LSVCFW LSVCKLST LSVCKPRV LSVCM1 LSVDEBUG LSVDEL LSVDIST2 LSVFDATE LSVFOR LSVFREE LSVHOLD LSVHLP LSVINFO LSVLHDR LSVPRIME LSVREG LSVREMRK LSVSERVE LSVSET LSVSUBSC LSVXREQ LSVXSTAT LSV822IN LSV822TO The following files have become obsolete with release 1.8a: $DEFAULT MAILFORM RXLSVUHF MODULE * CACHE ************************** * Compatibility Warnings * ************************** This new section describes planned changes which may affect compatibility (both external and internal). The release in which the change is planned to be introduced is indicated, with the letter 'x' denoting an unknown release of the specified major version (not necessarily posterior to the last release explicitly mentioned). Note that conversion of more REXX files to PREXX is assumed to take place with each new release and is not indicated here. - (1.8b) LSVIMAIL EXEC will be removed (see "New mail template processor" below). While LISTSERV no longer uses LSVIMAIL, the program will not be removed until version 1.8b as it is used by the LMON package. This lets you install the 1.8a update without having to worry about impacting LMON. A new version of LMON using the new template processor will be made available shortly, and will be a mandatory update for 1.8b. - (1.8x) The internal format of FILELIST/FILEID and attendant files such as AFDLIST FILE will change dramatically. Applications which alter them directly will no longer work. - (1.8x) The internal format of LIST and STATS files will change dramatically. Applications which alter them directly will no longer work. - (1.8x) LSVFRD1 and LSVFWR1 will be removed eventually. You should avoid using them. ******************************** * Minor fixes and enhancements * ******************************** This section has been removed for this release, due to significant changes made to key functions such as list management commands and internals, DISTRIBUTE, administrative messages delivery, etc. About 8,000 lines of new PASCAL code have been written to replace existing command processors and internals. As a rule of thumb, and as explained in the 1.7c release notes, it is not possible to generate the "Minor fixes and enhancements" section for code which has been redesigned and rewritten from scratch in PASCAL. REMINDER (BITNET sites): Support for manual installation of INSTALL decks was withdrawn with release 1.7e. Use automatic installation with local password validation instead. Sites which insist on installing manually should note that shipments contain an installation pre- and post-processing exit. If that exit is not executed, or if it is executed in a different order or environment than during automatic installation, serious problems may occur. Manual installation is treated as a "Customer modification of licensed programs" under GA-9305-2(8)/GA-9312-3(6), where the "modification" is the manual CARD LOAD operation and the "licensed program" is the previous version of the software. ****************************************************** * Miscellaneous usability/human factors enhancements * ****************************************************** - The ADD command now accepts entries in RFC822 format, in addition to the "old" format. The following commands are now equivalent: a. ADD XYZ-L [log in to unmask] John Smith b. ADD XYZ-L John Smith <[log in to unmask]> c. ADD XYZ-L [log in to unmask] (John Smith) Note however that LISTSERV must first decide whether the address is in RFC822 format or in the old format. This additional difficulty may cause some exotic forms of RFC822 syntax to be incorrectly interpreted as "old format" addresses with unexpected results. - The SERVE function is no longer case sensitive, and will also act on registered Internet aliases. That is, serving Joe@XYZVM1 off automatically takes care of [log in to unmask] (assuming XYZVM1 has a :internet.XYZ.EDU tag in its BITEARN NODES entry). Note that the SERVE data is now kept in PERMVARS FILE rather than LASTING GLOBALV. The migration procedure will convert LASTING GLOBALV entries automatically. - LISTSERV will now wait until the 21th invalid command (formerly the 11th) before serving off a user. As before, up to 10 invalid commands are accepted before command processing stops. This makes it easier for users of systems such as QuickMail, All-in-one, PROFS, OV/VM or any other mail system which inserts a letterhead-like header in the message body before the text supplied by the user to send commands to LISTSERV. - The acknowledgements LISTSERV sends to confirm that a message has been posted now include the number of recipients. - The default loop check options have been changed so that 'To:' fields are no longer counted by default, ie all lists now operate with "Loopcheck= NoToCount" by default. This method of loop checking is long obsolete and may be disruptive on lists where large documents are regularly posted. The old behaviour remains available by adding "Loopcheck= ToCount" to the list header. - The FOR command now allows the LISTSERV administrator to issue commands on behalf of non-existent local users, such as mail-only aliases or addresses of the form owner-xxx, and will deliver command replies via mail. Note again that the FOR command is for debugging/troubleshooting only. LISTSERV commands which routinely need to be issued on behalf of other users all have a FOR option which is what you should use. For instance, 'SET XYZ-L NOMAIL FOR *@BADNODE' should be used rather than 'FOR *@BADNODE SET XYZ-L NOMAIL'. - When a reference to a remote, network-wide list (ie a list present in the output of a LIST GLOBAL command) is made, LISTSERV will no longer consider non-peered lists when there is a peered list by that name. That is, conflicts between peered and non-peered lists by the same name are now automatically resolved, with LISTSERV selecting the peered list. Many sites set up local redistributions of peered lists of interest, for access via a local bulletin board system or for more convenient access to database searches, and the maintainers often forget to mark these lists with "Confidential= Service". With version 1.8a, these omissions will no longer impact the end users' ability to subscribe to the list without specifying the host name. - The CHECKMDISK definition in LOCAL SYSVARS no longer causes an exception during startup if the total length of the variable exceeds 255 bytes. Note that this does not extend to other configuration statements. The 255 bytes limit comes from the GLOBALV facility, over which LISTSERV has no control. The CHECKMDISK variable is used only at startup and need not be recorded in the GLOBALV pool, unlike most of the other variables. ********************************************************** * Performance/Compatibility: New list caching algorithms * ********************************************************** The list caching algorithm has been redesigned with 1.8a in order to improve performance for medium-to-large lists. The 'listname CACHE' files become obsolete and will will be erased during installation of 1.8a. The new algorithm significantly reduces the amount of I/O operations performed by list management commands implemented in PASCAL (ADD, SUBSCRIBE, QUERY, SET, DELETE, SIGNOFF, CONFIRM, etc). For instance, a QUIET ADD to a 5000-users list only results in about 10 I/O operations rather than 300 (as usual, only the second and subsequent operations after a reboot benefit from this caching). As a result of these changes, a number of unsupported list management practices that used to work in previous versions will no longer be available. Unsupported practices that continue to work as before: - Editing the LIST file manually (with XEDIT) while LISTSERV is not running. - Deleting one or more subscribers from a LIST file via XEDIT while LISTSERV is running. - Changing the name (but not the address) of one or more subscribers while LISTSERV is running. - Removing blank lines from the LIST file while LISTSERV is running. Unsupported practices that will no longer work: - Adding new subscribers via XEDIT while LISTSERV is running: LISTSERV may later claim that the new subscribers are not on the list. The same applies to changes made to a subscriber's address. - Altering list header keywords via XEDIT while LISTSERV is running: the changes will not be picked up. - Using the 'listname CACHE' files as a means to obtain information about list header keywords: LISTSERV no longer creates these files, and as before the only supported mechanism for extracting list keywords is to use the LSVKEYWD function, which is a supported permanent programming interface. A temporary programming interface has been provided for cases where you absolutely need to modify LIST files via external programs while LISTSERV is running. You should make the changes as you used to and, after the last change is made and the file has been closed, but before returning control to LISTSERV, do: Call LSVFSI 'HR',listname You could place this instruction in your PROFILE XEDIT, for instance. If you specify '*' for the list name, all lists are affected at once. ******************************************************** * Usability: Support for RFC1312/MSP ("Internet TELL") * ******************************************************** Version 1.8a introduces support for the RFC1312/MSP interactive message protocol, through third-party MSP servers only. That is, LISTSERV does not attempt to take control of the MSP port or deliver MSP messages on its own, but relies on a separate virtual machine for this function. This avoids duplication of code and effort, providing a simpler and more robust interface to MSP for other servers and users. To enable MSP support, you must install suitable MSP software (see below) and register the userid of the MSP server (traditionally MSGD) in LOCAL SYSVARS. The corresponding variable is called MSGD, so the definition will usually read: MSGD = 'MSGD' /* MSP server runs under the 'MSGD' userid */ In order to be usable with LISTSERV, a MSP server must provide the following interface: 1. Incoming messages must be delivered via CP MSG or CP MSGNOH. The text of the (CP) message must be the following: From hhhhh(uuuuu): ttttt Blanks are allowed before the word 'From', and before or after the message text (ttttt). 2. The server must accept requests to deliver MSP messages via SMSG, with the following syntax: CP SMSG xxxxx M hhhhh uuuuu ttttt 3. When possible, it is recommended that the server be configured to reject messages from users for which no reverse address translation is available, when the target is a server such as LISTSERV (the MSP protocol provides a mechanism for rejecting messages with an error code). This usually happens when a name server is down. LISTSERV cannot process commands coming from addresses such as [log in to unmask] and will ignore any such message. A free MSP server for VM, written by Bert Barbe, is available from LISTSERV@BLEKUL11 under the name MSGD PACKAGE. At the time of this writing this was the only MSP implementation known to L-Soft that complies with points 1 and 2 above. Disclaimer: L-Soft international ("Licensor") does not represent or warrant that the foregoing information is accurate, that the MSP product mentioned above ("Software") will be freely available to Licensor's customers, that it will remain compatible with Licensor's products, or that it will operate to the customer's satisfaction. Licensor will not provide error information or correction for the Software, or assume any liability whatsoever concerning its use, including, but not limited to, liability for consequential damages, even in the event that Licensor had been notified of the possibility of such damage. The customer's use of the Software is at his own risk. Note: the word "Software" in the disclaimer refers to MSP, not LISTSERV. L-Soft will, of course, accept software trouble reports for LISTSERV from customers with a valid service license. *********************************** * Security considerations for MSP * *********************************** Before enabling MSP support, you should note that the level of security currently available through MSP is lower than what is available from NJE interactive messages. In particular, it is very easy for a non-privileged users to forge messages coming from a different user at the same host. ****************************************************************** * Maintenance: DOMAIN NAMES now updated by LISTSERV Coordination * ****************************************************************** In order to facilitate LISTSERV maintenance, the DOMAIN NAMES file will now be updated by LISTSERV Coordination, in the same fashion as PEERS NAMES, LINKSWT2 FILE et al. CONTROL FILELIST was modified to include an entry for the file DOMAIN NAMES with CMS fileid DOMAIN NAMES A1. This file will require about 50k of disk space, which is less than the amount saved by the installation of 1.8a (which is more compact than 1.7f). You need not worry about updating DOMAIN NAMES on LISTSERV any longer unless you were modifying this file before installing it on LISTSERV's A-disk. Since this procedure will no longer work, you will have to use the supported LSV$DNT exit introduced in version 1.7f to implement your changes in a reliable manner. Refer to the 1.7f release notes (available from LISTSERV@SEARN via GET V17F RELNOTES) for more information on the LSV$DNT exit. ************************************* * Usability: New list exit facility * ************************************* ------------------------------------------------------------------------- Background for non-technical users: an "exit" is a program supplied by the customer to modify the behaviour of a product (such as LISTSERV) in ways that the supplier of the product could not anticipate, or could not afford to support via standard commands or options. The product checks for the presence of the "exit" program and calls it on a number of occasions, called "exit points". In some cases, the "exit" program supplies an answer ("return code") to the main program, which adjusts its behaviour accordingly. For instance, LISTSERV may ask an exit program "Is it ok to add [log in to unmask] to the ABC-L list?", and the program will answer yes or no, and possibly send a message to the user explaining why his subscription was accepted or rejected. In other cases, the "exit point" call is purely informative: the exit program gets a chance to do something, such as sending an informational message to a user, but does not return any answer. Because the exit is a computer program, it must be prepared by a technical person and installed by the LISTSERV maintainer. ------------------------------------------------------------------------- Starting with version 1.8a, list "exits" will be available to control the major events associated with list maintenance. This will make it easier to tailor the behaviour of LISTSERV to local requirements that are too specific to be addressed through standard facilities. For now, the following commands make use of this facility: ADD, DELETE, SIGNOFF and SUBSCRIBE. More exit points may be added in the future based on user demand, although it should be noted that the exit facility is only vailable to commands and functions implemented in the PASCAL language. The LIST, REVIEW, GET, MOVE and EXPLODE commands are still implemented in REXX and will not have access to this facility until they are rewritten in PASCAL. An exit is enabled by adding "Exit= filename" to the list header. For security reasons, all exits must be explicitly declared in the new LIST_EXITS configuration variable (in LOCAL SYSVARS). This prevents list owners from causing the invocation of arbitrary executable files through the use of the "Exit=" keyword. However, once a particular program has been identified as a list exit, any list may refer to it. If you write exits which could compromise confidential data if executed on behalf of the wrong list, it is your responsibility to test the 'listname' parameter in the exit program and take appropriate action. Note that the exit name itself is available to anyone who can review the list; make sure to program the exit to test the list name, and not to rely on the confidentiality of the exit name. Under VM, list exits are regular REXX programs which must be prepared to handle calls from an arbitrary number of exit points. The parameter string always include the exit point name, followed by a blank, the name of the list, another blank, and parameters that depend on the exit point. In other words: Parse arg ep listname more The exit returns a numeric return code, which may or may not be checked, depending on the exit point. The value 0 always means "do the same thing as if there had not been any exit for this list". Always return the value 0 for exit point names your program does not handle or recognize. At the moment, the following exit points are available: ***************************** * ADD/SUBSCRIBE exit points * ***************************** Name: SUB_FILTER Parameters: One; originator's e-mail address Return code: 0=Accept, 1=Reject Description: This exit point gives you a chance to reject a subscription request by returning a nonzero value. This adds to rather than replaces the "Filter=" keyword. Name: SUB_FAIL Parameters: One; originator's e-mail address Return code: Ignored/reserved - always return 0 Description: This exit point is called whenever a subscription is rejected (in particular, it is called if you return 1 from SUB_FILTER). You may send additional messages to the user, log the event, and so on. Name: SUB_REQ Parameters: One; originator's e-mail address Return code: Ignored/reserved - always return 0 Description: The user's request for subscription is being forwarded to the list owners. You may send additional messages to the user, log the event, and so on. To implement custom subscription/rejection, use the SUB_FILTER exit with "Subscription= Open". When SUB_REQ is called, it is too late to let the user through. Name: SUB_NEW, ADD_NEW Parameters: One; originator's e-mail address Return code: Ignored/reserved - always return 0 Description: This exit point notifies you that the user has been successfully subscribed to the list (SUB_NEW is for the SUBSCRIBE command, ADD_NEW corresponds to the ADD command). You can send additional messages, log the event, and so on. ******************************* * DELETE/SIGNOFF entry points * ******************************* Name: DEL_FILTER Parameters: One or two; target e-mail address '15'x originator's address Return code: 0=Accept, 1=Reject Description: This exit point is called for all SIGNOFF commands, and for DELETE commands issued by a registered Node Administrator. It is NOT called for DELETE commands originating from the list owner. If you return the value 1, LISTSERV does not remove the target address from the list. If the request is rejected, you should check for the presence of the second parameter and send an explanatory message to that address, or to the target address if only one parameter was specified. Name: DEL_SIGNOFF Parameters: One; originator's e-mail address Return code: 0=Continue, 1=Do not notify owner Description: This exit point is called for the SIGNOFF command only, when a user has been successfully removed from the list. The farewell message, if any, has already been sent. You may issue additional messages, log the event, and so on. A return value of 1 directs LISTSERV not to mail the "SIGNOFF1" form to the list owners. Name: DEL_DELETE Parameters: Two; target e-mail address '15'x originator's address Return code: 0=Continue, 1=Do not notify owner Description: This exit point is called for the DELETE command only, after a user has been removed from the list. You may issue additional messages, log the event, and so on. A return value of 1 directs LISTSERV not to mail the "DELETE1" form to the list owners. ******************* * General remarks * ******************* IMPORTANT: List exits may not make any change to the LIST file, because the command processor may have changes of its own to make to the file, or may have kept the file open for further processing. If you really have to change the file, use CP MSG * CMS EXEC to schedule a new call to the exit after command processing completes. The new template processor can be called from list exits. The syntax is: Call LSVTMAIL recipients,template_name,listname,keywords keywords = 'KWD1 value of first keyword'||'15'x'KWD2 second keyword'||... Note the change in calling sequence from LSVIMAIL. Do not use LSVIMAIL as it will be removed in version 1.8b. ************************************************ * Usability: New "SPLIT=" command line keyword * ************************************************ The new "SPLIT=" command line keyword can be used when ordering large files via electronic mail throught networks or gateways that reject messages larger than a certain size (typically 100 kilobytes, although certain PC mail products have limits as low as 20k). It is not normally needed when ordering files via BITNET. When present, the SPLIT= keyword requests that the file be sent as a multi-part MIME message with the specified maximum size (in kilobytes). For instance, "GET XYZ.ZIP SPLIT=100" requests delivery of the file called XYZ.ZIP in messages of 100k or less. The minimum value for the SPLIT= keyword is 20 kilobytes, mostly to avoid bad surprises if you mistype the number and forget a zero. It is important not to confuse the SPLIT=nnn keyword with delivery format specifications, such as F=NETDATA or F=MIME/APPL. The F= keyword tells LISTSERV what format to use when sending the file to you (uuencode, MIME, one of the BITNET formats, etc). The SPLIT= keyword instructs LISTSERV to further cut the file into a number of smaller pieces before sending it to you; it does not replace but rather adds to the F= keyword. However, the SPLIT=nnn option is valid only for mail delivery formats: UUENCODE, XXENCODE, MIME/TEXT, MIME/APPL and of course MAIL. If you specify SPLIT= without selecting a file format, F=MAIL is assumed. The MIME multipart format was chosen because it is straightforward and reasonably intuitive to a human reader: people without a MIME user interface are not being sacrificed to a "chosen few". ******************************************************************** * Usability: Partial support for X.400 addresses containing blanks * ******************************************************************** To answer an EARN requirement, a number of changes have been made to selected LISTSERV internal functions in order to provide a temporary solution to the problem of X.400 addresses containing spaces. For technical reasons, it is not possible to provide a satisfactory solution to this problem until all commands and functions that somehow refer to e-mail addresses have been rewritten in PASCAL. With version 1.8a, X.400 users whose address contains blanks may issue commands to LISTSERV, join mailing lists, order files via the GET command, and otherwise use LISTSERV like any other user, with the following restrictions: 1. Some list owner commands where a reference to the user's address is necessary will require special attention. ADD commands will have to be issued in RFC822 format, using proper quoting, to prevent the blank from being interpreted as a delimiter between e-mail address and user name. DELETE commands will require the use of a wildcard. 2. Some error messages and the output of some commands may be improperly formatted. While this is harmless, it may, due to the general syntax of X.400 addresses, look like a LISTSERV bug or "stack dump" to an inexperienced user. Addresses that look like a list of keywords separated by semicolons or slashes are usually X.400 addresses, for instance: "G=joe;S=smith;PRMD=XYZ Inc.;C=US"@x400.xyz.com "/G=joe/S=smith/PRMD=XYZ Inc./C=US"@x400.xyz.com 3. Mail delivery may fail if the user's domain is served by a DISTRIBUTE server running a version older than 1.8a (or LISTEARN). *************************************************** * Usability/Security: New mail template processor * *************************************************** Release 1.8a introduces a new system for the generation of administrative mail messages, such as the messages sent to users as they subscribe to a list or are requested to confirm their subscription. This new system is easier to use, can be placed under direct list owner control, and will work without change under the new environments LISTSERV is being ported to, as it no longer relies on the REXX interpreter for expression evaluation via the INTERPRET statement. Unfortunately, this means that it was not possible to maintain compatibility with the old system, which allowed arbitrary REXX expressions and statements. Both the old and the new systems are based on a collection of "forms" or "templates", each of which corresponds to a particular administrative event, such as a subscriber requesting to be added to a mailing list. The templates are gathered together and stored as a single disk file, called listname.MAILFORM with the old system and listname.MAILTPL with the new system. When LISTSERV needs to use a particular template, it first looks in the file called listname.MAILTPL, if there is one. If the file is not present or if that particular template is not found in the file, it then searches a "language template" file (as specified by the "Language=" list header keyword), and finally examines the default template file, called DEFAULT.MAILTPL. To customize a particular administrative message, one simply orders a copy of DEFAULT.MAILTPL, extracts and modifies the template of interest, and stores the resulting file as listname.MAILTPL. It is not necessary to carry over the templates that need not be changed, and in fact it is not advisable to do so as your list would not benefit from improvements to these other messages in future versions. There is no change in this general structure; only the name of the template files and their format has changed. When you upgrade to version 1.8a, LISTSERV no longer examines the old MAILFORM files and all your list will revert to the standard messages. The individual MAILFORM files are not erased, however, except for the $DEFAULT MAILFORM file with all the system defaults. If you are using customized MAILFORM files, it is advisable to prepare the new MAILTPL files before the upgrade to 1.8a and place them on LISTSERV's A-disk (they will have no effect on 1.7f of course). Once the migration is over and the templates are converted to your satisfaction, you should erase the old MAILFORM files to avoid future confusion. The main advantages and end-user benefits of the new system are: - The single quote character no longer has any special role or meaning. Previously, single quotes had to be doubled in expressions such as "you're about to", or a REXX error would result. This was the single largest source of template errors and confusions. - Because the new mail template processor is a closed environment, list owners can be given direct access to the templates for their lists without jeopardizing system security. The individual MTPLATE files can be maintained in the same fashion as the WELCOME and FAREWELL message files, without any intervention by the LISTSERV administrator. List owners do not have to worry about their LISTSERV administrator's willingness to install and update customized mailforms for them, and the LISTSERV administrator is relieved from this clerical duty. With version 1.8a, any list owner with write access to a filelist can create customized mail templates for his lists. - The text and format of most administrative messages has been updated and reworded for easier comprehension by non-technical users. A more restricted vocabulary has been used for the benefit of non-native speakers. - Substitutions are now introduced by the ampersand (&) character, which makes them easier to identify than with the old quote-parity system. List header keywords can be queried in a simpler way than formerly possible; for instance, &KWD(NOTEBOOK,3) replaces 'Word(LSVKEYWD(listname,"NOTEBOOK"),3)' ******************************** * Format of the template files * ******************************** Each template starts with an form name and subject line, such as: >>> EXAMPLE1 This is the subject line The template starts with the line containing the form name and subject, and ends with the next line starting with '>>>', or at the end of the file. There is no ':END' line like in the old MAILFORM files. The form names are exactly the same as with the old system. As before, the subject line may contain substitutions (such as "&LISTNAME: &WHOM requested to join"). The template contains text and, optionally, formatting/editing commands, which start with a period in column 1. All other lines are treated as normal text: sequences starting with an & sign are substituted, then lines are joined together to form a paragraph, which is finally formatted like with any non-WYSIWYG text processor. You can suspend formatting with .FO OFF and resume it with .FO ON; when formatting is suspended, LISTSERV no longer joins lines to form a paragraph, but simply writes one line of text to the message for each line read from the template. This makes it possible to include tables or a text-mode logo, but can create seriously imbalanced text if substitutions are used. For instance, a typical &WHOM substitution can range from a dozen characters to 60 or more, even though it only takes up 5 characters on your screen when you enter it. The following substitutions are always available: &DATE Long-style date (29 Jul 1993) &TIME hh:mm:ss &WEEKDAY Three-letter day of the week, in English &MYNAMES The substitution you will use most of the time when you need to refer to LISTSERV. For BITNET-only or Internet-only servers, this will display LISTSERV's only e-mail address. For servers with both BITNET and Internet connectivity, it will say "[log in to unmask] (or LISTSERV@hostname)". &MYSELF LISTSERV's address, in the form [log in to unmask] or, if no Internet hostname is available, [log in to unmask] &MYNODE LISTSERV's BITNET nodeid, without the '.BITNET', or its Internet hostname if no NJE address is available. &MYHOST LISTSERV's Internet hostname or, if none is available, its NJE address (with '.BITNET'). &MBX(addr) Looks up the specified address in LISTSERV's signup file and displays "name <addr>" if a name is available, or just the original address otherwise. This is typically used to give the name of the command originator or target, along with his e-mail address: &MBX(&WHOM) or &MBX(&INVOKER). The following substitutions are also available for templates related to mailing lists: &LISTNAME Short (CMS) name of the list. &TITLE Title of the list, or empty string. &KWD(kwd) Value of the specified keyword for the list. You do not need to specify the name of the list - it is implicit. You need not put quotes around the keyword names either, although quotes will be accepted if present. Optionally, you can specify a second numeric argument to extract just one of the terms of a list header keyword; for instance, if the list header contains "Notebook= Yes,L1,Monthly, Private", &KWD(NOTEBOOK,4) has the value "Private". A third argument, also optional, specifies the default value for the keyword in case it was not initialized. It is meant to be used for conditional formatting in the default templates and list owners should not worry about it. In addition, many templates have their own specific substitutions, meaningful only in their specific context. For instance, a message informing a user that he was removed from a mailing list may have an &INVOKER substitution for the address of the person who issued the DELETE command. This is not meaningful for a template informing a user that he must confirm his subscription to a list within 10 days, so it is not generally available. If you attempt to use a substitution which is not available, the template processor writes an error message to the mail message it is generating, but sends it anyway, in the hope that the recipient will be able to figure out the meaning of the message in spite of the error. If you need to include a sentence with an ampersand character, you will have to double it to bypass the substitution process, as in "Thelma && Louise". Any line starting with a period in column 1 is processed as a formatting command. Note that neither substitutions nor formatting commands are case sensitive. Here is a list of the formatting commands list owners may need to use: .* Comment: anything on this line is simply ignored. This is useful for recording changes to template files when there are multiple owners. Just add a comment line with the date and your initials every time you make a change, for the benefit of the other owners. .FO OFF Turns off formatting: one template line = one line in the final message. You can resume formatting with .FO ON. .CE text Centers the text you specify (just the text you typed on the same line as the .CE command). This can be useful to highlight the syntax of a command. .RE OWNERS Adds a 'Reply-To:' field pointing to the list owners in the header of the generated message. Use this command when you think users are likely to want to reply with a question. You can also use .RE POSTMASTER to direct replies to the LISTSERV administrator, if this is more appropriate. .CC OFF Removes all "cc:" message recipients, if any. You can also add message recipients by specifying a series of e-mail addresses after the .CC statement, as in .CC [log in to unmask] PC mail users should note that in this context "cc:" is a RFC822 term that stands for "carbon copy". RFC822 messages may have "cc:" recipients in addition to their "primary" recipients. There is no real technical difference between the two, the "cc:" indicator just denotes a message that is being sent for your information. Some administrative messages sent to list owners are copied to the user for their information, and vice-versa; this behaviour can be disabled by adding a .CC OFF statement to the template. .QQ Cancels the message. LISTSERV stops reading the template and does not send anything. This is useful if you want to completely remove a particular message; note however that this can be confusing with certain commands, as LISTSERV may say "Notification is being sent to the list owners" when in fact nothing will be sent because of the .QQ command in the template. A number of more advanced commands are available to list owners with more sophisticated needs and some programming experience. If you encounter one of these commands in a template, you will probably want to leave it alone. .IM name Imbeds (inserts) another template at this point in the message. This is used to avoid duplicating large pieces of text which are mostly identical, such as the templates for "you have been added to list X by Y" and "your subscription to list X has been accepted". .DD ddname Copies the contents of the specified DD into the message. This is meaningful only if a DD has been set up by LISTSERV for this purpose. As a rule of thumb, you should either leave these statements unchanged or remove them. .BB cond Begin conditional block. The boolean expression following the keyword is evaluated and, if false, all the text between the .BB and .EB delimiters is skipped. Conditional blocks nest to an arbitrary depth. The expression evaluator is recursive but not very sophisticated; the restriction you are most likely to encounter is that all sub-expressions have to be enclosed in parentheses if you are using boolean operators. That is, ".BB &X = 3" is valid but ".BB &X = 3 and &Y = 4" is not. String literals do not require quoting unless they contain blanks, but quotes are accepted if supplied. Comparison operators are = <> ^= IN and NOT IN (the last two look for a word in a blank-separated list of options, such as a keyword value). These operators are not case-sensitive; == and ^== are available when case must be respected. Boolean operators are AND and OR. .SE var text Defines or redefines a substitution variable. This is convenient for storing temporary (text) expression results which need to be used several times. Even standard variables such as &LISTNAME can be redefined - at your own risk. You must enclose the text expression in single quotes if you want leading or trailing blanks. .TY text Types one line of text on the LISTSERV console log. This can be useful to the LISTSERV maintainer for debugging, and also to record information in the console log. ********************************************************************* * Usability: "INFO listname" now returns owner-supplied information * ********************************************************************* The INFO command has been modified to return list owner supplied information when the parameter is the name of a mailing list. The INFO command retains its former role as a means to order online documentation (eg "INFO DATABASE"). When the parameter does not correspond to any known help file, LISTSERV looks for a mailing list by that name and, if there is such a list, mails a copy of the template with the form name "INFO" to the command originator. The default INFO template simply states that there is no specific list owner provided information and returns a copy of the list header. List owners can, at their discretion, modify this template to include appropriate information about the list. ****************************************************** * Usability: Global List Exchange (GLX) enhancements * ****************************************************** LISTSERV version 1.8a and LMail version 1.2a, when operating together, extend the "global list exchange" support (also known as GLX) to the listname-request pseudo-address, and to the new listname-server address. This means you can write to the owners of any LISTSERV list without having to know where it is located, with the usual reservations about confidential lists and lists which are visible only to local users. For instance, to send mail to the owners of the INFOVAX list, you could write to [log in to unmask] and the message will be passed on to the LISTSERV that actually runs the list, which will in turn forward it to the list owners. Note that owner-xxxx requests are not forwarded as this mailbox is normally used only for delivery errors. It is an anomaly to receive a delivery error report for a list you are not hosting. The new listname-server convention can be used to send commands to the LISTSERV hosting a list without having to know where the list is located. While most commands are properly forwarded when sent to the wrong server, certain commands and in particular GET requests for material maintained by the list owner cannot be forwarded as they only embody a naming convention. That is, there is no way for LISTSERV@XYZ to know that 'GET CBPD13 MINUTES' should be forwarded to the server that hosts the CBPD-WG list. On the other hand, if you mail your GET request to [log in to unmask] (or CBPD-WG-Server@LISTSERV for BITNET users), it can be forwarded to the server hosting the list, where it can be executed successfully. This is particularly useful when ordering list archive files (eg INFO-XYZ LOG9307). ****************************************************************** * Security/Usability: Changes to "Validate=" list header keyword * ****************************************************************** Release 1.8a introduces changes to the "Validate=" list header keyword allowing the use of the "OK" command confirmation mechanism for all list management commands, with the exception of the PUT command. Rather than have to register a LISTSERV password that is easily forgotten or rely on the list owner for verification and confirmation, users may now be given full control of their subscriptions while at the same time improving system security over the former "Validate= All commands" setting. The "OK" command confirmation mechanism was introduced in version 1.7f, where it was used to implement the "Subscription= Open,Confirm" address verification mechanism. When a user tries to subscribe to a mailing list with that setting, he is mailed a confirmation request with a randomly generated confirmation key, also known as "magic cookie". The user replies to the message, types "OK" in the message body, and the command is confirmed. If for any reason the user's address cannot be replied to, the confirmation request is never received (or the "OK" message never arrives) and the user is not added. In version 1.8a, this procedure is also used for authentication purposes. Since the confirmation codes are valid only for a single command, this provides better security than personal passwords, while simplifying book-keeping. As before, the security level of the mailing list is controlled through the "Validate=" keyword. The contents of this keyword, however, have changed from version 1.7f (the old values are still accepted for compatibility reasons, but generate a warning with an explanatory message when you update the list header). The following security settings are available: - "Validate= No" (formerly "Validate= Store only"): all commands except PUT are taken at face value with no validation. While users are not bothered with validation requests, the list is totally unprotected from attacks by hackers. For compatibility reasons, this is the default setting. - "Validate= Yes" (formerly "Validate= All commands"): "protected" commands, such as DELETE or ADD, require password validation. For list owner commands, both personal and list passwords are accepted. Some user commands may accept a personal password, while others will cause the request to be forwarded to the list owners for verification. - "Validate= Yes,Confirm" (new level): protected commands are validated using the "OK" mechanism by default, although passwords are also accepted where appropriate. This is a good compromise between list security and list owner convenience. - "Validate= Yes,Confirm,NoPW" (new level): protected commands are validated using the "OK" mechanism. Passwords are not accepted, as they are not as safe as "cookies". This is the recommended setting for secure lists. - "Validate= All,Confirm" and "Validate= All,Confirm,NoPW" (new levels): all commands causing a change in state, except the PUT command, are validated using the "OK" mechanism, with or without a password alternative. Commands such as QUERY do not require any validation. Requests coming from the local system via CP MSG or CP SMSG never require validation, as they cannot be forged. The PUT command must always be validated with the list password, because there is presently no mechanism to suspend execution of a request to which a file is attached. If the list password is used only for that purpose, the exposure is minimal as PUT operations are not part of everyday list management routine. Note that PUT requests require no validation when submitted via SENDFILE from the machine on which LISTSERV is running, as the operating system then guarantees the authenticity of the transaction. For lists operating with "Validate= Yes" (without the "Confirm" option), LISTSERV may still use the "OK" mechanism in certain cases if it is deemed appropriate. LISTSERV's rationale is that the "Validate=" keyword describes the desired behaviour for interaction with the list owner and people who can be expected to use the list on a regular basis. SIGNOFF requests and DELETE requests from registered node administrators on behalf of a user on their machine, for instance, may be validated using the "OK" mechanism even though that was not requested, because users and node administrators are not generally expected to have a password with which to validate such requests. ********************************************************** * Usability: Miscellaneous changes to DELETE and SIGNOFF * ********************************************************** A number of minor usability changes were implemented as the DELETE and SIGNOFF commands were converted to PASCAL: - For peered lists, DELETE no longer attempts to forward the command to the "best" peer. Nowadays, with most subscribers being non-BITNET users, the "best" peer is often the one closest to the INTERBIT node. Since there are dozens of INTERBIT sites, most of the time this "best" peer is just a randomly selected peer with an equal probability of hosting the subscriber. Instead of forwarding, LISTSERV now executes the command and, if the user could not be found on the list, suggests the use of the GLOBAL option, which continues to work as before. - Consequently, DELHERE and DELETE now do exactly the same thing. DELHERE will be kept for compatibility. - When a user tries to sign off a "Validate= Yes" list, or when a NAD tries to remove a user from such a list, LISTSERV now generates a cookie instead of informing the list owner. The 'DELETE2' mailform is no longer used. - Users will no longer receive the "list is locked" or "list requires password validation" messages when trying to leave a list unless they are actually subscribed. This is still done for DELETE as the list owner should be informed that the list is locked. - A user leaving a list is no longer sent a message with the list of all the people who are being notified. - QUIET SIGNOFF and QUIET DELETE from registered node administrators were disabled (the command is executed, but the QUIET option is ignored). This ensures that the list owner is always notified of such action. ****************************************************** * Usability: New "List-Address=" list header keyword * ****************************************************** Version 1.8a adds a new "List-Address=" list header keyword, and a corresponding LIST_ADDRESS configuration option (to be added to LOCAL SYSVARS) to establish a system default. This keyword determines how LISTSERV announces the list address in the header of messages delivered to the list: NJE vs Internet address, short vs long list name, etc. The default options (when neither "List-Address=" nor LIST_ADDRESS are defined) corresponds to the behaviour of version 1.7f, ie short list name and NJE address. The format of both keyword and configuration statement is simply: name_info@host_info The first token (name_info) can be either LISTNAME or LIST-ID. Do not attempt to specify the actual list name. Use LISTNAME if you want LISTSERV to use the "short" list name (always available), and LIST-ID if you would rather see the "long" list name ("List-ID=" keyword). If there is no "long" name, the short name is substituted. The second token (host_info) can be either NJE, FQDN, or the fully qualified domain name of your choice. That is, you may type the actual hostname that you want LISTSERV to use, which may be useful if the machine on which LISTSERV is running is known under several hostnames. If you only want to override one of the two parts of the list address, you do not need to specify the other. For instance, if you only want to change the hostname, you can enter "List-Address= XYZ.EDU" in the list header and let the left-hand part default from the value of the system default in LOCAL SYSVARS. Similarly, "List-Address= List-ID" takes the right-hand part from the system default. To avoid bad surprises, LISTSERV will also accept a comma in lieu of @-sign in the list header, or a blank in LOCAL SYSVARS. Here are a few examples: - "List-Address= FQDN" announces the list under the Internet address for the LISTSERV host, if one is available (for BITNET-only sites this setting has no effect). - "List-Address= List-ID@FQDN" uses the long list name and the Internet hostname. - "List-Address= [log in to unmask]" uses the short list name and the hostname XYZ.EDU. - "List-Address= [log in to unmask]" is not valid. Always specify LISTNAME or LIST-ID for the left-hand part. To announce all lists under the Internet hostname by default, add the following statement to LOCAL SYSVARS: LIST_ADDRESS = 'FQDN' ------------------------------------------------------------------------- LMAIL and L-SOFT are trademarks of L-Soft international. Unix is a registered trademark of UNIX Systems Laboratories, Inc. OV, OV/VM and System/370 are trademarks of International Business Machines Corporation. PROFS is a registered trademark of International Business Machines Corporation. -------------------------------------------------------------------------