Description of the changes for release 1.7f of LISTSERV ------------------------------------------------------- April 2nd, 1993 ************** * Highlights * ************** Version 1.7f introduces the following major enhancements: - Most of DISTRIBUTE and the sections of incoming mail processing which most depend on the size of the mailing list have been rewritten in PASCAL, improving performance by a factor of 3 to 40 for DISTRIBUTE jobs. A distribution to a test list with 8,500 recipients now takes a mere 2.5 seconds of CPU time on a 3090-180x (over 100 seconds with 1.7e). Even small machines can now host lists with thousands of recipients. Furthermore, with release 1.7f and LMail it is now more economical, in most cases, to be on the DISTRIBUTE backbone than to have the files duplicated upstream and delivered via RSCS. - Significant productivity and ease-of-use enhancements have been made to the mailing list functions: automatic digests and indexes, list topics, dual-headers option for plain-VMSmail and PC users, easier command submission for QuickMail and All-in-One users, easier retrieval of indexed list postings through the generic 'listname-search-request' mailbox, etc. - New functions have been added to facilitate list maintenance: automatic deletion of nonexistent accounts when the error report is in LMail format (also supported by MX V3.2 and PMDF V4.2), list header keywords parser to verify keyword syntax, subscription confirmation to prevent addresses that cannot be replied to from being added to the list, farewell message, subscription and posting filtering under list owner control, more robust duplicate postings rejection, etc. - A new driver has been written for the LISTS (list-of-lists) database, with performance improvements of more than an order of magnitude. Running the LISTS database under 1.7f requires less system resources than simply managing the LIST GLOBAL summary with 1.7e. Since the change is transparent to end users and since only a couple dozen sites run the LISTS database, the technical aspects will not be described in these release notes; refer to the LSTSRV-L archives for more information. - When used in conjunction with version 1.1e or higher of LMail, a 1.7f backbone server may implement a "Global List Exchange" allowing users to reach all public lists in the LISTSERV network as though they were all located on the same machine. See the LSTSRV-L archives for a full description. - The database functions have been made 20-30% faster. - Support for VM/ESA release 2 and CMS release 9 has been added. Given the stability of current service levels of CMS release 6, the "unsupported environment" warnings have been removed and current levels of CMS release 6 are now supported. **************** * Availability * **************** Release 1.7 is available free of charge to sites which qualify under one (or more) of the categories listed below: a. Members of CREN or one of the CREN Cooperating Networks, excluding: 1. Members of the European Academic and Research Network (EARN). 2. Members whose computing facilities are located in an EEC country. b. Members of EARN, CREN or a CREN Cooperating Network whose computing facilities are located in a nordic country (Denmark, Finland, Iceland, Norway, Sweden). c. Members of EARN whose computing facilities are not located in an EEC country and which have formally applied (in writing) for a license, and complied with the appropriate EARN directives. International organizations are not considered as belonging to the EEC, regardless of where their computing facilities are physically located. Release 1.6 licenses from EEC countries remain valid for release 1.6, but will not be carried over to release 1.7. ******************************* * Supported operating systems * ******************************* Release 1.7 introduces the following changes to the list of supported operating systems: - (1.7a) Withdrawal of support for CMS release 3. - (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA. - (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes. - (1.7b) Support for the Shared File System, with a few restrictions concerning the setup of the A and D-disks and the default filepool setting. See the corresponding section of the 1.7b release notes for more information. - (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes. - (1.7e) Support for the Shared File System for the D-disk. - (1.7f) Support for VM/ESA 1.2, CMS release 9 and CMS release 6. Support for other versions of CP and CMS is unchanged from 1.6e. ************************** * External Compatibility * ************************** Release 1.7f is generally compatible with 1.7e, from the perspective of the end-user (including list managers and maintainers), and with the exception of the changes listed below. Release 1.7f introduces major changes to the LISTSERV internals, making compatibility with locally developed extensions or local modifications problematic. These changes are briefly described in the next section, and only affect sites which made alterations or additions to the LISTSERV code. Changes which affect all LISTSERV sites are highlighted below; note that more details are available, when appropriate, from the longer descriptive section of these release notes. 1. For the reasons described on the NODMGT-L list, the hashing function used by the Internet<->BITNET lookup code has been inverted. Some sites whose ':internet.' tags were not globally unique may find that their Internet and BITNET addresses are no longer recognized as being the same and will have to correct their ':internet.' information. Refer to the NODMGT-L and LSTOWN-L archives for more information. 2. The DISTRIBUTE command no longer removes duplicate recipients. If you have applications which submit their own DISTRIBUTE jobs, you will have to ensure that they do not supply the same mailbox twice as the second and following occurrences will no longer be removed. 3. The daily GET quotas for sites using the default LSV$GETQ exit have been changed from 20 files and 256k to 50 files and 2M. Sites with good connectivity may want to consider further increases in these quotas (MAXGET and MAXGETK configuration variables). 4. Lists operating with "Safe= Yes" (which is the default when the local mailer is is Netdata-capable) have a BSMTP origin of [log in to unmask] VM systems running XMAILER set the spool fileid from the BSMTP origin, rather than from the RFC822 header, and mail from safe lists will thus appear as 'owner-xx MAIL' in RDRLIST. LMail sites are not affected. See the LSTOWN-L archives for a comprehensive description of this change. 5. The behaviour of the LIST command was changed with respect to mailing lists for which no dummy VM userid has been created. Previously LISTSERV assumed the list had just been created and was not yet operational, and displayed a warning to that effect. But with the advent of LMail it is no longer necessary to create a dummy account if the list is to carry only RFC822 mail from sites which have a RFC822-compliant mailer (this excludes BITNET sites whose mail software delivers directly via NJE and not through the registered BITNET mailers). Such lists are operational, but only for RFC822 mail, and this is what the LIST command now reports. Release 1.7f is to be installed directly on top of the base 1.7e code, and includes all the fixes available from LISTSERV@SEARN for release 1.7e (ie all the "17Ennnt FIX" files from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits", which always have to be installed manually. ************************** * Internal Compatibility * ************************** Major changes have been made to LISTSERV internals. The most important ones are briefly described below; refer to the LSTSRV-L archives for more information. 1. Release 1.7f introduces PERMVARS FILE, a book-keeping file similar in function to LASTING GLOBALV but without the 255-bytes restrictions. Ultimately, all the information kept in LASTING GLOBALV will be moved to this new PERMVARS FILE. As a rule of thumb, when the last REXX program requiring access to the information is converted to PASCAL, the code is changed to use the "PMV" routines rather than GLOBALV. As this happens, LASTING GLOBALV entries will become obsolete and they may or may not be migrated to PERMVARS FILE by the installation exit. With release 1.7f, the "held list" information is migrated and copied over to PERMVARS FILE as the new code installs itself. 2. The information formerly in CRC FILE and LUPDTIME FILE will now be kept in PERMVARS FILE. The old files are erased and their contents are not copied to PERMVARS FILE. 3. Digest checkpoint variables have been migrated, but not copied over, to PERMVARS FILE. Site which had installed development fix 17E-002D will see the next issue of each digest claim to be the "first ever". This will only happen once. 4. While LSVDIST2 EXEC is still present, it is no longer the command entry point for the DISTRIBUTE command. Any call to LSVDIST2 in local applications must be replaced with calls to LSVDS4('DISTRIBUTE') - see LSVXMAIL EXEC for an example. The following REXX files have been become obsolete with release 1.7f and are deleted during migration: LSVBLC LSVDFOPT LSVHELD LSVPRSUM LSVXAM LSV00D The following files have become obsolete with release 1.7f: PEERS DISTSUM CRC FILE DS1WNG FILE LUPDTIME FILE ************************** * Compatibility Warnings * ************************** This new section describes planned changes which may affect compatibility (both external and internal). The release in which the change is planned to be introduced is indicated, with the letter 'x' denoting an unknown release of the specified major version (not necessarily posterior to the last release explicitly mentioned). Note that conversion of more REXX files to PREXX is assumed to take place with each new release and is not indicated here. - (1.7g) The DELETE * (NETWIDE command will be removed, as it does not scale up and places an unacceptable burden on LISTSERV sites. Sites running LMail, MX V3.2 or PMDF V4.2 (or higher) need not submit netwide delete jobs, as the delivery errors generated by their mailers will be automatically processed by LISTSERV (starting from 1.7f). - (1.7x) The internal format of FILELIST/FILEID and attendant files such as AFDLIST FILE will change dramatically. Applications which alter them directly will no longer work. - (1.7x) The internal format of LIST and attendant files (CACHE, STATS, et al) will change dramatically. Applications which alter them directly will no longer work. - (1.7x) LSVFRD1, LSVFWR1 and RXLSVUHF will be eventually removed. You should avoid using them. ******************************** * Minor fixes and enhancements * ******************************** This section has been removed for this release, due to significant changes made to key functions such as incoming mail files processing, DISTRIBUTE, the list keyword processor, etc. About 10,000 lines of new PASCAL code have been written to replace existing internals. As a rule of thumb, and as explained in the 1.7c release notes, it is not possible to generate the "Minor fixes and enhancements" section for code which has been redesigned and rewritten from scratch in PASCAL. REMINDER: Manual installation is not supported for release 1.7, and the code that implemented it has been removed with version 1.7e. Automatic installation with local password validation must be used instead. Sites which insist on installing manually should note that shipments contain an installation pre- and post-processing exit. If that exit is not executed, or if it is executed in a different order or environment than during automatic installation, serious problems may occur. ****************************************************** * Miscellaneous usability/human factors enhancements * ****************************************************** - The acknowledgements LISTSERV sends to confirm that a message has been posted now include the date and subject line. - LISTSERV will now accept up to 10 invalid commands in a mail message before command processing stops. This makes it easier for users of systems such as QuickMail, All-in-one, PROFS, OV/VM or any other mail system which inserts a letterhead-like header in the message body before the text supplied by the user to send commands to LISTSERV. Previously it was necessary for the user to either remove the letterhead lines manually before sending the message, or instruct LISTSERV to ignore them by typing '// job' before the first command. While this still works of course, the users can now send the commands directly and simply ignore the error messages generated by LISTSERV as it attempts to process the letterhead. Note that these letterheads are usually in the local language and nearly impossible to identify. - Unsolicited mail notices (formerly with "Subject: Message") now include the beginning of the first message line in the subject field. This is mostly useful for LISTSERV maintainers, who may receive a lot of these messages in case of system problem - repeated occurrences of one or more unexpected problems. - Subscription renewal confirmation requests no longer have a Reply-To: pointing to the list owner. This makes it possible for users to reply to the message and type the CONFIRM command. - Welcome messages are now passed back to the list owner if they cannot be delivered because the subscriber's address was invalid. - A "farewell message" can be defined for a mailing list, using the same procedure as for the welcome message but with a filetype of FAREWELL. The farewell message is sent to any user who signs off the list by himself, but not to users who are removed by the list owner, as it is expected to contain a message to users who chose to leave on their own, rather than to misbehaving users. The first line of this farewell message must be a 'Subject:' field. - Double quotes surrounding a user name (usually entered by mistake) are now internally removed, to avoid addresses of the type: From: "\"John Smith\"" <[log in to unmask]> which are perfectly correct but which many mail systems cannot handle. - The 'all-request' mailbox has been provided as a convenient means for the LISTSERV maintainer to send important warnings (unexpected downtime, changes in server configuration, etc) to all list owners. - The "Sender=" list header keyword has been extended to allow the specification of a second mailbox, to be used for IETF headers. The first value is used, as before, for non-IETF headers and is expected to contain the name and address of the list, or the keywords LIST or NONE. The second mailbox is used for IETF headers; if it is omitted, the generic 'owner-listname' mailbox is substituted. Example: Sender= "Test list <[log in to unmask]>","[log in to unmask]" - When delivering messages to a mailing list, LISTSERV scans the mail header for the name of the poster. But some user interfaces do not provide any personal name - just an e-mail address, and often one that does not have much in common with the person's real name. In such cases LISTSERV will now check if the poster is subscribed to the list and, if so, it will use the name registered in the subscription. ************************************** * Usability: Change to short headers * ************************************** The 'Organization:' field is now included in short headers (SHORTHDR and SHORTBSMTP options), as it provides useful information that is generally desired by the human reader, and does not require any special user interface to exploit. The criteria for adding fields to the short headers are that they should be useful to non-technical subscribers with unsophisticated user interfaces, such as plain VMSmail, PEEK, and so on. By definition, fields which are useful only with a specialized user interface, such as MIME fields, do not belong in the short header. Full headers (FULLHDR and FULLBSMTP) contain all fields, including MIME fields, and the list owner selects the default header type he finds most appropriate for his user population. This has been the case from day one. Every month, someone or other states the opposite on a public forum, usually to try to convince people not to choose LISTSERV to run their mailing lists. You can help preventing this misinformation from being spread by posting a copy of this paragraph in answer to any statement that LISTSERV and MIME are not compatible. Thank you. ********************************* * Usability: New DUALHDR option * ********************************* Quite a lot of non-technical users are relying on non-RFC822 user interfaces for reading their mail. Quite often these user interfaces are user-friendly, quality implementations of a proprietary mail protocol which the users are proficient with, but which happens not to lend itself to bidirectional mapping to RFC822. The users may have a good reason for using this particular program, and they complain that it is not always clear what list the postings come from, or who posted them. Other users have very primitive mail programs which do not preserve the original RFC822 header and may not even have a "message subject" concept. The user knows which list the message came from, but not who posted it, making private replies impossible. To solve this problem, a new type of headers has been added: DUALHDR (minimum abbreviation: DUAL). Dual headers are regular short (SHORTBSMTP) headers followed by a second header inside the message body. This second header shows what list the message is coming from ('Sender:'), the name and address of the person who posted it ('Poster:'), the poster's organization, if present, and the message subject. The date is not shown because even the most primitive mail programs appear to supply a usable message date. ************************************** * Usability: New list keyword parser * ************************************** The list keyword parser has been enhanced to perform basic syntax checking, to be more tolerant in its handling of commas and blanks, and to provide a permanent solution to the problem of lowercase tokens in keywords whose value is normally translated to upper case. Whenever a list is updated via the PUT command, the syntax of the various keywords is checked and a report is mailed back to the command originator if errors were found. Serious errors prevent the list from being stored, while the list is updated if only warnings are issued. The keywords are checked individually for proper syntax and consistency within the keyword itself (for instance, "Digest= No,A1" would be rejected, even though both sub-values are syntactically valid). For now, "global" consistency (across the various keywords) is not checked; this might be added in a future version if the current level of checking turns out to be insufficient. Note also that a few keywords are not checked at all, either because it is not possible or because the programming effort it would require is not worth the additional safety; for instance, it would be possible to check that the "Newsgroup=" keyword respects the xx.yy.zz syntax, but without checking that the newsgroup actually exists this would not be very useful. Here is a sample error report: ------------------------------------------------------------------------- The following problems have been detected in the list header: > Default-Options= XYZ,... Error: Invalid delivery option - "XYZ". > NATEBOOK= NO Warning: There is no keyword by that name - check the spelling. > Filter= YES,... Error: Invalid parameter value. Choose one from the following (without the quotes): "Also", "Only" or "Safe". > Digest= ...,X1/401,... Error: Invalid directory specification - incorrect syntax or directory does not exist. > Sizelim= XYZ Error: Value must be numeric. PUT operation rejected, old list remains unchanged. ------------------------------------------------------------------------- The new parser ignores blanks after or before commas, removing one of the most frequent causes for mistakes ("Reply-To= List, Respect" becomes valid with 1.7f). Furthermore, any text enclosed in double quotation marks stays in mixed case. Unlike version 1.7e, which supported this mechanism only if the first character of the keyword was a double-quote, with 1.7f you can use this anywhere in the keyword text. For instance, * Owner= ERIC@SEARN,"eric"@sunic.sunet.se defines ERIC@SEARN and [log in to unmask] as list owners. *********************************************** * Usability/Productivity: Digests and indexes * *********************************************** Release 1.7f introduces an automatic digestification function allowing subscribers who do not have the time to read large numbers of messages as they arrive to subscribe to a digestified or indexed version of the list. The list owner decides whether digests are available or not, the frequency at which they are issued and the day of week or time of day when the digest should be distributed. The digests conform to RFC1153 with an acceptable deviation from the recommended subject line (verified with the RFC author). Digests are larger messages containing all the postings made by list subscribers over a certain period of time. Unlike real-world digests, LISTSERV digests are not edited; what you see is exactly what was posted to the list. The only difference is that you get all the messages for a given day, week or month in a single batch. This is mostly useful if you are just "listening in" to the list and prefer to read the postings at your leisure. Digests are kept separately from list archives and can be made available for mailing lists which do not archive postings (ie which run with "Notebook= No"). Indexes, on the other hand, only provide a few lines of information for each posting: date and time, number of lines, name and address of poster, subject. The actual text is not included. You select just the messages you are interested in, and order them from the server. This is useful for mailing lists where most messages really don't interest you at all, or as an alternative to SET NOMAIL: when you come back from vacations, you can quickly order the messages you are most interested in. Note that, since indexes are not useful without the ability to order a copy of the messages you do want to read, they are not made available unless the list is archived and digests are enabled. Users sign up for digestified rather than immediate delivery with 'SET listname DIGests', while indexes are selected with 'SET listname INDex'. These two new options are alternatives to MAIL and NOMAIL. When switching around between these delivery options, users will observe the following behaviour (digests will be assumed to be daily for the sake of clarity): - When switching to NOMAIL: delivery stops immediately. The day's digest is not sent, as the user is assumed to request immediate termination of traffic from the list. - When switching from any option to DIGESTS or INDEX: mail delivery stops immediately, and the first index or digest may contain some items the user has already seen (if switching from MAIL to DIGESTS/INDEX). This is because the digests and indexes are global to the list - they are the same for everyone, just like regular issues of newspapers. - When switching from DIGESTS or INDEX to MAIL, the current, unfinished digest or index is immediately mailed to the user. New messages are delivered normally, as they arrive. Thus, a "trick" to get a copy of the current digest is to switch to MAIL and then back to DIGESTS. You can send both commands in the same mail message to make sure they are executed together. The list owner controls the availability and frequency of digests through the new "Digest=" list header keyword, which defaults to "Digest= No" for lists without an archive and "Digest= Yes,Same,Daily" for archived lists. Again, it is not necessary for the list to be archived to keep a digest; LISTSERV just attempts to avoid having to store large amounts of digest data on its A-disk for lists which, lacking a "Notebook= Yes,xxx" keyword, do not specify any suitable filemode for the digest data. Conversely, having daily as the default frequency keeps the additional cost in disk space to a minimum. The syntax of the keyword is "Digest= Yes,where,frequency,when,max" when digests are enabled, or then "Digest= No". The second parameter is a disk or directory specification, just as with the "Notebook=" keyword, or "Same", which means that the digest must be stored on the same disk as the list archives. The third parameter is either "Daily" (the default), "Weekly" or "Monthly". The third parameter is optional and specifies when the digest is to be actually distributed. For daily digests, specify 'hh:ss' or just 'hh' in the usual 00-23 scale (24 is also accepted for midnight). For weekly digests, specify a weekday such as "Tuesday". For monthly digests, you may specify a number from 1 to 31 corresponding to the day of the month when the digest will be distributed, although this is not recommended. The purpose here is to make it possible for digests to be issued at mid-month rather than on the first of the month - if you code a number larger than 28, you may not get a digest every month. Finally, the last and optional parameter takes the form "Size(nnnn)" and specifies the maximum size the digest is allowed to reach before a "special issue" is cut. Bear in mind that most unix systems do not accept messages larger than 100 kilobytes, so values larger than 1500 should be avoided. The default is to have virtually no limit - 10,000 lines. The list owner must take special care when disabling digests for a list, as LISTSERV does not presently have any facility which would allow it to alter subscription options automatically on the basis of changes to the list header. Subscribers who had opted for digests would continue not to receive mail as it arrives, but will not get the digests either. The best way to solve this problem is to announce the change long enough in advance, so that people can switch back before digests are suspended. The reason nothing has been done to remove this limitation is that it is not expected to be a frequent condition. Daily digests take up very little disk space and there is no reason to disable them for a typical list. ***************************************** * Usability: Changes to the SET command * ***************************************** As the number of subscription options increases (especially with the addition of digests/indexes and topics, whose behaviour depends on the list configuration), so do the potential for confusion and the risk of having activated the wrong options by mistake. Another concern for "secure" lists ("Validate= All commands") is that the new digest and topics options are likely to cause subscribers to issue a lot more SET commands, which must all be confirmed by the list owner. To address both problems, the SET command has been modified to return a status report to the user after successful completion. This status report shows all the options which are currently active (in the same format as a 'QUERY listname' command), and is always sent by mail so that it can be conveniently filed for future reference. This may be annoying to power users who know exactly what they are doing, but the benefits to novice users outweigh this small inconvenience. Finally, since the user is always informed of the change, it is no longer necessary to require list owner confirmation for "Validate= All" lists, which otherwise could have become unmanageable. ************************************************************* * Usability: Miscellaneous changes to the SUBSCRIBE command * ************************************************************* When a subscription request is sent via mail and the user did not supply his name (that is, when LISTSERV is sent just 'SUBSCRIBE XYZ-L'), the name will be taken from the mail header. If the mail header does not include a personal name (as in 'From: [log in to unmask]), LISTSERV will search its signup file as before and possibly reject the subscription. But as long as your mail program supplies a personal name, you will be able to subscribe to mailing lists without having to retype your name every time. Novice VMS users often supply all-caps personal names because they did not know they had to put double quotes around their name when typing the SEND command; there is a similar problem with users on the LISTSERV host who use the MSG command to communicate with LISTSERV because they learnt it from a computer expert who demonstrated with a command where case doesn't matter, and did not realize that the user would not know to use TELL when case does matter. In both cases the warning LISTSERV used to issue was not very helpful, as it only pointed out the problem without explaining how to solve it. On the other hand, LISTSERV has no way to know why exactly the name arrived in upper case and a description of all the possible operating system specific problems would be inappropriate and confusing (many users don't even know what kind of system they are using). From now on, LISTSERV simply converts the name to mixed case, inserting capitals where appropriate, and completes the subscription normally. A similar change was made to the ADD command. ************************************************************ * Productivity: Optional subscription confirmation feature * ************************************************************ One problem plaguing some mailing lists is one-way or non-repliable addresses. Most of the time this is due to a small number of faulty gateways, which one can just ban from the list until their maintainers address the problem. But sometimes the very topic of the list is bound to attract a large number of people connected through such gateways, and the amount of domains to filter out becomes unmanageable. This is particularly problematic when the gateway keeps quiet for a few days, and suddenly returns hundreds of delivery errors with a promise to keep doing so every day for another 6 days. This problem can be avoided by probing the return address before accepting the subscription. If the address cannot be replied to, only one delivery error will be bounced to the list owner (perhaps for several days, but there will be a single undeliverable message). With a gateway that waits 3 days before sending its first delivery error, however, there can be hundreds of messages "in the pipe" if the subscription is accepted directly. This probing is activated by specifying "Subscription= Open,Confirm" in the list header. When a user attempts to subscribe to the list, he is mailed a confirmation request with a randomly generated "confirmation code". The procedure for confirming the subscription is simple - you just reply to the message, type "ok", and send. If the return address does not work, the request will never be confirmed and the user will not be subscribed. And since the user cannot guess the confirmation code he will be assigned, he cannot "cheat" by sending the confirmation along with his request. The subscription request also expires after a certain amount of time, as determined by the "Confirm-Delay=" keyword (the default is 48h). Many unreliable gateways have a turnaround time of several days, and this is another way to filter them: if the confirmation delay is low enough, they will never manage to subscribe and you will not have to put up with gateways which take a week to realize that the subscriber's account has expired and return a week worth's of delivery errors. On the other hand, if you do want to let these people in, you will have to increase the confirmation delay to a week or so (1 week = 168h). ************************************************************ * Productivity/Reliability: LMail support and "safe" lists * ************************************************************ Both LMail and current versions of XMAILER (R2.10 or higher) provide support for the administrative 'owner-listname' and 'listname-request' mailboxes introduced in release 1.7b. Previously this support was only available via a patch to XMAILER R2.08 which only a handful of sites had applied. Now that over 50% of LISTSERV sites (and 70% of backbone sites) are running LMail or XMAILER R2.10, LISTSERV is finally in a position to use the standard Internet conventions for error return addresses to decrease the risk of mailing loops. Note again that without one of the mailers mentioned above there is no way for LISTSERV to intercept delivery errors sent to the 'owner-listname' mailboxes and that these errors are likely to be lost, without anyone ever being notified. You must not activate the use of these mailboxes unless your mailer supports them. For sites which do run a mailer that supports the administrative owner-* mailboxes, a new list header keyword, "Safe= Yes/No", controls the e-mail address LISTSERV places in the BSMTP MAIL FROM: field, which is where well-behaved mailers will return delivery errors. With "Safe= No", these errors are sent to the list address as before, hopefully to be intercepted by the loop detector and passed on to the list owner. With "Safe= Yes", the error address is set to 'owner-listname', and delivery errors sent to that address are passed on to the list owner without the risk of creating a mailing loop. The default is "Safe= Yes" if you are running LMail or XMAILER R2.10 with NDMAIL=1 in LOCAL SYSVARS. Again, you must not attempt to use "Safe= Yes" if your mailer does not support the owner-* mailboxes. IMPORTANT: The use of "Safe= Yes" does not guarantee that all errors will go to the 'owner-listname' mailbox. Unfortunately, there are many non compliant mailers which will continue to send the error back to the list (usually because it is listed in the 'Reply-To:' or 'Sender:' field). The use of the "Safe= Yes" option significantly decreases the potential for mailing loops, but not enough to actually code "Loopcheck= No", unless you are sure that all your subscribers have compliant mailers. Another productivity change which has already been announced on various mailing lists is support for automatic deletion of users whose account has expired or whose system has permanently disconnected. When the delivery error is generated by LMail (any version), MX V3.2 or higher, or PMDF V4.2 or higher, which all implement the same delivery error format, LISTSERV may be able to automatically process the delivery error and take action based on the value of the "Auto-Delete=" list header keyword. If you have coded "Auto-Delete= No", or if the delivery error is not in LMail format and LISTSERV cannot understand it, LISTSERV simply passes it to the list owner as before. Otherwise LISTSERV processes the message automatically. The algorithm may be refined in a future version, but at present the following steps are taken: - LISTSERV looks for "permanent" errors (no such user, no such host, and so on). If the failing recipients are subscribed to the list, LISTSERV removes them and notifies you. No action is required from you. - If there are permanent errors for users LISTSERV could not find on the list (for instance because the account subscribed to the list is a totally different one which forwards mail to a dead account), or if there are only "temporary" errors (host unreachable for 3 days, system error, disk quota exceeded, and so on), LISTSERV further examines the "Auto-Delete=" keyword and passes the message to the list owner unless you specified "Auto-Delete= Yes,Full-Auto". - In a future version, LISTSERV might keep track of temporary errors when the list is running in full-auto mode and remove users after a certain amount of temporary errors. For now, it simply ignores them. - When running in full-auto mode, LISTSERV never passes back a delivery error unless it took action on it. This means that certain errors may remain unsolved, as LISTSERV presently ignores temporary errors and some of them are virtually permanent (if the owner of the account has left but for some reason his account was not closed, his disk quota is bound to remain exceeded until someone takes action). Full-auto mode is to be used only when you positively do not have the time to handle the delivery errors LISTSERV is sending you every day. The default value is "Auto-Delete= No" for lists with "Validate= All" and "Auto-Delete= Yes,Semi-Auto" for other lists. ************************************************* * Usability: Subscription and posting filtering * ************************************************* In order to give list owners better control over troublesome users or gateways, and to remove the generic ban on some traditionally reserved usernames such as 'root' or 'postmaster', a new filtering system has been added to supplement the SERVE OFF mechanism (which is restricted to the LISTSERV maintainer and affects all lists on the server). This is implemented via a new list header keyword, "Filter=", which is checked when a user attempts to post or subscribe to a list (but not when the list owner issues an ADD command). The first word of this keyword is either "Only", "Also" or "Safe", and it is followed by a list of patterns such as 'X400MAIL@*' or '*@*.XYZ.EDU' (without the quotes). If "Also" is specified, your filter is used in addition to the standard LISTSERV filter; this is useful to register additional looping mailers, to prevent users behind broken gateways from subscribing until the problem is addressed, or to ban anonymous posters. LISTSERV has two built-in filters: a "minimal" one, which is used for safe lists, and a "safe" one (similar to the one used by 1.7e) which is used for lists running with "Safe= No". That is, the unsafe lists need a safe filter to avoid mailing loops; safe lists only need the minimal filter, but can be made even safer by selecting "Filter= Safe". This, however, prevents usernames such as 'root' or 'postmaster' from posting to the list, because they are included in the safe filter. If "Filter= Only" is used, the addresses you specify are the only ones which LISTSERV prevents from posting to the list; you should not use this option unless you also code "Safe= Yes", and even then you will want to ask your LISTSERV maintainer for permission. In fact, this option has been added mostly for LISTSERV maintainers with very specific problems to solve. The minimal filter is very small and you should never need to override it. Messages sent to the LISTSERV userid for execution are always checked with the minimal filter, as people with userids such as 'root' would otherwise not be allowed to subscribe to lists which were set up to allow them. This may cause some extra traffic while LISTSERV and various gateways engage in ping-pong wars, but after 10 iterations the gateways will be served off and this will stop. Note that LISTSERV extracts as many e-mail addresses as it can from the userid being checked and runs them all through the filter. For instance if your list receives mail from [log in to unmask], LISTSERV will check [log in to unmask], [log in to unmask] and 'mailer@searn' (via the 'internet.' tag). Generally speaking, userids intercepted by the 1.7e filter will not pass the safe 1.7f filter. ************************************** * Usability: Support for list topics * ************************************** List topics provide a way to run a mailing list (preferrably moderated) where several sub-topics are being discussed in parallel but some subscribers are only interested in a subset of the topics. For instance, a working group might have general discussions, decisions, and messages related to meetings. People who cannot attend the meetings can then opt out of last calls for hotel reservations and discussions about seafood restaurants, whereas people who have no time to follow the discussions can elect to get just the decisions. At any rate, such a compartmented list requires a certain discipline in order to be successful, as the posters must label their messages to indicate which topic(s) they belong to. Through the "Topics=" keyword, the list owner can define up to 11 topics for the list. For instance, the list owner could code: Topics= News,Benchmarks,Meetings,Beta-tests ******************************************************** * WARNING - YOU MUST NEVER REORDER THE TOPICS= KEYWORD * ******************************************************** To save disk space, LISTSERV remembers which topics users have selected through their ordering in the "Topics=" keyword. That is, "News" is "topic number 1" for LISTSERV, "Benchmarks" is "topic number 2", and so on. This means you can change the name of a topic without requiring users to alter their subscriptions (for instance, you could decide that "Tests" is a better name than "Beta-tests" and just make the change). However, you must never change the order of the topics in the "Topics=" keyword. If you want to remove a topic, replace it with a comma. For instance, to remove the "Meetings" topic, you would change the keyword to: Topics= News,Benchmarks,,Beta-tests This restriction might be removed in a future release. Topic names can contain any character except space, colon and comma; the use of double quotes or equal signs is discouraged, as they require special attention when coding list header keywords. In addition, topic names may not start with a plus or minus sign, and the words ALL, NONE, RE, OTHER and OTHERS are reserved. Posters label their messages through the subject field. LISTSERV first skips any possible sequence of 'Re:' keywords, and takes anything to the left of a colon as a list of topics, separated by commas. The posting is considered to belong to all the topics listed before the colon. If none of these topics is valid for the list, it is classified in a special, 12th topic, "Other". If some of the topics are valid but others are undefined, the invalid ones are ignored. At any rate the subject field is left unchanged. Here is an example: Subject: Benchmarks,News: Benchmarks for XYZ now available! Messages which should be read by everyone can be posted to the special topic "All". Topic names can be shortened to any unambiguous abbreviation. In our example, "Be" is ambiguous because it could be either "Beta-tests" or "Benchmarks", but "Bench" is acceptable. Subscribers select the topics they wish to receive with the SET command. The syntax is 'SET listname TOPICS: xxx' where 'xxx' can be: - A list of all the topics the user wishes to receive. In that case these topics replace any other topics the user may have subscribed to before. For instance, after 'SET XYZ-L TOPICS: NEWS BENCH', the user will receive news and benchmarks, and nothing else. - Updates to the list of topics the user currently receives. A plus sign indicates a topic that should be added, a minus sign requests the removal of a topic. For instance, 'SET XYZ-L TOPICS: +NEWS -BENCH' adds news and removes benchmarks. If a topic name is given without a + or - sign, + is assumed: 'SET XYZ-L TOPICS: +NEWS BENCH' adds news and benchmarks. The first topic name must have the plus sign to show that this is an addition, and not a replacement. - A combination of the above, mostly useful to enable all but a few topics: 'SET XYZ-L TOPICS: ALL -MEETINGS'. The colon after the keyword TOPICS: is optional, and TOPICS= is also accepted. Do not forget to include the special OTHER topic if you want to receive general discussions which were not labelled properly. On the other hand, if you only want to receive properly labelled messages you should not include it. ALL does include OTHER. A "Default-Topics=" list header keyword is available to define the initial topics for new subscribers. The syntax is the same as for the SET command, except that topic names are separated by commas in the usual fashion and that the first topic may not start with a + or - sign (there is nothing to add to, as this is a new subscription). This is similar to "Default-Options=" in that it does not affect existing subscribers. Users who signed up before topics were enabled on the list are automatically subscribed to all topics. Finally, it is important to note that topics are active only when your subscription is set to MAIL. Digests are indexes always contain all the postings that were made, because the same digest is prepared and sent to all the subscribers. Depending on the success of topics support, this restriction might be lifted in a future release. ************************************ * Serviceability: New LSV$DNT exit * ************************************ A new exit, LSV$DNT EXEC, has been added to make customization of DOMAIN NAMES-based routing more convenient. This exit is similar to LMail's LSM$DNT, but note that the format of the tables is different: you cannot use your LSM$DNT EXEC directly for LISTSERV. After rebuilding its DOMAIN NAMESUM file, LISTSERV looks for LSV$DNT EXEC on any accessed disk. If it finds it, it invokes it and the exit can then modify the DOMAIN NAMESUM file as it sees fit. With release 1.7f the various entries in the file no longer need to be sorted from longest to shortest domain name. For the maintainer's convenience, the code that sorts this file has not been removed - it makes it easier to see how mail for a given domain is going to be routed, using ALL/.XX (just the top level domain) under XEDIT. The LSV$DNT exit may insert entries out of sort order, but in that case that is what the final file will look like. LISTSERV does not sort the file after calling LSV$DNT because the purpose of this exit is to ultimately control the final contents of the file. ***************************************** * Reliability: PASCAL region monitoring * ***************************************** Servers with a large workload and/or many large mailing lists may report unexpected "Machine storage exhausted" REXX errors even though the virtual storage size is 12M or more. This problem, which is difficult to avoid, is due to the fact that LISTSERV is being rewritten in PASCAL (and, indeed, the "core" functions of LISTSERV are now mostly in that language) while there are still a number of storage-hungry commands written in REXX. The PASCAL compiler, which like most IBM compilers was designed for MVS and made to work on VM using OS simulation, works on the assumption that it is the only application running in the virtual machine. While it will not grab storage it has no use for, it never releases storage it no longer needs; instead, it keeps it around for the next time a PASCAL routine requests some storage. This unused storage is not available to REXX and may cause "Machine storage exhausted" errors (or error 1001 from LSVFILER). Rebooting usually solves the problem. Release 1.7f introduces a "PASCAL region watchdog" which monitors the amount of storage the PASCAL run-time environment has allocated (use the SHOW STORAGE command to check out on that value). During end-of-job cleanup, LISTSERV resets the level-2 PASCAL run-time environment if this value is unreasonably high. This RTE corresponds to the PASCAL subroutines called by REXX command processors and by the REXX code that processes new postings to mailing lists (in an all-PASCAL implementation, this RTE would not be present). In most cases this is sufficient to recover locked storage. However, storage held by the level-1 RTE (the one that corresponds to LISTSERV's main program) cannot be reclaimed without rebooting the server. SHOW STORAGE has no way to tell the level-1 and level-2 allocations apart and can only report their sum.