1.6e is being released today; this new version introduces some important changes to a number of commands for the support of the ':internet' tag in BITEARN NODES, and I advise you to read the summary of changes more carefully than for a "normal" release. I would like to take opportunity on this posting to thank the beta-test sites for their continued support and patience, without which the installation of new LISTSERV releases would be much more painful - for everybody. Eric Description of the changes from release 1.6d to 1.6e of LISTSERV -------------------------------------------------------------- July 3rd, 1990 **************** * Requirements * **************** LISTSERV release 1.6e is supported on any hardware/operating system configuration supporting release 1.6d. For political reasons, release 1.6 of LISTSERV is available only to BITNET and NetNorth sites, and to selected EARN sites which have explicitly requested to obtain it (contact ERIC@SEARN for more information). Other EARN sites now receive support for the EARN version of LISTSERV from Turgut Kalfaoglu <TURGUT@TREARN>, to whom EARN users should address any and all inquiries regarding LISTSERV. ***************** * Compatibility * ***************** Release 1.6e is compatible with 1.6d, with the exception of the usual minor changes in messages et al, and of the changes made for the support of the ':internet' tag and host-aliasing, as outlined below. Release 1.6e is to be installed directly on top of the base 1.6d version, and includes all the fixes available from LISTSERV@SEARN for release 1.6d (ie "16Dnnnt FIX" from filelist "LFIXES"), with the exception of those containing replacement "recommended user exits" which always have to be installed manually. ******************************** * Minor fixes and enhancements * ******************************** This section contains a brief description of all the "small yet not unimportant worries" that have been fixed in release 1.6e, along with the affected releases, if known ("All" meaning "any release in which the facility is available", ie the problem was present since the very beginning). Minor enhancements which do not warrant a detailed description are also included; they are all located at the bottom of the list, and can be identified by the word "New" in the "Affected releases" column. Finally, changes which might introduce compatibility problems are flagged with one asterisk for minor problems (change in a message or in the output of some commands originally designed for human readers rather than programs, but which some people might still have decided to feed to programs), or two asterisks for more serious problems, for which a detailed description will always be provided further on. Affected releases Description of the bug, problem or enhancement -------- ---------------------------------------------- All LISTSERV-Punch format incorrectly forced for some AFD'ed files when Punch format was requested. Fixed. All 'CP SPOOL D PURGE' instructions would sometimes display junk messages on the console log under VM/XA. Changed to "Diag 8" to suppress message. All Incorrect error message returned when user attempts to search (LOC)UDD database on server which is not a registered UDD node. Fixed. All Abend in LSVFILER when processing sparse files. LISTSERV does not use or create such files, but might be presented with some specimen created by other software. Sparse (pre-CMS6) files are now supported by LSVFILER. All Domain-form synonyms of the local node (ie nodes in MYDOMAIN variable) not treated as local nodes for mail distribution. Fixed. All Specification of zero-length formats fields for database LIST and INDEX commands was not allowed, contrary to what the documentation stated. Use of zero-length has been allowed to create null columns in the output. All Problem causing occasional return code 2012 from LSVCARDR (with file transferred to postmaster) identified and solved. Note that in pre-1.6a versions the symptoms were addressing exception, high-core pointers corruption or infinite loop. All LSVLUPD would wait "forever" for a refresh job to return updated list information. Logic changed to resend a new refresh job after approximately 1 month. All LSVFILER sometimes caused fatal (DIE=YES) DMS1072T error when run under CMS 5.5 *and* VM/XA (CMS6 should therefore not be affected), due to a bug in CMS. Since IBM is unlikely to fix it within a reasonable time frame, if at all, code has been added to LSVFILER to bypass the problem. All List owners were unable to update their list subscription options with the 'SET' command for 'Validate= All commands' lists. Fixed. All WAKEPARM FILE events of the form '==/nn/==' were incorrectly ignored. Fixed. All Command from 'MAINT@hostname' (where hostname is either a domain-style address or a NJE nodeid without registered NAD) incorrectly rejected with instructions to resend from the same address (MAINT@hostname). Fixed. All 'FOR' command with a domain-style target address generates a jobname of '' (empty string). Changed to use the userid of the command originator (ie the person issuing the 'FOR' command). All Various problems with STOP command sent via mail or job file now fixed. 1.6c REXX error in LSVDIST2 creating job error report with certain types of files. Fixed. * New To solve further LISTSERV/MAILER interaction problems, the 'Sender:' and 'Reply-To:' fields on list mail, as well as the 'From:' field on mail coming from LISTSERV itself (eg output from a command), have been changed to append '.BITNET' to unqualified addresses. This was supposed to have been done in 1.6d, but the fix was apparently not released. * New "End of distribution" message on console log modified to show distribution statistics - number of inter-LISTSERV jobs generated and associated number of recipients, number of other outbound files and corresponding amount of recipients in "service area". * New Default value of "Ack=" list header keyword changed to NO for file distribution (was previously YES) for consistency with the new mail distribution default (NO) introduced in release 1.6b. QUERY command changed to display the proper default value (NO instead of YES). New Generation of the BITEARN database index now done "on the fly" during the NODESGEN process. New BITEARN VERSION file obsoleted - decision to reject or accept GET BITEARN NODES commands now based solely on info from the file header. New "Fast SIGNOFF" logic should significantly improve performance of 'SIGNOFF/DELETE * (NETWIDE' commands (in cases where the addresses are not found on any list). New Country name to ISO country code translator for PEERS and BITEARN databases improves to accept abbreviations and "fuzzy" spelling. * New PEERS database altered to remove obsolete DIST1 specification and change index format. New New "built-in" file format 'MAIL' added to force transmission via RFC822 mail whenever desired. New ':techrep.' and ':inforep.' addresses both recognized as registered Node ADministrators (NAD's). * New 'Resent-Message-ID:' tag now discarded on moderated lists when editor forwards a message to the list (in the same way as 'Resent-From:' is discarded in favour of 'From:'). New Error messages from 'FOR' command enhanced to explain the wildcard restriction. New Dummy 'PUT' command added to provide some information to the user in case 'TELL LISTSERV PUT FOO BAR' is issued. * New New "dumb user" detector attempts to filter commands sent to the list rather than the LISTSERV userid. See description. New Support for the message suppression functions of the VM30091 SPE. See description. **New Support for ':internet' tag and host-name aliasing. See description. * New Generalized wildcard support for commands and some list header keywords - see description. New Usage notes regarding the wildcard support on the DELETE command - see description. New Performance improvement for LSVNAD (a function used by each netwide DELETE job) - 2.5 times less CPU, 5 times less I/O and 4 times lower elapsed time on the average. New Performance improvement for access to R/O files (BITEARN NODES, etc) - unnecessary ACCESS commands are now avoided. New 'File rerouted' and second line of 'QUERY linkid' output from RSCS V2.3 now filtered from console log; the former is counted as a 'Sent file' message, the latter as a 'Network status' message. ******************************************* * Usability: Generalized wildcard support * ******************************************* With release 1.6e, an attempt has been made at providing a "better quality" of wildcard support for most commands and relevant list header keywords, while preserving compatibility with previous versions. All commands and list header keywords which previously supported limited wildcard specifications (one asterisk for most functions, two for the 'Service=' keyword) now support any number of asterisks in the pattern. The AFD/FUI, CONFIRM, DELETE/DELHERE, PW, QUERY and SET commands now support full wildcard specification (both in the userid and nodeid part) for target addresses. In addition, AFD DEL and DELETE/DELHERE support a new option, 'TEST', which allows you to check out wildcard patterns without actually performing any deletion; for instance, 'DEL XYZ-L *@QZ*OM (TEST' will display the list of users who would have been deleted if you had not specified the TEST option, and any possible error message related to your authorization to execute the command (or lack thereof). The SET and CONFIRM commands now support a 'listname' specification of '*', meaning "all the lists on the local server I am subscribed to". Support for wildcards of the type 'FOO*' has not been provided, under the assumption that such a syntax would have little or no interest given the present distribution of list names, and that it is more likely to be a typing error. 'REVIEW *' is purposefully not supported. The 'Local=' keyword now supports wildcard specifications, making it possible to enter definitions of the form: 'Local= *.XYZ.EDU'. Unfortunately, for performance considerations such definitions will be considered as non-local by the mailing list exploder, ie it will attempt to batch up to 5 recipients per message (unless of course BSMTP headers are used). Finally, because of this change it has been necessary to disable execution of commands from users whose RFC822 address contains an asterisk; such users could be unregistered only by means of a wildcard, which in turn would be likely to delete other "innocent" users. The only case of asterisks in userid's that has been observed so far had the asterisk in column 1, and this type of addresses were already not supported (lines starting by an asterisk being treated like a comment). ************************************************************************ * Usability: Usage notes regarding wildcard support and netwide DELETE * ************************************************************************ Node administrators should take note of the fact that the DELETE command now supports wildcard specifications in the target addresses, and should read the following IMPORTANT NOTES carefully: - As a node administrator and as a person issuing a "long" job to a large number of servers operated by other institutions, you are responsible for taking reasonable steps to allow these institutions to spend as little computing power as possible on these jobs which, from the point of view of local management, are viewed as "pure overhead". This short description has been written in order to assist you in making the "right" decisions concerning these jobs; please read it carefully. - By its very nature, the "Fast SIGNOFF" logic is likely to result in very significant CPU and I/O savings when the number of users it has to look for is fairly small, and to have absolutely no effect if ten thousands of users are the target of a single job. There is no clear rule to decide what the cut-off point is, however it would seem that, in most cases, it can handle a few hundred users without problem. You should therefore try to keep your requests to a "few hundred users" (not more than 500 in any case) per job; submitting larger requests also introduces the risk that some servers may run out of storage when executing it, if they have very large lists. - Wildcard support is available ONLY from servers running release 1.6e, or a more recent release. You should therefore not attempt to use it until the majority of the network has converted to release 1.6e; if you did, only a small fraction of your users' subscriptions would be deleted, although the same machine and network resources as for a normal job would been wasted on your request. - If your userid naming scheme allows it, you might be able to take advantage of the function by sending commands like 'DELETE * CS*@FOO (NETWIDE'. Please observe the following "common sense" rules when deciding whether or not to use a wildcard: a. The specification of wildcards disables the "Fast SIGNOFF" logic and may significantly affect performance, as compared to what can be achieved with a list of "constant" userid's. You should therefore NOT use wildcards to dispose of 3 userids. b. If on the other hand you have 2000 users to unregister which all match a particular pattern that no other user on your system matches, you should DEFINITELY attempt to use a wildcard pattern. This will significantly reduce both network load and system resource usage at the server site, as 2000 iterations of the "Fast SIGNOFF" logic are much slower than one iteration of the "Wildcard SIGNOFF" code. c. The number of users after which it is better to use a wildcard from the point of view of performance is very hard to define. It depends on the number of lists on the server, the size of these lists, and the distribution of users among the lists. In most cases, you will have either a very small or very large amount of users to process, and the decision will be an easy one. - Please note that the specification of ONE target with a wildcard disables the "Fast SIGNOFF" logic for ALL targets, therefore: a. If you have decided to use wildcards for one of the types of account names, you might as well save network bandwidth and do the same for the other types (except, of course, if you have only one user matching the pattern). b. If you have a wildcard pattern plus 300 users which do not match any pattern (for instance because the userid is the initials of their owners), make two separate jobs. - If you have to delete identical users from several nodes (for instance, a JNET cluster - FOOBAR, FOOVAX1, FOOVAX2, FOOVAX3), do NOT request the deletion of 'CS*@*' under the assumption that you are authorized only for the nodes in question and 'CS*' users at other nodes will not be affected. If the userid-pattern ('CS*' in our example) is common, it will create significant unnecessary processing overhead at the server sites AND return lots of error messages to your mailbox, both being undesirable. 'CS*@FOOVAX*' is the best way to take care of nodes FOOVAX1 to FOOVAX3, and you should add 'CS*@FOOBAR' for the cluster node. This is all assuming that your cluster aliases are not registered; if they are, 'CS*@FOOBAR' will suffice. ***************************************************************** * Usability: Support for ':internet' tag and host-name aliasing * ***************************************************************** Starting with release 1.6e, LISTSERV will support the ':internet' tag and any future tag related to host-name aliasing. The ':alias' tag is presently supported for that purpose as well, although it will eventually become obsolete with the "new" BITEARN NODES format. This is a very important improvement for the end-user. "Host-name aliasing" means that LISTSERV will now be able to identify electronic mail addresses which all point to the same mailbox, but are written in a different "style". For instance, [log in to unmask], 'ERIC@SEARN' and [log in to unmask] are all different "names" for the same mailbox (but they are NOT the same thing as 'ERIC@KTHVM' or [log in to unmask]). Unfortunately, chances are that, when you send commands to the same LISTSERV machine from the same computer account, a different address will be generated every time depending on the command you have used and, for VAX clusters, on other factors (CPU load, etc) which will probably look quite "random" to you. You can thus end up being subscribed under one "name", and be told 1h later that you are not subscribed to the very same list because the command you used to attempt to leave it came from a different "name". Unfortunately, you will still get mail from that list, and chances are that your only recourse will be to write to the list owner and ask him to delete you. Since he may not know that VM.UNI-C.DK is the same thing as NEUVM1, this, too, might take a while to happen. Fortunately, LISTSERV is now able to identify these "synonym" names by itself, PROVIDED that the node administrator has supplied the required information (':internet' tag) in his node entry; otherwise there is no way for LISTSERV to "guess" possible synonyms. The basic "rule" (which will be explained in greater detail below) is that mail from distribution lists will be sent to the address your subscription request came from; however, you will be able to "act" on these subscriptions from any "synonym" address. NOTE: If you are a node administrator yourself and need more information about the ':internet' tag or how to update your node entry, please contact the BITNIC or your NCC, *not* LISTSERV Coordination! Before reading the description of the exact changes which have been implemented in release 1.6e, it is necessary to have a clear understanding of what the *intent* of the changes is. In the following description, the term "registered" means "subscribed to a distribution list", "registered in the user-passwords file", "subscribed to a LISTSERV file via the AFD or FUI commands", "registered in the SIGNUP file", etc. 1. A user may be registered under different, equivalent addresses. Such a condition, subsequently referred to as "multiple registration", can happen because: a. LISTSERV does not know that the addresses are equivalent, because the necessary information has not been supplied by the node administrator. b. The condition existed before release 1.6e was installed. c. The condition has been purposefully set up because it is explicitly desired (for instance for an "alert" list which much reach people as quickly and reliably as possible, and different addresses routed through different physical lines is a good way to achieve that). 2. With the exception of case 1a, the user must be able to terminate a possible multiple registration by himself, using "normal" LISTSERV commands; the end-user must be prevented from setting up such a condition by mistake. 3. A corollary of point 2 is that, in cases where the software allows multiple registration conditions to exist at all, it may be necessary for the user to request assistance from a list owner or LISTSERV maintainer if he desires to set up such a condition. Once the condition is set up, he should be able to manage it by himself, and to terminate it (as already stated in point 2) if he so desires. 4. When updating or deleting his registration, the end-user should not have to worry about the address he is sending his command from; that is, the command must be the same regardless of whether the address he is going to send from is an exact match of the address he is registered under, or only a synonym. 5. However, the power-user should have control over the address he is going to receive LISTSERV material from. That is, it is not acceptable for LISTSERV to automatically "translate" the address used by the user before performing the registration, UNLESS the address in question is not used to deliver material to the user. That is, it is not acceptable to translate registrations in mailing lists; the software is however free to translate password-file registrations, as the addresses therein are purely "internal" - they are only used to associate a password with a mailbox, not to deliver information. 6. What is true for end-users should also hold true for "semi-privileged" users - list owners, list moderators, node administrators, etc. For technical reasons, it may or may not be possible (or even desirable) to apply these rules to LISTSERV maintainers as well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Description of the implemented changes >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Scope of the support -------------------- - Host-name aliasing support is provided for end-users, list owners, list editors and node administrators. This support extends to all the functions normally used by such people, including file server functions, authority to mail to a list and list management commands. Locally written extensions which use the "standard" authority validation routines (LSVCKPRV, LSVNAD, LSVALLOW) will automatically benefit from this support. - Host-name aliasing is not provided for postmasters, because it could not be provided in all circumstances; it has been deemed more consistent never to allow synonym addresses for postmaster than to accept them only on an (apparently) "random" subset of commands. This ensures, in addition, that "emergency" commands from postmasters will still be accepted in case the host-name aliasing logic fails and starts, for instance, to translate all host-names to binary junk (since this logic is table-driven, the failure might be triggered by a new version of BITEARN NODES without any change in the software itself). SIGNUP file ----------- - Multiple registration is not permitted within the SIGNUP FILE, whose sole purpose is to associate names with mailboxes. The SIGNUP FILE is "compressed" during the NODESGEN process, as fresh information about host aliases is being acquired from the new BITEARN NODES file; whenever a duplicate is detected, one of the two registrations is discarded. It is not possible to predict which one will be kept, and, given the nature of this registration, there is no particular reason to favour a domain-style registration over a NJE one, or vice-versa. - The format of the SIGNUP FILE has been slightly altered; blank records should be ignored by local extensions which read the file directly, and left untouched (these records are deleted to save space every time LISTSERV is rebooted). Authors of local extensions should note that LISTSERV now leaves the SIGNUP FILE open under certain circumstances; if you try to read from it, make sure to specify a starting record number of 1, or to close it first if you are using LSVFILER! - The SIGNUPCACHE configuration variable becomes obsolete with release 1.6e, as the SIGNUP registration code has been entirely rewritten. - The REGISTER command has been modified to accept wildcards with the OFF (ie un-register) option, and to reject them in other cases. This prevents the insertion of entries of the form 'A*@XYZ' (where XYZ is a valid NJE node) by node administrators - these entries could only be removed through the user of a wildcard, which would also cause other entries to be deleted. However, 'REGISTER *@XYZ OFF', which previously resulted in a "user not found" error, is now a supported way of removing entries for deleted nodes from the SIGNUP FILE. Passwords --------- - The password file allows multiple registration conditions to exist; they can, however, be set up only by the postmaster. Whenever such a condition exists, the password picked up for a given user (for purposes of checking it against the one supplied on the command line) is the one corresponding to the address "closest" to the one the command came from, where "closest" is defined by an algorithm which will not be documented, but which obviously favours an exact match over a synonym. - The PW command will act on the "closest" password. The PW DEL command will delete the "closest" password, and notify the user in case more passwords are defined; additional PW DEL commands can be used to delete such passwords. Distribution lists ------------------ - Distribution lists allow multiple registration conditions to exist; they can, however, be set up only by the list owner (or postmaster). Whenever such a condition exists, the entry picked up for a given user when a CONFIRM, SET or SUBSCRIBE/SIGNUP command is received is the one "closest" to the one the command came from. - The SIGNOFF and DELETE/DELHERE commands will, however, remove all synonym registrations for the target user. This is different from the PW command, but consistent with the fact that these commands are often used (with the NETWIDE option) to delete all subscriptions for an "expired" account. It would be undesirable to require several such commands to be issued by the NAD to completely wipe out all the subscriptions of a given user. - The QUERY command will show all synonym registrations for the issuing user. - For peered lists, host-name aliasing is used to determine the closest peer to the subscriber, for both subscription requests and the EXPLODE command. File-server functions --------------------- - The AFD and FUI files allow multiple registration conditions to exist; they can, however, be set up only by the postmaster. The AFD and FUI commands operate on the "closest" registration, with the exception of the DELETE command, which deletes all registrations. - Host-name aliasing is used for purposes of detecting the default NJE file format and class for the GET command (and any other command which results in a file being sent to the user - INFO, LIST, GIVE, etc). However, when the request comes in from a domain-style (ie non-NJE) origin, NJE file formats are NOT automatically used; LISTSERV first tries to determine whether or not the file being sent is "text" and, if so, delivers it via mail (whereas binary data is delivered via NJE). The reason for this apparent inconsistency (which preserves compatibility in most cases where it does make sense to preserve it) is to present a more "friendly" interface to end-users who know only about mail and use the GET command only to retrieve list archives, minutes of meetings, etc. These people might not know what to do with a Netdata file called 'MEET9001 MINUTES' sitting in their "reader" - indeed, the first thing they would probably do if they knew how would be to "put it in the mail with everything else". When the file is a program, it does not make any sense to send it via mail and the incompatibility with 1.6d is no longer a concern. Finally, it should be noted that an explicit 'F=' keyword on the command line will override this behaviour. That is, if [log in to unmask] sends a 'GET SOME FILE' command and LISTSERV decides that it "looks like text" (the algorithm is not foolproof), it will be sent as mail; if Joe finds out that the file is unusable, he can resend a 'GET SOME FILE F=NETDATA' to force the use of Netdata format (or, similarly, 'GET SOME PROGRAM F=LPUNCH' to prevent the use of Netdata). - Host-name aliasing is used for purposes of calculating the daily GET quotas of end-users; that it users with an internet address no longer get twice the normal quota through their ability to send commands from two possible origins. DISTRIBUTE ---------- - For FILE (as opposed to MAIL or RFC822) distributions to domain-style addresses, the equivalent NJE address is used whenever available to deliver the final file, under the assumption that it is unlikely to be text in PUNCH format. - For all types of DISTRIBUTE jobs, host-name aliasing is used to select the closest backbone server to the final recipient. Final delivery is still done through the local (LISTSERV-node) MAILER. ************************************************************************* * Performance: Support for the message-suppression functions of VM30091 * ************************************************************************* LISTSERV will now make use of the message-control functions of the VM30091 SPE, if available, in order to suppress undesired 'Sent file' and 'File spooled to' messages. Support for the 'QUIET' modification is NOT discontinued - this new support is in addition to the FORM QUIET specification. The VM30091 SPE is part of the base code of RSCS V2.3, and available from any recent PUT tape for RSCS V2.2; it is not available at all for earlier versions of RSCS (V2.1, V2.0 and V1.3). This SPE allows LISTSERV to cause RSCS to set flags in the NJE headers indicating that file transmission messages are not desired; these flags are ignored by versions of RSCS not running the SPE. JNET does not presently (as of V3.4) respect these flags either, but this might of course change in a future version. You are advised to code a 'VM30091 = 1' statement in your LOCAL SYSVARS (see the description in SAMPLE SYSVARS) if you have the SPE available, or 'VM30091 = 0' otherwise. If you do not code anything, LISTSERV will attempt to determine the availability of this function by itself - and may guess incorrectly if you are running V2.2. *********************************************************************** * Enhancement: "Dumb user" detector filters commands sent to the list * *********************************************************************** At popular request, a "dumb user" detector has been included in release 1.6e in an attempt to identify and reject commands erroneously sent to list addresses, rather than to the LISTSERV userid. It should be clearly understood that this new logic is not and can never become 100% fool-proof; its goal is to catch most of the "silly commands" sent to distribution lists, without rejecting any but a negligible fraction of "valid" postings. Since the logic is expected to be refined with every release, the algorithm will not be documented; instead, the following statements of intent are provided: 1. The code does not (and will not be changed to) attempt to detect commands sent to the list as a file, rather than as mail. The whole philosophy of the present LISTSERV file distribution functions requires LISTSERV not to attempt to "receive" the distributed file (which might actually be a "deck" of several files, or data in a format that VM is not able to process). LISTSERV can therefore not gain access to the contents of the file and attempt to analyze it. 2. The goal is to catch 90% (in terms of volume) of commands sent to the list, while rejecting less than 1% of valid postings. 3. The code will NOT be modified to recognize "new" types of commands unless a significant enough amount of them are being seen. Occasional "random commands" such as 'database info' will not be trapped. 4. The code WILL be modified to stop trapping "valid" messages if a case where a significant amount of them are erroneously rejected is found. In any case, rejected postings are NOT processed by the command interpreter, but forwarded back to the originator with an explanatory text, which tells the user how to send the command for actual execution and suggests ways to bypass the filter if the message was a "valid" one. Note that this explanatory text is not to be considered as a "specification" of (parts of) the filtering algorithm, but merely as an aid to the irritated user who does want his posting to be processed. This text will be modified as necessary if the algorithm is changed.