On Thu, 23 Feb 1995 19:59:02 EST Roger Fajman <[log in to unmask]> said:
 
>Well,  the incident  that  prompted my  question was  a  new list  owner
>attempting to  add 10,000  subscribers to  his new  list in  6 humongous
>files of  multiple ADD  commands. It  bogged down  LISTSERV on  our poor
>little 9370-60  for hours  and we  still ended up  deleting most  of the
>files. The ADDs were QUIET, so the notifications weren't the cause.
 
The  algorithms for  lists of  that size  were significantly  enhanced in
FIX18AX2. The time  to fire up the ADD commands  does cost something, but
it would be  neligible compared to the inherent slowness  of adding users
to  a large  list  without the  improved algorithms  (in  fact there  was
nothing  wrong/broken with  the  old algorithms,  they  just didn't  have
optimizations  to linearize  performance for  large lists).  Now that  it
takes 0.2 sec wall clock on the average to add a user to a list with 100k
subscribers on a PC, the overhead for  firing up ADD 10k times might make
a difference, and I have no idea how  much. I always use the DD form when
adding a bunch of users because it's simpler once you know the syntax and
I know it will be somewhat faster. I've started a 10k ADD job on my PC in
the background and it's clearly going to take several minutes, although I
suspect  a large  fraction of  the elapsed  time is  spent scrolling  the
messages in the DOS box. The  PC definitely doesn't feel loaded. Maybe if
I could type HT it would go faster.  I guess I should just wait until the
list has grown to reasonable size. All right, it has reached 7000 now and
the PC feels slower  while the DOS box is not scrolling  quite as fast as
before. I think  it's no longer limited by the  scrolling speed. It seems
to be doing about 10 a second now.
 
>Eric mentioned a fix that we don't have that's supposed to speed up ADD,
>so that might help  some, but it's unclear to me if  it would be enough.
>Eric was vague (in a private exchange) about whether the ADD command job
>you describe  would be  significantly more  efficient than  multiple ADD
>commands. So I think we will go with the PUT method.
 
While PUT should currently be the fastest method, you should know that in
1.8a you still had LSVPUTL XEDIT. It's a tradeoff between the slowness of
XEDIT + REXX  and not running through the process  that wasn't scaling up
well past a couple thousand subscribers. If you install FIX18AX2 all that
junk goes away. Ah, the job just  finished. It took about 16 minutes, and
the output  file is  1.4M. It  would clearly  be better  to send  a dozen
smaller jobs.  Anyway PUT should  be a lot faster  since it works  on the
assumption  that the  list is  being  replaced. PUT  loads everything  in
memory and then checks for duplicates and the like entirely in memory, to
avoid disk I/O. ADD assumes you're going to be adding one user or maybe a
dozen, and that  it will be faster  to read the file every  time. It also
does a lot  more syntax checking and parsing than  PUT, because the input
address format is much more flexible. ADD  is not really meant to be used
with 10k entries.
 
  Eric