I have just became fed up with EXECIO's poor performances and above all with
the philosophical problem, should I attempt to take advantage of the release 4
'STEM' option? This does save a few CPU cycles but it forces me to write code
to 'emulate' the STEM processing under SP3 -- this costs me a lot of
programming time and there is no way I can do it as a separate exec, the
routine must therefore but included and customized in the program every time.
I have therefore decided to spend a few hours on an EXECIO substitute, once
and for all. My goal was to make it 5 times faster for program stack output,
and 3 times faster for STEM output (EXECCOMM eats up an enormous amount of CPU
time in both cases so...) This program can only read a WHOLE file, from top to
bottom, and always FINIShes it -- that's restrictive but that's what I end up
doing 95% of the time in the code and it's the only way I can use faster I/O
procedures.
Here are a few stats. I tried the stuff both on a RECFM V file (my favourite
EARNTECH NOTEBOOK, 9984 records) and a RECFM F one (a TEXT file of 3918
records).
A) RECFM V file:
- Time required for REXX to read up the data from the program stack:
--> 5.9 seconds
- EXECIO to program stack: 5.9 seconds
- EXECIO to stem: 10.6 seconds (better than 5.9+5.9)
- LSVFILER to program stack: 1.05 second (1.05+5.9 better than 10.6)
- LSVFILER to stem: 2.9 seconds
B) RECFM F file:
- Time required for REXX to read up the data from the program stack:
--> 2.5 seconds
- EXECIO to program stack: 2.5 seconds
- EXECIO to stem: 4.4 seconds
- LSVFILER to program stack: 0.5 second
- LSVFILER to stem: 1.3 second
LSVFILER is therefore 5.6 times faster on program stack reads and 3.7 times
faster on stem reads. That's better than the 5 and 3 I wanted to achieve, but
I knew it was difficult to do better. Actually LSVFILER reads the file in 10
times faster than EXECIO, but must still spend the same amount of time
queueing it to the program stack or EXECCOMMing it. The STEM option works for
both VM/SP 4 and VM/SP 3 of course, so I can now choose to use it
unconditionally :-) :-) :-)
Brief technical description: LSVFILER reads in the file in a single I/O
operation, and will then output its contents one record at a time. Should CMS
run out of virtual storage, LSVFILER would immediately release as much storage
as possible from the buffer housing the contents of the file and make a retry.
If the retry fails, an error is reported to the caller. If the retry is
successful, no error is reported (but CMS will type an INSUFFICIENT STORAGE TO
SATISFY DMSFREE CALL FROM <whatever>; this message is issued as a consequence
of the DMSCAT call and cannot be suppressed). If no storage can be released
because nothing in the buffer has been transferred to the program stack yet,
an error condition is reported. The same applies for STEM output processing.
Additionally, when STEM output is required, LSVFILER will obtain a 16k buffer
at initialization to act as a batch for EXECCOMM calls. If a 16k buffer cannot
be obtained, LSVFILER will take what the system can give it and work on this.
As EXECCOMM requests are built up, they are batched in the buffer until it
becomes full. Any request larger than the buffer is directed to EXECCOMM
without causing the buffer to be flushed. DMSCAT is called via BALR instead of
SVC 202. etc, etc. I don't think it could be made much faster, in particular,
increasing the buffer size to 32k or 64k did not yield any noticeable result.
The only way to speed the thing up is to optimize DMSFRE, but this is another
matter entirely... :-)
Anyway the thing will be part of release 1.5j of LISTSERV, and I will slowly
rewrite the various execs to make use of LSVFILER (STEM whenever possible. I
would be very interested in non-LISTSERV beta-testers as I never feel too
confident with new MODULEs, especially those which seem to work perfectly
after the first assembly... :-) I am going to make the module TEMPORARILY
available from my LISTSERV (TELL LISTSERV AT FRECP11 GET LSVFILER MODULE), so
anybody can retrieve it freely.
Ah, I was going to forget the syntax... It's very simple:
LSVFILER fn ft <fm> <(options>
options: FIFO LIFO STRIP STEM xxxx.
rc: 0 (ok), 1 (truncated -- if output to program stack), 24 (syntax), 28 (file
not found), 100 (file is on a CDF disk), 1xxx (DMSFREE error xxxx), 2xxx
(read error xxxx), 3xxx (error xxx from DMSCAT or EXECCOMM).
The default fm is '*', the default options are: FIFO.
Don't expect sophisticated error diagnostics from the program: LSVFILER |1_!%"
\SDF}E› XX,( will merely tell you rc=28 (file not found) without typing any
nasty message on the console :-)
Eric
|