Re: scanning user space -- RPG400-L

> I think I was one of the ones who suggested a user space. The reason I
> did that was to minimize the number of IFS I-Os that had to be done. A
> user space provides a bigger buffer that is available from an RPG data
> structure. You could ALLOC the memory, but a user space is less of a
> worry (maybe I shouldn't worry so much and have just worked with too
> many PC C-programmers who lose sleep over memory leaks?).

why is it less of a worry?   Seems to me that if you deallocate your
ALLOC'ed memory, it'll get cleaned up when the activation group ends.   If
you forget to clean up your user space, it'll sit out there indefinitely.

Aside from that, they seem like they'd be the same level of worry!

> My impression/surmise is that doing a single or (if the IFS file is very
> big and won't fit into a max-size user space entirely) a few I-Os into
> the user space would be quicker than doing a large number of small
> reads.

Depends on how small.   Yes, reading 8k from the file at a time will be
MUCH MUCH more efficient than reading 1 byte at a time.  But, you're
talking about reading 200mb at a time, writing it to another disk object
(the user space) and then reading that back again.

Sure, access to the user space is faster than access to the stream file.
But, in both cases, you have to read the entire stream file.   And then,
with your solution, you still have to read the stream file.   Will reading
it in a 200mb block save you so much time, that when you add reading the
user space to the equasion, it will still be shorter?   That's really the
big question.

I agree that reading the file in larger chunks is faster, but my guess is
that once the buffer gets larger than 2-4k, making it larger than that
will not make an appreciable difference.

Since his file has 2million records, and a user space is limited to only
16mb in size, it creates another problem.   I'm guessing that his average
record size is larger than 8 bytes.  :)

But, even if you could have a 200mb user space, it seems likely to me that
the system would not load it into RAM, since the object would be too
large.   (Though, I guess that would also depend on how much RAM you have,
and how many other jobs are using it!)   So, the user space becomes
another disk access, causing more slowdown.

>
> I'll try to find time to give my ideas on how to scan the user space
> later. I won't have a chance to test this out this morning, but I'm
> wondering whether the C-function sscanf might be useful to you. I think
> prototyping this in RPG could be a challenge but I'm sure somebody on
> this board is up to it! :-)

I've prototyped sscanf() before.   You can't make a generic prototype
for all uses of sscanf(), but you can make one for a specific use, like
this, quite easily.

But, I don't like sscanf()... especially when working with very large
buffers, because of the potential for buffer overflows.   It amuses me
that you're worried abot ALLOC, but you recommend sscanf!!   sscanf and
the other scanf functions are a MAJOR cause of security problems on the
internet.

Take this sample:

        sscanf(userspace, "%[^\n]%*[\n]", buf);

Since the contents of userspace are supplied by a 3rd party, how do you
know how long the largest possible value of buf will be?   What if a
record in the space exceeds that size?    sscanf() won't detect it, it'll
just cheerfully keep reading the data from the space and writing it to
whatever memory happens to follow buf.

On the iSeries, this would just cause the program to crash, most likely.
On the PC, however, you could use it to overwrite the program stack,
inserting your own code to be run...     This is where many of the
security flaws found in internet servers come from.