RE: scanning user space -- RPG400-L

Pete,
I'm surprised at the user space being slower than the memory allocation,
unless it was done all at once....
I'm wondering if the user space and for that matter, the ALLOC opcode were
done in chucks (like 32k chucks at a time) or all at once?


Bob Cozzi
Cozzi Consulting
www.rpgiv.com


-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx]
On Behalf Of Pete Clifford (@riadne software)
Sent: Tuesday, June 24, 2003 4:05 PM
To: RPG programming on the AS400 / iSeries
Subject: Re: scanning user space

Anyone know what the calorie count is for humble pie? I'm never going to
lose wait if I have to eat this much all the time!

I did some timing tests earlier today and Scott is 100% correct.

Working with a 16,381,440-byte stream file (i.e. just under 16Mb), I tried 4
options:

A. Reading the contents into a 16 Mb user space. This needed only a single
call to "read".

B. Reading the contents into 16 Mb of ALLOCed memory.

C. Using a 32Kb buffer (just a simple data structure in the program). This
needed about 500 reads.

D. Using a 20-byte buffer. This needed over 800,000 reads.

I ran 5 tests of each type and took the average elapsed time and CPU seconds
used.

The results were a surprise to me.

A.    43.4 seconds elapsed 3.0 seconds CPU time.
B.    20.4 seconds elapsed 2.6 seconds CPU time.
C.    4 seconds elapsed 2 seconds CPU time.
D.    210 seconds elapsed 89 seconds CPU time.

Several things come out of this.

a) I guess A and B show the relative overhead of a user space and
dynamically allocated memory. I'll be doing more ALLOCs in future.
b) I expected D to be bad and C to be much better, but I didn't expect C to
be much better than A and B. Why? Have I somehow chosen some magic buffer
size that corresponds to an internal IFS buffer size or something?

So, I suppose the conclusion is, ditch the user space idea and either ALLOC
the memory, or, if you believe the results above (and I'm still not sure I
do though I've run C in debug and watched it all the way through to make
sure it was doing all I asked it to), use a simple 32K data structure.

Can anyone explain why C is better than A and B? I wonder whether these
results would be repeated on an empty machine (mine wasn't - no such
luxury). I wonder whether A and B are getting paged out while the I-O is
occurring while C isn't because it's doing more CPU intensive work (multiple
reads). The CPU usage is similar but the elapsed times are significantly
different. Any suggestions?

Pete
----- Original Message ----- 
From: "Scott Klement" <klemscot@xxxxxxxxxxxx>
To: "RPG programming on the AS400 / iSeries" <rpg400-l@xxxxxxxxxxxx>
Sent: Tuesday, June 24, 2003 8:52 AM
Subject: Re: scanning user space


>
> > I think I was one of the ones who suggested a user space. The reason I
> > did that was to minimize the number of IFS I-Os that had to be done. A
> > user space provides a bigger buffer that is available from an RPG data
> > structure. You could ALLOC the memory, but a user space is less of a
> > worry (maybe I shouldn't worry so much and have just worked with too
> > many PC C-programmers who lose sleep over memory leaks?).
>
> why is it less of a worry?   Seems to me that if you deallocate your
> ALLOC'ed memory, it'll get cleaned up when the activation group ends.   If
> you forget to clean up your user space, it'll sit out there indefinitely.
>
> Aside from that, they seem like they'd be the same level of worry!
>
> > My impression/surmise is that doing a single or (if the IFS file is very
> > big and won't fit into a max-size user space entirely) a few I-Os into
> > the user space would be quicker than doing a large number of small
> > reads.
>
> Depends on how small.   Yes, reading 8k from the file at a time will be
> MUCH MUCH more efficient than reading 1 byte at a time.  But, you're
> talking about reading 200mb at a time, writing it to another disk object
> (the user space) and then reading that back again.
>
> Sure, access to the user space is faster than access to the stream file.
> But, in both cases, you have to read the entire stream file.   And then,
> with your solution, you still have to read the stream file.   Will reading
> it in a 200mb block save you so much time, that when you add reading the
> user space to the equasion, it will still be shorter?   That's really the
> big question.
>
> I agree that reading the file in larger chunks is faster, but my guess is
> that once the buffer gets larger than 2-4k, making it larger than that
> will not make an appreciable difference.
>
> Since his file has 2million records, and a user space is limited to only
> 16mb in size, it creates another problem.   I'm guessing that his average
> record size is larger than 8 bytes.  :)
>
> But, even if you could have a 200mb user space, it seems likely to me that
> the system would not load it into RAM, since the object would be too
> large.   (Though, I guess that would also depend on how much RAM you have,
> and how many other jobs are using it!)   So, the user space becomes
> another disk access, causing more slowdown.
>
> >
> > I'll try to find time to give my ideas on how to scan the user space
> > later. I won't have a chance to test this out this morning, but I'm
> > wondering whether the C-function sscanf might be useful to you. I think
> > prototyping this in RPG could be a challenge but I'm sure somebody on
> > this board is up to it! :-)
>
> I've prototyped sscanf() before.   You can't make a generic prototype
> for all uses of sscanf(), but you can make one for a specific use, like
> this, quite easily.
>
> But, I don't like sscanf()... especially when working with very large
> buffers, because of the potential for buffer overflows.   It amuses me
> that you're worried abot ALLOC, but you recommend sscanf!!   sscanf and
> the other scanf functions are a MAJOR cause of security problems on the
> internet.
>
> Take this sample:
>
>         sscanf(userspace, "%[^\n]%*[\n]", buf);
>
> Since the contents of userspace are supplied by a 3rd party, how do you
> know how long the largest possible value of buf will be?   What if a
> record in the space exceeds that size?    sscanf() won't detect it, it'll
> just cheerfully keep reading the data from the space and writing it to
> whatever memory happens to follow buf.
>
> On the iSeries, this would just cause the program to crash, most likely.
> On the PC, however, you could use it to overwrite the program stack,
> inserting your own code to be run...     This is where many of the
> security flaws found in internet servers come from.
> _______________________________________________
> This is the RPG programming on the AS400 / iSeries (RPG400-L) mailing list
> To post a message email: RPG400-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
> or email: RPG400-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/rpg400-l.
>
>
>


_______________________________________________
This is the RPG programming on the AS400 / iSeries (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.