Re: scanning user space -- RPG400-L

The user space employed in Test A was created with an initial size of 16Mb
and not marked extensible.

I've run some further tests this morning. The machine I'm using is much less
busy at this time than it was when I ran the initial tests yesterday, so I
reran Test A as well for comparison.

Test E was with a non-extensible user space created with an initial size of
32K and overwriting the data on each read, i.e. as per Test C but with the
32K buffer in a user space rather than memory.

Test F was as per A, but the user space with created with an initial size of
32K (not 16Mb) and then marked as extensible. I read the entire stream file
into the user space with a single read (i.e. the user space would have
needed to be extended from 32K up to 16Mb).

Test G was as per F, but this time I read the stream file in 32K increments.
With every read I advanced the buffer pointer further into the user space
which presumably means that the user space was extended by 32K for every
read except the first.

I've not bothered giving details of the CPU times.

A (rerun)  :    Ave. Elapsed    14.6 s    Ave. CPU:    2.0s
E                :    Ave. Elapsed    3.8s        Ave. CPU    2.0s
F                :    Ave. Elapsed    11.8s    Ave. CPU       2.0s
G                :    Ave. Elpased    9s        Ave, CPU        3.0s

A couple of points:

The 5 runs of Test A this morning were:    23s, 9s, 9s, 9s, 23s, average
14.6s. All of the other tests were pretty consistent, i.e. the elapsed time
didn't vary by more than 3-4 s.

Not quite sure how to interpret these results, but I'll hazard a guess. I
think memory management could well be the issue here. If A can get the
memory it wants, it runs pretty quickly. If it can't, it suffers badly,
waiting for 16Mb to be available.  I think F and G  perhaps show the
overhead of the extensibility, again probably resulting in memory
contention. E is pretty similar to yesterday's C, i.e. it looks like it's
the buffer size that's the critical factor, not whether a user space or
program memory is used.

That's the best I can do. Maybe someone who's got access to an empty machine
with a much larger stream file available could do something similar. I think
that would produce more reliable results.

Pete

----- Original Message ----- 
From: "Bob cozzi" <cozzi@xxxxxxxxx>
To: "'RPG programming on the AS400 / iSeries'" <rpg400-l@xxxxxxxxxxxx>
Sent: Tuesday, June 24, 2003 10:20 PM
Subject: RE: scanning user space


> Pete,
> I'm surprised at the user space being slower than the memory allocation,
> unless it was done all at once....
> I'm wondering if the user space and for that matter, the ALLOC opcode were
> done in chucks (like 32k chucks at a time) or all at once?
>
>
> Bob Cozzi
> Cozzi Consulting
> www.rpgiv.com
>
>
> -----Original Message-----
> From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx]
> On Behalf Of Pete Clifford (@riadne software)
> Sent: Tuesday, June 24, 2003 4:05 PM
> To: RPG programming on the AS400 / iSeries
> Subject: Re: scanning user space
>
> Anyone know what the calorie count is for humble pie? I'm never going to
> lose wait if I have to eat this much all the time!
>
> I did some timing tests earlier today and Scott is 100% correct.
>
> Working with a 16,381,440-byte stream file (i.e. just under 16Mb), I tried
4
> options:
>
> A. Reading the contents into a 16 Mb user space. This needed only a single
> call to "read".
>
> B. Reading the contents into 16 Mb of ALLOCed memory.
>
> C. Using a 32Kb buffer (just a simple data structure in the program). This
> needed about 500 reads.
>
> D. Using a 20-byte buffer. This needed over 800,000 reads.
>
> I ran 5 tests of each type and took the average elapsed time and CPU
seconds
> used.
>
> The results were a surprise to me.
>
> A.    43.4 seconds elapsed 3.0 seconds CPU time.
> B.    20.4 seconds elapsed 2.6 seconds CPU time.
> C.    4 seconds elapsed 2 seconds CPU time.
> D.    210 seconds elapsed 89 seconds CPU time.
>
> Several things come out of this.
>
> a) I guess A and B show the relative overhead of a user space and
> dynamically allocated memory. I'll be doing more ALLOCs in future.
> b) I expected D to be bad and C to be much better, but I didn't expect C
to
> be much better than A and B. Why? Have I somehow chosen some magic buffer
> size that corresponds to an internal IFS buffer size or something?
>
> So, I suppose the conclusion is, ditch the user space idea and either
ALLOC
> the memory, or, if you believe the results above (and I'm still not sure I
> do though I've run C in debug and watched it all the way through to make
> sure it was doing all I asked it to), use a simple 32K data structure.
>
> Can anyone explain why C is better than A and B? I wonder whether these
> results would be repeated on an empty machine (mine wasn't - no such
> luxury). I wonder whether A and B are getting paged out while the I-O is
> occurring while C isn't because it's doing more CPU intensive work
(multiple
> reads). The CPU usage is similar but the elapsed times are significantly
> different. Any suggestions?
>
> Pete
> ----- Original Message ----- 
> From: "Scott Klement" <klemscot@xxxxxxxxxxxx>
> To: "RPG programming on the AS400 / iSeries" <rpg400-l@xxxxxxxxxxxx>
> Sent: Tuesday, June 24, 2003 8:52 AM
> Subject: Re: scanning user space
>
>
> >
> > > I think I was one of the ones who suggested a user space. The reason I
> > > did that was to minimize the number of IFS I-Os that had to be done. A
> > > user space provides a bigger buffer that is available from an RPG data
> > > structure. You could ALLOC the memory, but a user space is less of a
> > > worry (maybe I shouldn't worry so much and have just worked with too
> > > many PC C-programmers who lose sleep over memory leaks?).
> >
> > why is it less of a worry?   Seems to me that if you deallocate your
> > ALLOC'ed memory, it'll get cleaned up when the activation group ends.
If
> > you forget to clean up your user space, it'll sit out there
indefinitely.
> >
> > Aside from that, they seem like they'd be the same level of worry!
> >
> > > My impression/surmise is that doing a single or (if the IFS file is
very
> > > big and won't fit into a max-size user space entirely) a few I-Os into
> > > the user space would be quicker than doing a large number of small
> > > reads.
> >
> > Depends on how small.   Yes, reading 8k from the file at a time will be
> > MUCH MUCH more efficient than reading 1 byte at a time.  But, you're
> > talking about reading 200mb at a time, writing it to another disk object
> > (the user space) and then reading that back again.
> >
> > Sure, access to the user space is faster than access to the stream file.
> > But, in both cases, you have to read the entire stream file.   And then,
> > with your solution, you still have to read the stream file.   Will
reading
> > it in a 200mb block save you so much time, that when you add reading the
> > user space to the equasion, it will still be shorter?   That's really
the
> > big question.
> >
> > I agree that reading the file in larger chunks is faster, but my guess
is
> > that once the buffer gets larger than 2-4k, making it larger than that
> > will not make an appreciable difference.
> >
> > Since his file has 2million records, and a user space is limited to only
> > 16mb in size, it creates another problem.   I'm guessing that his
average
> > record size is larger than 8 bytes.  :)
> >
> > But, even if you could have a 200mb user space, it seems likely to me
that
> > the system would not load it into RAM, since the object would be too
> > large.   (Though, I guess that would also depend on how much RAM you
have,
> > and how many other jobs are using it!)   So, the user space becomes
> > another disk access, causing more slowdown.
> >
> > >
> > > I'll try to find time to give my ideas on how to scan the user space
> > > later. I won't have a chance to test this out this morning, but I'm
> > > wondering whether the C-function sscanf might be useful to you. I
think
> > > prototyping this in RPG could be a challenge but I'm sure somebody on
> > > this board is up to it! :-)
> >
> > I've prototyped sscanf() before.   You can't make a generic prototype
> > for all uses of sscanf(), but you can make one for a specific use, like
> > this, quite easily.
> >
> > But, I don't like sscanf()... especially when working with very large
> > buffers, because of the potential for buffer overflows.   It amuses me
> > that you're worried abot ALLOC, but you recommend sscanf!!   sscanf and
> > the other scanf functions are a MAJOR cause of security problems on the
> > internet.
> >
> > Take this sample:
> >
> >         sscanf(userspace, "%[^\n]%*[\n]", buf);
> >
> > Since the contents of userspace are supplied by a 3rd party, how do you
> > know how long the largest possible value of buf will be?   What if a
> > record in the space exceeds that size?    sscanf() won't detect it,
it'll
> > just cheerfully keep reading the data from the space and writing it to
> > whatever memory happens to follow buf.
> >
> > On the iSeries, this would just cause the program to crash, most likely.
> > On the PC, however, you could use it to overwrite the program stack,
> > inserting your own code to be run...     This is where many of the
> > security flaws found in internet servers come from.
> > _______________________________________________
> > This is the RPG programming on the AS400 / iSeries (RPG400-L) mailing
list
> > To post a message email: RPG400-L@xxxxxxxxxxxx
> > To subscribe, unsubscribe, or change list options,
> > visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
> > or email: RPG400-L-request@xxxxxxxxxxxx
> > Before posting, please take a moment to review the archives
> > at http://archive.midrange.com/rpg400-l.
> >
> >
> >
>
>
> _______________________________________________
> This is the RPG programming on the AS400 / iSeries (RPG400-L) mailing list
> To post a message email: RPG400-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
> or email: RPG400-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/rpg400-l.
>
>
>
>
> _______________________________________________
> This is the RPG programming on the AS400 / iSeries (RPG400-L) mailing list
> To post a message email: RPG400-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
> or email: RPG400-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/rpg400-l.
>
>
>