Re: RSE performance against very large source files

Hi Dave and others,

I'll start with some general performance tips, a quick summary of what
we've done so far and what we are planning to do. Then I'll finish off
with some specific ideas on your problem Dave.

General Performance Tips:

- Do not put your workspace on a network drive. Always keep it on a local
hard drive. WDSC / RDi reads and writes lots of files while it is running.
Putting your workspace on a shared drive will slow it down drastically.

- Periodically de-fragment your hard drive (for the same reason as #1).

- I have seen some virus scanners that scan every file read / written to
the hard drive (at least for "untrusted applications"). This can have a
serious impact on performance as well. I'm not suggesting you turn off
your virus scanner, but it may be something you want to investigate and
see if there is anything that can be configured to improve it. Diagnosing
if this is the case can be tricky. The one way we were able to determine
this before was to setup WDSC to download a whole pile of smaller files to
the local hard drive and then open Windows task manager, sort by the CPU
column by descending then watch for processes that spike up to the top and
disappear. Not an exact science, but a starting point. For anyone who
finds out this is a problem please let me know. I'll like some more
information on this one.


Changes so far:

- In 7.0.0.2 we changed the algorithm used for retrieving member lists
which should improve performance for member lists by at least a factor of
10 compared to 7.0.0.1. Especially for long member lists.

- In RDi 7.1 we changed some of the caching algorithms to improve caching
performance when a large list had been cached. Prior to this if a large
list (> 10,000 objects or members) got cached it would subsequently affect
performance.

- Over the WDSC 7.0 and RDi 7.1 releases we did memory and performance
profiling (and made some fixes) to make things run faster with less
memory.


Performance is an ongoing work item for us, but we do need help in
identifying and fixing specific problems that occur in the real world.
It's difficult for us to take a posting that describes a general
performance problem and be able to map that back to a fix in the code.
Both of the above 7.0.0.2 and 7.1 fixes resulted from PMRs (which turned
into APARs) where the WDSC user reporting the problem worked closely with
us to help identify the exact problem and fix it. I've read the separate
thread about the difficulties in opening PMRs but unfortunately don't have
a fix for that :(

I've asked Roshane Silva from our team to take a deep dive into the
performance issues and to a) open defects for problems we need to fix and
b) publish some performance tuning / best practices for WDSC / RDi. He's
going to start by scanning the mailing list archive for leads. Feel free
to email him directly if you have some suggestions (rhsilva@xxxxxxxxxx).

And now over to Dave's problem:

There is definitely something going wrong if takes 3 1/2 minutes to return
a list of 13 members. A couple things I can specifically think of:

1. You may have expanded the QDDSSRC file at one point in time and the
entire list of 19,892 members got added to the cache and that is not
affecting performance (see my note above about the problem and fixes we
made to caching in RDi 7.1). The workaround for this is to clear your
cache (via the Remote Systems > iSeries >Cache preferences page) and then
immediately shutdown and restart WDSC (the clear cache preference clears
the disk cache but not the in-memory cache, if you don't shutdown and
restart then you may end up writing the memory cache back to disk. We have
an open requirement to fix this so clear cache clears both disk and
memory).

2. What are the library, file, member and member type fields for the
filter (right click on the filter and select Change). RSE filters allow
you to enter a generic name for the library, file and member fields but
the i5/OS API we call (yes it's QUSLMBR) only allows a generic member
name. So if there is a generic library or file name we end up having to do
multiple calls to first resolve the library, then the files within the
library, then the members within each file. So the multi-generic support
is powerful, but can lead to poor performance. For example; you don't want
to have something like: * for the library name, then QDDSSRC and PRF* for
the member name. This would first resolve all libraries on the system,
then call the member list api for each library. (I just opened a
requirement to add a warning when creating filters with multi-generic
fields).

Let me know if those work or not. If they don't work then we can take it
offline to diagnose the problem (if you're interested). It will likely
require us sending you some updated code with extra logging to narrow down
where the time is being spent.

Thanks.

Don Yantzi
Technical Lead
WebSphere Development Studio Client for iSeries
IBM Toronto Lab





"Dave Shaw" <daveshaw@xxxxxxxxxxxxx>
Sent by: wdsci-l-bounces@xxxxxxxxxxxx
02/04/2008 09:43 AM
Please respond to
Websphere Development Studio Client for iSeries <wdsci-l@xxxxxxxxxxxx>


To
wdsci-l@xxxxxxxxxxxx (wdsci-l)
cc

Subject
[WDSCI-L] RSE performance against very large source files






I didn't think the title 'EGL vs. Java' applied anymore. ;-)

A case in point on my specific performance issues:

This morning I created a new member filter. The library is our production
source library, the file name is QDDSSRC, and the member name filter is
PRF*. I expanded the filter. After 2 minutes, I got the message asking
if I wanted to cancel the request. I answered No, and finally got the
results after 3 1/2 minutes, getting back 13 member names. I did a DSPFD
on this source file just to check the total number of members. It took 39
minutes to tell me it has 19,892 members. Note that our QRPGLESRC has a
LOT more.

I can, of course, do a WRKMBRPDM on this file, and then a position to
using PRF, and get to what I'm looking for in a couple of seconds. I will
stipulate that doing an F17 and putting PRF* in the name filter produces a
very slow response as well, but I don't need to do that with PDM since it
has the position to function.

All I wanted to do was open a particular member in CODE Designer so I
could look at the screens. Even though in this case I knew the member
name, Ctrl-Shift-A was useless because it doesn't open CODE Designer. I
could have found the member and used the PDM user option to open it in
CODE and had the thing up in less than 30 seconds (by the time I set my
screen groups up in Designer), vs. 4 minutes for the RSE to CODE process.
My co-workers would just use PDM/SDA, and be happy with the more limited
function because of the up-front time savings.

This is one of the issues that's killing acceptance of the product in this
shop. It would be very hard to justify NOT having a copy of PDM for
everyone with this kind of issue, and if the company has to buy everyone a
copy of ADTS, why should they also buy RDi? Yes, I know why, but it
hasn't been an easy sell getting people to use WDSCi for the price of a
decent PC and some learning time; now I have to justify buying it, too.

Dave Shaw
Mohawk Industries