RE: How to present large datasets on the Web?

Buck,

My biggest result set that I allow is 999 records to be viewed at any one
time (e-RPG program).  It takes about 10 seconds to run that many(I need to
rewrite the program to have varying length fields so I don't have so many
%trim()'s), and another 4 seconds for the browser to download all of.  The
way this program works is it allows the user to select how many results they
want to see at any given time (usually 30 - 300) and also allows them to
select the criteria for the result (maybe they have a date range they want
to look for).  It is an intranet app so I can't show you exactly how it
works; but is it an option for you to show pieces of the phone bill and not
the entire 5000 right up front? What are your customers using the website
for, printing out a copy of the bill?  Or do they just peruse the entire
5000 records all at once?

Does the Web person need to go through an RPG pgm for security reasons?  If
you don't want them to have business logic in the Java program to develop a
result set, maybe you could pass them back an SQL statement that will then
be executed in the Java Servlet(or what have you).
hth,

Aaron Bartell

-----Original Message-----
From: Buck Calabro [mailto:Buck.Calabro@commsoft.net]
Sent: Monday, December 02, 2002 4:40 PM
To: web400@midrange.com
Subject: [WEB400] How to present large datasets on the Web?


I am the RPG guy, not a Web guy and so my question needs to be viewed in
that light.

Our Web person is struggling with her adaptation to the iSeries.  We have
customers who have multi-million record database files and this is
apparently none too common in the Java/Web universe.  In particular, we're
looking at ways to present a telephone bill on the web.  Most residential
users' bills can fit on two or three printed pages, and that's easy to do.
What's harder is a commercial customer who might have 5,000 toll calls a
month.  Or more.

We started by using the Java toolbox and direct program calls to RPG
programs to fetch the data.  Call the "customer summary" program and get
back the 20 or so data elements that make up the top of the bill.  Call the
"taxes" program and get back the dozen or so data elements that make up the
taxes section.  The ProgramCall class is nice because it has connection
pooling.

Then we get to long distance calls.  First, the direct program call method
has a 35 parameter limitation.  Second, it costs about .3 seconds per call.
Doing the math, it would take 25 minutes to call the "get a toll record"
program 5000 times.  Not a bargain.  By blocking the call records up (that
is, stuffing as many records as we can into a 64kb parameter) we can reduce
the number of calls considerably.  This does not seem terribly promising
from a maintenance standpoint.  Ultimately, there's a limit.  We're
prototyping a JDBC solution now, but data queues have been bandied about
too.

So.  Has anyone tried to routinely display a list that contains five or ten
thousand entries on the web?  What architectural approach did you take?
  --buck
_______________________________________________
This is the Web Enabling the AS400 / iSeries (WEB400) mailing list
To post a message email: WEB400@midrange.com
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/cgi-bin/listinfo/web400
or email: WEB400-request@midrange.com
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.