[WEB400] How to present large datasets on the Web?

[Buck Calabro <Buck.Calabro@xxxxxxxxxxxx>]

I am the RPG guy, not a Web guy and so my question needs to be viewed in
that light.

Our Web person is struggling with her adaptation to the iSeries.  We have
customers who have multi-million record database files and this is
apparently none too common in the Java/Web universe.  In particular, we're
looking at ways to present a telephone bill on the web.  Most residential
users' bills can fit on two or three printed pages, and that's easy to do.
What's harder is a commercial customer who might have 5,000 toll calls a
month.  Or more.

We started by using the Java toolbox and direct program calls to RPG
programs to fetch the data.  Call the "customer summary" program and get
back the 20 or so data elements that make up the top of the bill.  Call the
"taxes" program and get back the dozen or so data elements that make up the
taxes section.  The ProgramCall class is nice because it has connection
pooling.

Then we get to long distance calls.  First, the direct program call method
has a 35 parameter limitation.  Second, it costs about .3 seconds per call.
Doing the math, it would take 25 minutes to call the "get a toll record"
program 5000 times.  Not a bargain.  By blocking the call records up (that
is, stuffing as many records as we can into a 64kb parameter) we can reduce
the number of calls considerably.  This does not seem terribly promising
from a maintenance standpoint.  Ultimately, there's a limit.  We're
prototyping a JDBC solution now, but data queues have been bandied about
too.

So.  Has anyone tried to routinely display a list that contains five or ten
thousand entries on the web?  What architectural approach did you take?
  --buck