Re: How to present large datasets on the Web?

Buck,

just finished a project for a customer "web-enabling existing statistics"
with the following scenario:

- about 6 millions statistical records now; the number is expected to
increase to 10 millions.
- various statistics with unpredictable selections. Users were forced into
SEU to uncomment/enter/adapt a SORT source member that made an addrout-sort.
- some of the stats run for more than two hours. Nobody can/should wait
that long for a report.
- from summary screens there should be details accessible.

Users submit these statistics to the job queue via web. They see the status
of their jobs (JOBQ, RUN or a link to the finished stats run). I have all
the html pages ready (which are up to about 30 at the moment) and the .csv
files for PC data export. Some of them are quite big (500KB showing several
thousand lines is the average of the bigger ones); that's the detail with
#anchors so that one can position to the correct detail when there is a
click on the summary. So it's just simple web serving of static pages and
it's as fast as the web server and the line (LAN in this case) can be. MS
IE shows these pages normally in about 2 to 5 seconds for the first time on
a modern PC; later repositioning is much faster. (When the user keeps the
window open; and as these pages can be cached, they aren't re-submitted.)

Implemented with Net.Data and SQLRPGLE programs. Net.Data is also used to
produce the html and csv pages in batch. (Not called from the HTTP server,
a job of its own on a separate jobq in a separate sbs)

User comments are quite good.


best regards /  Mit freundlichen Grüssen

Anton Gombkötö
Avenum Technologie GmbH

http://www.avenum.com