Re: massive document storage in the IFS - how to handle Mimix

I just got done dealing with this issue. I had two completely different
systems using the IFS for XML document storage (one for order
management, the other for EDI). It turns out that the IFS in particular
and the IBM i in general really aren't set up for handling millions of
documents.

Two issues. First is that any directory over 16000 entries begins to
fail on most QShell or PASE commands, although the native commands such
as MOV and DEL still work. But that does you no good when you want to
JAR up a directory with 150,000 files. Seconds, user profiles don't
like owning millions of objects. They just don't. They get bigger and
bigger and bigger and eventually your SAVSECDTA starts taking hours.
Not exactly the best situation.

So what I did is move everything back to the database. I use CLOBs and
store everything in a file. That allows me to easily manage the data
using traditional keys rather than crazy nested directory structures and
archival is as simple as copying records from one file to another. It's
a beautiful thing. As I said, I'm using it for XML but it's just as
applicable to PDFs or indeed any other stream file.

TIP: For XML you can use the XML data type rather than BLOB or CLOB. It
checks that the data is well formed and hopefully some day will also
allow PureXML querying like it does in DB2 LUW.

Joe
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.