Re: massive document storage in the IFS - how to handle Mimix -- MIDRANGE-L

I like the SAV benefit - IFS saves have the potential for being really slow, since they seem to process each STMF by itself - lots of start-stop when the STMFs are small.

On 5/17/2013 10:54 PM, Joe Pluta wrote:

The size of the file directly mirrors the size of the CLOB storage
(since the key data is measured in 100s of bytes vs. the CLOB measuring
in 100s of KB, the non-CLOB data is negligible). Note there are still
practical issues: the maximum size for a CLOB field is around 16MB; I
wouldn't store files larger than that in the database anyway. But for
anything smaller you're in good shape.

Generally speaking SAV performance is a lot faster. We actually had to
exclude the folders containing millions of files just to get backups to
complete. Saving a single 4GB file is much more efficient than saving a
million 4KB files.

Note that in order to use CLOBs you need commitment control which means
you have to journal the files. In most applications, the journals are
not useful so you can just delete the receivers as they roll over.

Joe

Joe, have you seen any differences in the amount of storage each record takes?

I can understand the benefits you see with access, and such. Any differences with saves, other than SAVSECDTA?

--
Bryan

On May 17, 2013, at 9:34 PM, Joe Pluta <lists@xxxxxxxxxxxxxxxxx> wrote:

I just got done dealing with this issue. I had two completely different
systems using the IFS for XML document storage (one for order
management, the other for EDI). It turns out that the IFS in particular
and the IBM i in general really aren't set up for handling millions of
documents.

Two issues. First is that any directory over 16000 entries begins to
fail on most QShell or PASE commands, although the native commands such
as MOV and DEL still work. But that does you no good when you want to
JAR up a directory with 150,000 files. Seconds, user profiles don't
like owning millions of objects. They just don't. They get bigger and
bigger and bigger and eventually your SAVSECDTA starts taking hours.
Not exactly the best situation.

So what I did is move everything back to the database. I use CLOBs and
store everything in a file. That allows me to easily manage the data
using traditional keys rather than crazy nested directory structures and
archival is as simple as copying records from one file to another. It's
a beautiful thing. As I said, I'm using it for XML but it's just as
applicable to PDFs or indeed any other stream file.

TIP: For XML you can use the XML data type rather than BLOB or CLOB. It
checks that the data is well formed and hopefully some day will also
allow PureXML querying like it does in DB2 LUW.

Joe
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.