Re: massive document storage in the IFS - how to handle Mimix

[Jim Oberholtzer <midrangel@xxxxxxxxxxxxxxxxx>]

Yep:  PowerHA.  The biggest downside is the IFS has to be in an iASP, 
but with symbolic links there does not need to be any software changes, 
although you'd have to check with RJS on that for sure.

PowerHA will keep up since is does not do a save/restore of each object, 
it is replicating at the memory page level so the overhead is 
significantly less.  Keep in mind that no replication product will be 
worth it's beans if there is not sufficient bandwidth between the source 
and target machines.

I am in the process of implementing PowerHA for a customer that has very 
significant use of the IFS for documents as well, although they wrote 
their own document management system for it.  PowerHA did not even break 
a sweat keeping up.

Jim Oberholtzer
Chief Technical Architect
Agile Technology Architects


On 5/16/2013 11:30 AM, Stone, Joel wrote:
> We are storing a large number of documents in the IFS.  There are 500,000+ docs stored each month.  At the end of every month, a new directory is created on the IFS and new docs are stored in the new dir.
>
> For example, docs stored today are stored in directory  201305 until the end of May.  Then we will create 201306 and June docs will go there.
>
> We are using RJS' Webdocs product, although the same issues would exist with any document storage system.
>
> There is a searchable database (the index files) on the iseries to locate a doc, and the corresponding actual doc image on the IFS.
>
>
> We purge docs after their retention period has lapsed - from both the index files AND the actual doc from the IFS.
>
>
> Mimix cannot keep up with the large amount of documents.  So it was turned off for this part of the IFS.
>
> We would like to backup only the CURRENT directory with Mimix, which is 201305 today.  All NEW docs go there and all revisions to OLD docs reside in the current directory 201305 also.
>
> The only changes to the OLD directories occur when documents are purged.
>
> In this case, the LIVE system is OK, but the backup system will not purge the old docs that were purged on the LIVE system.
>
> Any solutions to this dilemma?  Some possible solutions:
>
>
> 1)      Run the expired document purge on both the LIVE system and the backup (Mimix) system?  This solution may have inherent issues because the BACKUP system index file is being updated by Mimix for users adding new docs AND the purge pgm.
>
>
> 2)      Mimix the index files and the current IFS directory 201305, which will produce orphan docs on the backup system.  Periodically read thru the IFS and delete any orphan docs.
>
>
> 3)      Other solutions to keep the LIVE and BACKUP system in sync?
>
> Thanks!