Re: Calculate hash for a *very* large file

Hi Dan,

I repeat... it's not precisely right for your application. You'll need to modify it. But you should be able to get what you want with a little work.

Change it to read the file with an F-spec and calculate the MD5 on the record. Remove the stream file logic. Add any additional logic you need to translate the file to ASCII. etc, etc.


Dan wrote:

On Thu, Jun 18, 2009 at 5:36 PM, Scott Klement <rpg400-l@xxxxxxxxxxxxxxxx>wrote:

You don't have to load the entire file into memory ahead of time in
order to perform an MD5 hash.

I wrote an article that demonstrates creating an MD5 hash over a stream
file in the IFS. Not quite what you're looking for, but it's written in
RPG, you should be able to modify it for your purposes. The point is...
it performs an MD5 hash over a (potentially) very large file without
loading the whole thing into memory first.

Thanks Scott! I downloaded the source code and compiled it. Got it to work
on a 150,000 record file, but it choked on a file with 4,750,000 records.
Both files had a record length = 2000. Error message given was: Object is
too large to process. (CPE3525) The problem is that message is issued in
the ReportError procedure, and I am unable to determine where the error
actually occurred.

I'll take another look at this in the morning.

- Dan