Re: Calculate hash for a *very* large file

Hi Dan,

You don't have to load the entire file into memory ahead of time in order to perform an MD5 hash.

I wrote an article that demonstrates creating an MD5 hash over a stream file in the IFS. Not quite what you're looking for, but it's written in RPG, you should be able to modify it for your purposes. The point is... it performs an MD5 hash over a (potentially) very large file without loading the whole thing into memory first.

Here's a link to that article (System iNetwork ProVIP subscription is required)
http://systeminetwork.com/article/utility-calculate-md5-hash-stream-file

Also, there's a small bug in the *CMD interface for the utility.. I forgot to specify LEN(5000) on the pathname parameter... but I'm sure you can fix that.


Dan wrote:

On Thu, Jun 18, 2009 at 4:04 PM, Charles Wilt <charles.wilt@xxxxxxxxx>wrote:

Here's one issue that may byte you with the hash....

When you have the data on the i, you're dealing with EBCDIC, on the Windows
server the data will be ASCII.

You'd need to convert the data to ASCII on the i before calculating the
hash
if you expect it to match the has calculated on Windows.

Having said that, you might for instance consider a program that runs in
PASE to produce the hash. Since PASE is ASCII, I believe the system will
translate the data for you automatically.

Thanks Charles. I did a small test using the same string on the i as well
as in a text file in Windows, and got the same MD5 value. I suspect the
service program might already be converting the input data:
// Get conversion table for 819 from 37
Qtqcvrt(Ccsid1:St1:StartMap:L1:Ccsid2:St2:Gccasn:L2:To819:L3:L4:Fb);
// Change message to ccsid 819 (ascii).
Convert(Inputdata:%Addr(To819):InputLength);
This is from the link I provided in my original post.

Unfortunately, I just tried it on a larger file, one of the smallest I'll
need to be able to test, and got RNQ0425:
The length requested for storage allocation is out of range.

I guess I need to figure out if I can allocate more than 16776704 bytes.
FWIW, we're running on V5R4.

- Dan