Re: Dealing with very large blocks of character data

Hi, David:

Could you post an sample of the data you are trying to search, at http://code.midrange.com, and indicate what kinds of values you are looking for? We do not need to see any "real" data, but perhaps you could create a small "example" of what the text data looks like, and some of the values you are searching for, without revealing any "proprietary" information, and then perhaps someone might come up with some additional approaches?

For example, if you can copy the data to a stream file, then perhaps you could use a "Unix" approach, and use some of the AIX tools in PASE or QShell, such as grep, to help narrow down the amount of text you actually need to search, and then "pipe" those results into your program that looks for certain values, or redirect the output of grep to another (temporary) stream file, and then process that (much smaller) file?

All the best,

Mark S. Waterbury

> On 11/30/2011 3:16 PM, David Gibbs wrote:

On 11/30/2011 2:07 PM, Mark S. Waterbury wrote:

Does this data contain any embedded null characters (x'00')? If so,
then you may find using the C functions can be somewhat
"problematic" as C strings are "null terminated" -- this means as
soon as a X'00' is encountered, it is considered to be "end of
string." *:-/*

No, it will be simple text.

You might also want to look at the following New MI instructions
available for use as built-in functions in any ILE language:

I'm more familiar with C functions than I am with MI.

Where is this data coming from? Is it in a stream file in the IFS? If
so, you might also want to consider using the _mmap() or _mmap64()
API to map the entire stream file into teraspace, where you can then
easily search it, without the overhead of having to read the file
first.

Hmmm ... the data isn't coming from a stream file, but it's possible that the file could be written to a stream file.

Obviously I'll still need to use the C functions to scan& extract the contents ... but using mmap could eliminate some of the problems with loading the data.

Thanks!
david