Re: Dealing with very large blocks of character data

On 12/1/2011 9:30 AM, Mark S. Waterbury wrote:

Could you post an sample of the data you are trying to search, at
http://code.midrange.com, and indicate what kinds of values you are
looking for? We do not need to see any "real" data, but perhaps you
could create a small "example" of what the text data looks like, and
some of the values you are searching for, without revealing any
"proprietary" information, and then perhaps someone might come up with
some additional approaches?

I really can't ... but suffice it to say, the data is basically free form text that needs to be parsed.

Think in terms of a humungous CL program with all the continuation characters removed.

For example, if you can copy the data to a stream file, then perhaps you
could use a "Unix" approach, and use some of the AIX tools in PASE or
QShell, such as grep, to help narrow down the amount of text you
actually need to search, and then "pipe" those results into your program
that looks for certain values, or redirect the output of grep to another
(temporary) stream file, and then process that (much smaller) file?

No, this text needs to be parsed in an RPG program to get specific beginning and ending positions in order to extract data.

Additionally, I don't want to put a requirement of having PASE (or even QShell) installed on the system ... not to mention the incredible performance hit that would be incurred by invoking those programs.

david