RE:Slow Response on "grep" when called within RPG thru QSH Cmd -- MIDRANGE-L

Are you saying that grep is always slow for you ?

Or is there a scenario where it works quickly for you ?

If it takes a while, one option might be to submit the grep to batch as a background job and come back and view the output from RPG once grep is done.

Regards,
Richard Schoen
Web: http://www.richardschoen.net
Email: richard@xxxxxxxxxxxxxxxxx

----------------------------------------------------------------------

message: 1
date: Thu, 29 Jun 2023 15:11:27 -0400
from: Mohan Eashver <mohankva@xxxxxxxxx>
subject: Slow Response on "grep" when called within RPG thru QSH Cmd

Hello,

I have an IFS folder with thousands of XML files.

And I am developing an application coded in RPG, that will list on a subfile, each IFS filename that contains a specific search string.

My program is working as expected, but the "grep" took about 4 to 5 minutes to return successfully with the search results.

Here are my steps, please let me know what I am doing Wrong OR what I can do to increase the performance.

One thing I can think of is that there are 745 files in the folder that I am searching.

The size of each file ranges from 18 KB up to 1 MB. Is that too many files & sizes huge ?

So far I am only testing in a Test environment.

And the production system could have up to a half a million IFS files.

1. Change to Directory "/i3pl/XX/arc"
Cmd_Str = 'CD DIR(' + Quote + Source_Dir + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

2. Create the QTEMP file that will have the file listing from from "grep"
Cmd_Str = 'CRTDUPOBJ OBJ(I3DIR_LIST) FROMLIB(*LIBL) OBJTYPE(*FILE) TOLIB(QTEMP)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

3 Direct the output of grep command from STDOUT to the QTEMP file
Cmd_Str = 'OVRDBF FILE(STDOUT) TOFILE(QTEMP/I3DIR_LIST) ' +
'OVRSCOPE(*JOB) SHARE(*YES) ' +
'OPNSCOPE(*JOB) SECURE(*YES) INHWRT(*NO)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

4 Run the Shell command to search
* QSH CMD('grep -Rli "00353970985316338146" *.xml')
Cmd_Str = 'grep -Rli ' + D_Quote + %Trim(The_Search) + D_Quote
+ ' *.xml';

Cmd_Str = 'QSH CMD(' + Quote + %Trim(Cmd_Str) + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

----
Regards,
Mohan Eashver
-------------- next part --------------
* Change to Directory "/i3pl/XX/arc"
Cmd_Str = 'CD DIR(' + Quote + Source_Dir + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

* Create the QTEMP file that will have the file listing from from "grep"
Cmd_Str = 'CRTDUPOBJ OBJ(I3DIR_LIST) FROMLIB(*LIBL) ' +
'OBJTYPE(*FILE) TOLIB(QTEMP)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

* Direct the output of grep command from STDOUT to the QTEMP file
Cmd_Str = 'OVRDBF FILE(STDOUT) TOFILE(QTEMP/I3DIR_LIST) ' +
'OVRSCOPE(*JOB) SHARE(*YES) ' +
'OPNSCOPE(*JOB) SECURE(*YES) INHWRT(*NO)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

* Run the Shell command to search
* QSH CMD('grep -Rli "00353970985316338146" *.xml')
Cmd_Str = 'grep -Rli ' + D_Quote + %Trim(The_Search) + D_Quote + ' *.xml';

Cmd_Str = 'QSH CMD(' + Quote + %Trim(Cmd_Str) + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));