Re: Slow Response on "grep" when called within RPG thru QSH Command

Am 29.06.2023 um 21:47 schrieb McNierney_Chris via RPG400-L <rpg400-l@xxxxxxxxxxxxxxxxxx>:

The IFS is notoriously slow when it comes to searching. But, there is a much more straightforward way to list files using SQL.

Select Path_Name,
Object_Type,
Create_TimeStamp,
Last_Used_TimeStamp,
Data_Size
From Table(QSYS2.IFS_OBJECT_STATISTICS('/PrincetonTMX/Outbound','YES'))
Where Path_Name Like '%20230530%';

This is a good article on how to use the table function:
https://www.rpgpgm.com/2019/11/using-sql-to-list-directories-and-files.html

Chris

From: RPG400-L <rpg400-l-bounces@xxxxxxxxxxxxxxxxxx> on behalf of Mohan Eashver <mohankva@xxxxxxxxx>
Date: Thursday, June 29, 2023 at 3:13 PM
To: RPG programming on IBM i <rpg400-l@xxxxxxxxxxxxxxxxxx>
Subject: Slow Response on "grep" when called within RPG thru QSH Command
Hello,


I have an IFS folder with thousands of XML files.

And I am developing an application coded in RPG, that will list on a
subfile, each IFS filename that contains a specific search string.


My program is working as expected, but the "grep" took about 4 to 5 minutes
to return successfully with the search results.


Here are my steps, please let me know what I am doing Wrong OR what I can
do to increase the performance.

One thing I can think of is that there are 745 files in the folder that I
am searching.

The size of each file ranges from 18 KB up to 1 MB. Is that too many files
& sizes huge ?

So far I am only testing in a Test environment.

And the production system could have up to a half a million IFS files.

1. Change to Directory "/i3pl/XX/arc"
Cmd_Str = 'CD DIR(' + Quote + Source_Dir + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

2. Create the QTEMP file that will have the file listing from from "grep"
Cmd_Str = 'CRTDUPOBJ OBJ(I3DIR_LIST) FROMLIB(*LIBL) OBJTYPE(*FILE)
TOLIB(QTEMP)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

3 Direct the output of grep command from STDOUT to the QTEMP file
Cmd_Str = 'OVRDBF FILE(STDOUT) TOFILE(QTEMP/I3DIR_LIST) ' +
'OVRSCOPE(*JOB) SHARE(*YES) ' +
'OPNSCOPE(*JOB) SECURE(*YES) INHWRT(*NO)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

4 Run the Shell command to search
* QSH CMD('grep -Rli "00353970985316338146" *.xml')
Cmd_Str = 'grep -Rli ' + D_Quote + %Trim(The_Search) + D_Quote
+ ' *.xml';

Cmd_Str = 'QSH CMD(' + Quote + %Trim(Cmd_Str) + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

----
Regards,
Mohan Eashver
--
This is the RPG programming on IBM i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.