RE: Slow Response on "grep" when called within RPG thru QSH Command -- RPG400-L

Can be done with SQL:
The following SQL Statement lists all IFS files under '/home/Hauser' (including all Subtrees) that contain 'whatever' in the content.
The only requirement is the SELECT statement needs to be executed under commitment control:

Select Path_Name
From Table (Qsys2.Ifs_Object_Statistics(Start_Path_Name => '/home/Hauser',
Subtree_Directories => 'YES',
Object_Type_List => '*ALLSTMF')) x
Where Lower(Get_Clob_From_File(Path_Name)) like '%whatever%'
Order By Path_Name ;

Mit freundlichen Grüßen / Best regards

Birgitta Hauser
Modernization – Education – Consulting on IBM i

IBM Champion since 2020

"Shoot for the moon, even if you miss, you'll land among the stars." (Les Brown)
"If you think education is expensive, try ignorance." (Derek Bok)
"What is worse than training your staff and losing them? Not training them and keeping them!"
"Train people well enough so they can leave, treat them well enough so they don't want to. " (Richard Branson)
"Learning is experience … everything else is only information!" (Albert Einstein)

-----Original Message-----
From: RPG400-L <rpg400-l-bounces@xxxxxxxxxxxxxxxxxx> On Behalf Of Daniel Gross
Sent: Thursday, 29 June 2023 22:23
To: RPG programming on IBM i <rpg400-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: Slow Response on "grep" when called within RPG thru QSH Command

My first idea was also SQL - IFS_OBJECT_STATISTICS is usually fast.

But the OP needs to scan the IFS files for a given string. And reading IFS files with SQL is notoriously slow.

But maybe reading the IFS with RPG and scanning for the string could be faster.

HTH
Daniel

Am 29.06.2023 um 21:47 schrieb McNierney_Chris via RPG400-L <rpg400-l@xxxxxxxxxxxxxxxxxx>:

The IFS is notoriously slow when it comes to searching. But, there is a much more straightforward way to list files using SQL.

Select Path_Name,
Object_Type,
Create_TimeStamp,
Last_Used_TimeStamp,
Data_Size
From
Table(QSYS2.IFS_OBJECT_STATISTICS('/PrincetonTMX/Outbound','YES'))
Where Path_Name Like '%20230530%';

This is a good article on how to use the table function:
https://www.rpgpgm.com/2019/11/using-sql-to-list-directories-and-files
.html

Chris

From: RPG400-L <rpg400-l-bounces@xxxxxxxxxxxxxxxxxx> on behalf of
Mohan Eashver <mohankva@xxxxxxxxx>
Date: Thursday, June 29, 2023 at 3:13 PM
To: RPG programming on IBM i <rpg400-l@xxxxxxxxxxxxxxxxxx>
Subject: Slow Response on "grep" when called within RPG thru QSH
Command Hello,

I have an IFS folder with thousands of XML files.

And I am developing an application coded in RPG, that will list on a
subfile, each IFS filename that contains a specific search string.

My program is working as expected, but the "grep" took about 4 to 5
minutes to return successfully with the search results.

Here are my steps, please let me know what I am doing Wrong OR what I
can do to increase the performance.

One thing I can think of is that there are 745 files in the folder
that I am searching.

The size of each file ranges from 18 KB up to 1 MB. Is that too many
files & sizes huge ?

So far I am only testing in a Test environment.

And the production system could have up to a half a million IFS files.

1. Change to Directory "/i3pl/XX/arc"
Cmd_Str = 'CD DIR(' + Quote + Source_Dir + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

2. Create the QTEMP file that will have the file listing from from "grep"
Cmd_Str = 'CRTDUPOBJ OBJ(I3DIR_LIST) FROMLIB(*LIBL)
OBJTYPE(*FILE) TOLIB(QTEMP)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

3 Direct the output of grep command from STDOUT to the QTEMP file
Cmd_Str = 'OVRDBF FILE(STDOUT) TOFILE(QTEMP/I3DIR_LIST) ' +
'OVRSCOPE(*JOB) SHARE(*YES) ' +
'OPNSCOPE(*JOB) SECURE(*YES) INHWRT(*NO)';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

4 Run the Shell command to search
* QSH CMD('grep -Rli "00353970985316338146" *.xml')
Cmd_Str = 'grep -Rli ' + D_Quote + %Trim(The_Search) +
D_Quote
+ ' *.xml';

Cmd_Str = 'QSH CMD(' + Quote + %Trim(Cmd_Str) + Quote + ')';
Exec_OS_Command (Cmd_Str : %size(Cmd_Str));

----
Regards,
Mohan Eashver
--
This is the RPG programming on IBM i (RPG400-L) mailing list To post a
message email: RPG400-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe,
or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.

--
This is the RPG programming on IBM i (RPG400-L) mailing list To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.