Re: Indexing source as HTML using search engine

Excellent!  That answers my questions.  I think using Domino would probably
work even better for us due to its views, as you mentioned.  After all,
this would only really be used for mass searches that would normally take a
long time using FNDSTRPDM2 or some in-house version.  My FNDSTR version
similar to FNDSTRPDM2 (not FNDSTRPDM) ran several hours when run over all
libraries and files and consumed a small chunk (not sure how much) of
iSeries memory during that whole time.  I was concerned about bringing
overall system performance down.  It sounds like using a search engine
would only use a small bit of PC memory and be lightning fast.  If we ever
went the Domino route, I think we would start small and only update the IFS
documents and index every morning when no one is here.  If programmers
wanted more current changes and used the searching quite a bit, then we may
take the next step and do some type of copy to IFS when promoting to
production.  I'm not sure.  I agree that this is all interesting and it is
just not worth the work for us, as you mentioned.  I only think this would
really be used for mass searches that would normally consume time and
system resources.  By the way, I have heard WDSC will plan on using the
FNDSTRPDM2 for mass searches which is an iSeries command running on the
iSeries just like WDSC runs FNDSTRPDM now for searches.  Maybe for some
shops, they would see the benefit of using a search engine.  If a shop felt
the justification for programming this, implemented it, and was willing to
share, maybe more would go that way since the work would already have been
laid out.  But then you still have to consider whether it is worth the
additional DASD for holding the copies of the source and how often mass
searches are performed.

Thanks again,
Craig Strong

** Mark wrote:
                                                                                
                     
 Generally speaking using a search engine should be exponentially faster.       
                     
 This is because the search engine will have indexed all of the pages and       
                     
 really just does a lookup in its index.  And yes, I was thinking you would     
                     
 use the iSeries Search engine that you mentioned.                              
                     
                                                                                
                     
 Tecnhically speaking, I think you could probably just use CPYTOSTMF to         
                     
 convert the source to the IFS.  The reason I was thinking of using HTML,       
                     
 is just that it would be nice when looking at the source in a browser to       
                     
 see it all color-coded and prettied up.                                        
                     
                                                                                
                     
 In terms of getting started, I think you would need to do a mass               
                     
 conversion of your source code and then index it with the search engine.       
                     
 Then, as an on going process, you could just incorporate this into your        
                     
 change management process.  When you promote a change to production, you       
                     
 could just re-convert that source and update the search engine index for       
                     
 those documents.                                                               
                     
                                                                                
                     
 I have not tried it myself, but unless the search engine index has             
                     
 problems with all of the cryptic names in source code, I do not see why it     
                     
 would pretty well.                                                             
                     
                                                                                
                     
 Of course, this would only be a useful equivalent to the Browse and Print      
                     
 List options of FNDSTRPDM.  It doesn't replace the ability to do an edit       
                     
 source on the matches, or to run a user option on the matches.  So how         
                     
 useful it would be might really depend on how you use the feature.  From a     
                     
 speed and memory perspective, this should be way faster and use less           
                     
 resources with the notable exception of DASD. This would obviously use a       
                     
 significant amount of disk space to store the copies of the source.            
                     
                                                                                
                     
 Finally, another option.  I know that you use Domino in your shop.  What       
                     
 if you used the same concept, and instead of converting source to HTML,        
                     
 you added it as documents in a Domino database.  You could then use            
                     
 Domino's great searching and indexing, and could possibly even use             
                     
 Domino's views to organize source by library and file to provide even more     
                     
 flexibility.                                                                   
                     
                                                                                
                     
 I think these are all somewhat interesting ideas, but in the end I just do     
                     
 not see it as important enough to justify the work.  Maybe if I had 50         
                     
 programmers constantly doing FNDSTRPDM it would be worth it, but               
                     
 otherwise, I am content to just use the Search feature in WDSC.                
                     
                                                                                
                     
 Mark                                                                           
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
 craigs@xxxxxxxxx                                                               
                     
 Sent by: midrange-l-bounces@xxxxxxxxxxxx                                       
                     
 10/15/2003 12:37 PM                                                            
                     
 Please respond to Midrange Systems Technical Discussion                        
                     
                                                                                
                     
         To:     midrange-l@xxxxxxxxxxxx                                        
                     
         cc:                                                                    
                     
         Subject:        Indexing source as HTML using search engine            
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
                                                                                
                     
 If most source members were already converted to HTML into the IFS, would      
                     
 using a search engine be that much faster than using FNDSTRPDM2 or             
                     
 FNDSTRPDM to search for strings?  Would we be using the memory of the          
                     
 iSeries like FNDSTRPDM or the PC memory?  Would this tie up the PC or          
                     
 possibly the iSeries for quite awhile or is it even more more efficient?       
                     
 Could we have a job that runs every morning to convert any new or changed      
                     
 source members to HTML and then search HTML objects for any deleted            
                     
 members                                                                        
                     
 in QSYS.LIB and delete those HTML members (keep in sync)?  My initial          
                     
 reaction to all this was that I didn't see how searching HTML would be         
                     
 faster than searching members directly.  I am now reminded of how fast         
                     
 searching for strings occurs on this midrange list and how Mark Phippard       
                     
 is                                                                             
                     
 sharp.  Does anyone know of a way or free utility to convert source to         
                     
 HTML                                                                           
                     
 (or how it would look)?  How would we go about indexing the source code        
                     
 and                                                                            
                     
 using the web interface.  I don't think I have the time or confidence to       
                     
 try something new soon but I would like some help in thinking about this       
                     
 for the future.  If it is faster, then maybe WDSC can consider exploiting      
                     
 this method, assuming with some setup and more storage of course.              
                     
                                                                                
                     
 Is the web engine we are talking about the iSeries Webserver Search            
                     
 Engine?                                                                        
                     
 Seen here:                                                                     
                     
 
http://www-1.ibm.com/servers/eserver/iseries/software/http/services/search.html 
                    
                                                                                
                     
                                                                                
                     
 How might this be used?                                                        
                     
 Any insight or comments would be greatly appreciated.                          
                     
                                                                                
                     
 Thanks,                                                                        
                     
 Craig Strong                                                                   
                     
                                                                                
                     
 ** Mark wrote:                                                                 
                     
                                                                                
                     
  If fast member searches are that critical why not do something like           
                     
                                                                                
                     
  convert your source to HTML when it is put into production and place it       
                     
 in                                                                             
                     
  the IFS.  Then use a search engine to index the source code and provide a     
                     
                                                                                
                     
  web interface for searching your source?                                      
                     
                                                                                
                     
  I do not know if anyone has written any wholesale utilities to convert        
                     
                                                                                
                     
  source to HTML, but the LPEX editor within WDSC can do it.                    
                     
                                                                                
                     
  Mark