Re: strtok

[Scott Klement <rpg400-l@xxxxxxxxxxxxxxxx>]

Hi Tim,

> I am using the strtok api and have used it many times.  I seemed to
> have come across a situation that it is not working.  I have a string
> that is has followed:

My advice to you:  Never use strtok() in an RPG program. I'm not sure how
RPG programmers started using this API, but it's not a very good one, and
it really should be avoided.

strtok() has a lot of flaws, and even on Unix systems where delimited
strings are a way of life and C is the most popular programming language,
the strtok() function is discouraged because of it's flaws.

There's a function called strsep() that has all but replaced strtok() --
the only people still using strtok() are people who want maximum
portability, since it's part of the ISO C89 standard, it's available just
about everywhere -- but that's not an issue for an RPG programmer.  Your
programs won't work on every version of Unix, anyway -- since most of them
don't support RPG at all.


> The problem is that the api seems to skip the delimiters that are
> stacked.  This is throwing off my fields that are returned.  I thought I
> would get back blanks for each field that has nothing between the
> delimiters.

No. Strtok() is clearly documented to skip over all consecutive
delimiters.

> Instead the api just seems to keep looking for data to
> return.  Is there a way to prevent this or have the number of delimiters
> returned so I know how many fields to skip as well?

I don't understand why you're using strtok() at all.  It would take, what,
15 minutes to write an equivalent routine in RPG?  I mean, all strtok()
does is read each character in a string looking for a delimiter.  That's
not exactly difficult to code.

Why not just write a routine that does the job, put it into a service
prgoram where you can call it from everywhere and be done with it?!

In fact, since it won't take any significant time, here... I'll write one
for you:

     H DFTACTGRP(*NO)

     D gettok          PR          1024A   varying
     D   string                   65535A   varying const options(*varsize)
     D   delims                      50A   varying const
     D   pos                         10I 0

     d blah            s            100A   varying
     D tokens          s            100A   varying dim(50)
     D x               s             10I 0
     D y               s             10I 0

     D msg             s             52A

      /free

          Blah = 'Test||||||123|x|||8900';
          x = 0;
          y = 0;

          //
          //  Get each token from the string and DSPLY it
          //

          dow ( x <= %len(blah) );
             y = y + 1;
             msg = 'token ' + %char(y) + '='
                 + gettok( blah : '|' : x );
             dsply msg;
          enddo;

          //
          // Or, perhaps just dump all tokens to an array
          //

          x = 0;
          tokens(*) = gettok( blah : '|' : x );

          for y = 1 to %elem(tokens);
             msg = 'elem ' + %char(y) + '=' + tokens(y);
             dsply msg;
          endfor;

          *inlr = *on;

      /end-free




      *+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      * gettok():  Get next token from string
      *
      *     string = (input) string to search
      *     delims = (input) delimiters
      *        pos = (i/o) next position to search.  At the start
      *                    of a string, set this to zero.  Pass
      *                    it on subsequent calls to remember where
      *                    you left off.
      *
      * Returns the token found.
      *+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
     P gettok          B
     D gettok          PI          1024A   varying
     D   string                   65535A   varying const options(*varsize)
     D   delims                      50A   varying const
     D   pos                         10I 0

     D res             s           1024A   varying
     D start           s             10I 0
     D c               s              1A

      /free

          start = pos + 1;
          %len(res) = 0;

          for pos = start to %len(string);

             c = %subst(string:pos:1);

             if %check(delims:c) = 1;
                res = res + c;
             else;
                leave;
             endif;

          endfor;

          return res;

      /end-free
     P                 E