Re: writing large single record stream file

Scott,
Appreciate the response and details.
Yes, my wording is very "record oriented". One of the earlier posts pointed
me to write() and that has worked well (assuming it passes their test
tonight).

Even though you call it a first foray into public writing, I have used the
samples in a hundred or so apps over the years, and it has served well (and
still running). In cases like this, I've learned some more details.

Thanks to all who responded, and I think we cleared up some of the limits
that no longer apply.

Jim Franz

On Tue, Aug 12, 2014 at 3:41 PM, Scott Klement <midrange-l@xxxxxxxxxxxxxxxx>
wrote:

Jim,

I'm trying to get caught up on this thread, and finding it full of
confusing (or downright incorrect) replies. Here are some of my
thoughts... my apologies if you already have some of these answers.

1) Do not use the crappy 'writeline' subprocedure. I wrote this IFS stuff
as one of my first forays into public writing, and I didn't have so much
experience then. Back then, I thought if I put together a simple example
that was easy to digest, the reader would go ahead with their new
understanding and write a much better & more robust routine for themselves.
What I've learned since then is that this never happens -- people just
take the overly simplified and not-very-well-tested routine that I put in
the book, use it verbatim... Please don't do that.

2) writeline() is for CRLF delimited data, which is NOT what you're
looking for here... another reason to not use it.

3) Stream files are NOT organized into records, which makes this whole
topic confusing. What is a "single record" stream file?!! I think you
just mean a continuous stream of data, right?

What may be confusing you is that certain programs (such as Windows
Notepad, Wordpad, or Word, or the IBM i DSPF, EDTF commands) will move to
the next line of the display if an LF or CRLF character sequence is found.
So, data appears on different rows on the screen/printer. This doesn't
really mean the data is in a different "record", unless the application has
decided to treat LF/CRLF as record delimiters... But simply not writing
LF or CRLF to your file might be all you're after here?

The other thing that might be confusing you is EDI has the concept of
"segments" that are sort of like records... but that's done by program
logic (read until you get the segment terminator, then call it a
"segment"... it's the program that does this logic, the OS has no clue.)

4) Each time you call write() it adds the data immediately after the last
write. So you can call write() many times in succession, and the data will
be continuous.

Someone said there's a 65535 limit... that's an old (no longer relevant)
RPG variable limit. It never applied to stream files (since you can call
write() more than once or use pointers...)

5) Someone said that stream files are limited to 2GB. This was a V5R2
limit that was changed in V5R3. BUT, that does not mean all file systems
support huge files -- some do and some do not. In particular, the QDLS
file system is always more limited than the root file system. The root file
system in V5R3+ is capable of 1TB files... I hope that's big enough?

If files larger than 2GB are important to you, make sure you include the
O_LARGEFILE flag on the open() call.

6) PLEASE use %trim() sparingly. Use VARYING fields so that the extra
blanks aren't added by RPG... %trim() can really waste a lot of CPU
cycles, it makes the code more cumbersome to read/maintain, and just...
yuck.

7) The concept of "wrapped/unwrapped" has to do with how EDI data is
adapted to fit in record-oriented files like PFs. It really doesn't apply
to stream files... and most EDI people probably won't even understand what
this concept is, since Windows, Unix, Mac, etc use stream files for
everything, they aren't even familiar with the idea of trying to force
stream data to work in a record-oriented file.

Not sure if any of that is helpful, but I wanted to get it off my chest :-)



On 8/12/2014 10:10 AM, Jim Franz wrote:

As Charles said, I can write x bytes, clear, read, write x more bytes,
even
beyond the 16Mb Jon mentioned.
Can someone point me to sample code that can append to the stream? The
current code I've always used only writes records that end with CRLF
eval @len = %len(%trimr(@textbig))
callp writeline(fd:%addr(@textbig):@len) (and writeline is part of
Scott's download sample code)
Jim