MIDRANGE dot COM Mailing List Archive

Home » MIDRANGE-L » August 2014

Re: writing large single record stream file



I'm trying to get caught up on this thread, and finding it full of confusing (or downright incorrect) replies. Here are some of my thoughts... my apologies if you already have some of these answers.

1) Do not use the crappy 'writeline' subprocedure. I wrote this IFS stuff as one of my first forays into public writing, and I didn't have so much experience then. Back then, I thought if I put together a simple example that was easy to digest, the reader would go ahead with their new understanding and write a much better & more robust routine for themselves. What I've learned since then is that this never happens -- people just take the overly simplified and not-very-well-tested routine that I put in the book, use it verbatim... Please don't do that.

2) writeline() is for CRLF delimited data, which is NOT what you're looking for here... another reason to not use it.

3) Stream files are NOT organized into records, which makes this whole topic confusing. What is a "single record" stream file?!! I think you just mean a continuous stream of data, right?

What may be confusing you is that certain programs (such as Windows Notepad, Wordpad, or Word, or the IBM i DSPF, EDTF commands) will move to the next line of the display if an LF or CRLF character sequence is found. So, data appears on different rows on the screen/printer. This doesn't really mean the data is in a different "record", unless the application has decided to treat LF/CRLF as record delimiters... But simply not writing LF or CRLF to your file might be all you're after here?

The other thing that might be confusing you is EDI has the concept of "segments" that are sort of like records... but that's done by program logic (read until you get the segment terminator, then call it a "segment"... it's the program that does this logic, the OS has no clue.)

4) Each time you call write() it adds the data immediately after the last write. So you can call write() many times in succession, and the data will be continuous.

Someone said there's a 65535 limit... that's an old (no longer relevant) RPG variable limit. It never applied to stream files (since you can call write() more than once or use pointers...)

5) Someone said that stream files are limited to 2GB. This was a V5R2 limit that was changed in V5R3. BUT, that does not mean all file systems support huge files -- some do and some do not. In particular, the QDLS file system is always more limited than the root file system. The root file system in V5R3+ is capable of 1TB files... I hope that's big enough?

If files larger than 2GB are important to you, make sure you include the O_LARGEFILE flag on the open() call.

6) PLEASE use %trim() sparingly. Use VARYING fields so that the extra blanks aren't added by RPG... %trim() can really waste a lot of CPU cycles, it makes the code more cumbersome to read/maintain, and just... yuck.

7) The concept of "wrapped/unwrapped" has to do with how EDI data is adapted to fit in record-oriented files like PFs. It really doesn't apply to stream files... and most EDI people probably won't even understand what this concept is, since Windows, Unix, Mac, etc use stream files for everything, they aren't even familiar with the idea of trying to force stream data to work in a record-oriented file.

Not sure if any of that is helpful, but I wanted to get it off my chest :-)

On 8/12/2014 10:10 AM, Jim Franz wrote:
As Charles said, I can write x bytes, clear, read, write x more bytes, even
beyond the 16Mb Jon mentioned.
Can someone point me to sample code that can append to the stream? The
current code I've always used only writes records that end with CRLF
eval @len = %len(%trimr(@textbig))
callp writeline(fd:%addr(@textbig):@len) (and writeline is part of
Scott's download sample code)

Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2014 by MIDRANGE dot COM and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available here. If you have questions about this, please contact