Re: CSV files with Byte Over Marks

On Tue, 19 Jun 2018 at 17:19, John Yeung <gallium.arsenide@xxxxxxxxx> wrote:

When you're inspecting a file using WRKLNK, option 5 (or,
equivalently, using DSPF directly on the file), what you get is a
character (not hex) display. In this mode, you CANNOT BE SURE what the
hex codes are.

You ***MUST*** press F10 to get the hex display. I don't care if you
don't see the BOM in character mode. That doesn't matter. Press F10.
The extra bytes will be there. EFBBBF. No matter what the CCSID is,
those bytes will be there at the beginning. Those three bytes are the
BOM for UTF-8, and CHGATR has no effect whatsoever on the bytes.

I don't want to speak for John, but I'm sure I missed out on why his
advice is, in general, useful and important.

Because the operating system understands different CCSIDs, it has
internal facilities to convert betwixt and between them. You can do
interesting and weird things like FTP a CCSID 1208 file from your IFS
to a CCSID 37 file in your IFS, and the IBM i FTP client will handle
the CCSID conversion. Likewise, it's /possible/ (but I haven't myself
tested it) that DSPF, WRKLNK and friends will try to do the conversion
for you as IBM i tries to display the text from the IFS file onto your
display screen. IBM i is pretty sure that a conversion should happen,
because the job is 65535 and the file is 1208. Now, whether DSPF
actually adds or removes a BOM as it displays the contents... I really
don't know.

Carel's latest advice is useful too, if only for a proof of concept.
Because CPYFRMIMPF is an IBM i utility, it knows about CCSIDs, and it
would handle the CCSID conversion.

I learnt quite a lot from Scott; searching his posts on CCSID, open()
and even the old code page threads has often been helpful to me. I
think if you look at his post I cited earlier on, it's exactly the
kind of change you need to make to his utility to allow it to handle
CCSID differences.
--buck