|
Changing the attribute didn't work... I changed it to 1208 and the parsing portion of my program failed.
-----Original Message-----
From: RPG400-L [mailto:rpg400-l-bounces@xxxxxxxxxxxxxxxxxx] On Behalf Of Vernon Hamberg
Sent: Tuesday, May 05, 2020 10:51 AM
To: RPG programming on IBM i <rpg400-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: CSVR4 and UTF-8
Seems CHGATR is the command to use - but the original file is in UTF-8,
according to Greg below. So the VALUE here should probably be 1208, if I
understand this a little.
On a little bit different question here - I don't remember if UTF-8 byte
strings use any of the hex values in NONDSP here - they might, since it
might use bytes from X'20' on.
Using %xlate assumes the content of the IFS file is in EBCDIC, right?
Enough for now! I'm very interested in this topic, as we are facing
Unicode issues here.
Vern
On 5/5/2020 8:36 AM, Eiresoft Software Solutions wrote:
Hi, not sure if this helps but have you tried converting 819 to 273 :
CHGATR OBJ('/tmp/test.csv') ATR(*CCSID) VALUE(273)
regards
Am 2020-05-05 15:05, schrieb Greg Wilburn:
Vern,
That is exactly correct... the CCSID of the file is 819. I'm not sure
how that is assigned other than the JOB ccsid that retrieves the file
via FTP (I have a CLP that runs a script saved in DDS text).
These are orders coming from a very popular website platform... an app
installed on the platform extracts orders and pushes them to our FTP
server for pickup. The app has a function called normalize_text() to
remove characters. Unfortunately, you can't apply it to the entire
tab-delimited string. It has to be applied to each-and-every field
we've selected for download.
EASY400 has a utility called CVT101 that contains StmfCvt to convert
from one to another, but that likely won't help.
I was hoping for an "easier" solution.
My service program (someone helped me with this too) uses %XLATE
against a constant that contains the hex representation of many
non-display characters.
dcl-c nondsp const(x'000102030405060708090A0B0C0D0E0F-
101112131415161718191A1B1C1D1E1F-
202122232425262728292A2B2C2D2E2F-
303132333435363738393A3B3C3D3E3F-
41');
dcl-s space char(%size(nondsp));
dcl-s result varchar(1024);
result = %trim(inchar);
result = %xlate(nondsp:space:result);
return result;
I think I also tried the SQL Translate() function, but RPG seemed a
bit faster.
Greg
-----Original Message-----
From: RPG400-L [mailto:rpg400-l-bounces@xxxxxxxxxxxxxxxxxx] On Behalf
Of Vernon Hamberg
Sent: Tuesday, May 05, 2020 2:59 AM
To: RPG programming on IBM i <rpg400-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: CSVR4 and UTF-8
Thomas - take a look at U+2018 and U+2019 and U+201C and U+201D. U+2019
in that page - a great site - is RIGHT SINGLE QUOTATION MARK.
U+2018 ‘ e2 80 98 LEFT SINGLE QUOTATION MARK
U+2019 ’ e2 80 99 RIGHT SINGLE QUOTATION MARK
U+201A ‚ e2 80 9a SINGLE LOW-9 QUOTATION MARK
U+201B ‛ e2 80 9b SINGLE HIGH-REVERSED-9 QUOTATION MARK
U+201C “ e2 80 9c LEFT DOUBLE QUOTATION MARK
U+201D ” e2 80 9d RIGHT DOUBLE QUOTATION MARK
These are among several characters that don't exist in EBCDIC - an
ellipsis or em dash. We had run into this challenge with text entered on
an iPhone app. And these can come form the Autocorrect options in MS
Word or Outlook.
I ended up using SQL on i to import the text, and it puts X'3F' where it
encounters characters it can't convert.
Greg asks about recommendations - maybe the only 1 to retain the
characters would be to use 1208 (UTF-8) as the CCSID - of course, that
can't be done without a whole lot of work.
And IBM do not provide conversion tables between UTF-8 and EBCDIC - how
could you, at least in their present form.
Does iconv have options to convert these typographer (another descriptor
of these things) characters into something like EBCDIC?
Cheers
Vern
On 5/5/2020 1:02 AM, Thomas Raddatz wrote:
I do not know what you mean with ' right single quotation mark '. I--
assume it is a ACUTE ACCENT or a GRAVE ACCENT according to UTF8
table https://www.utf8-chartable.de/.
I did a brief test with service program CSVR4 and the following test
data on our IBM i:
"ABC123","Scott Klement","123 Sesame St","Milwaukee,
WI","USA","","53132-1234",1000.00
"ABC123","Bärbel Böhm","Some Street","Some
City","Germany","","40721",1000.00
"ABC123","`Jürgen` ´Bärbeißer´","Some Street","Some
City","Germany","","40721",1000.00
The report produced by CSVDEMO shows the result expected:
File . . . . . : QSYSPRT
Control . . . . .
Find . . . . . .
*...+....1....+....2....+....3...
Acct Name
---------- ---------------------
ABC123 Scott Klement
ABC123 Bärbel Böhm
ABC123 `Jürgen` ´Bärbeißer´
The German Umlaute as well as the ACUTE ACCENT and GRAVE ACCENT are
correctly printed. Hence I assume that CSVR4 works fine.
We do not use CSVR4, so a brief test is all I can do.
Did you check the CCSID of your inpput? Is it 1208 (= UTF8)?
Thomas.
-----Ursprüngliche Nachricht-----
Von: RPG400-L [mailto:rpg400-l-bounces@xxxxxxxxxxxxxxxxxx] Im
Auftrag von Greg Wilburn
Gesendet: Montag, 4. Mai 2020 16:24
An: RPG400-L@xxxxxxxxxxxxxxxxxx
Betreff: CSVR4 and UTF-8
I have a program that is using the CSVR4 service programs to read
tab delimited text files that we pull down from a website. The site
is using UTF-8 character set... occasionally, we have issues with
character translation.
Example: x'e2 80 99' (right single quotation mark) makes a real
mess of the customer's name.
I have a utility that removes non-display characters, but in this
case I need to keep the character.
Any recommendations on changes that could be made to the process
that would eliminate some of these translation issues?
Thanks,
Greg
This is the RPG programming on IBM i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/rpg400-l.
Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription
related questions.
Help support midrange.com by shopping at amazon.com with our affiliate
link: https://amazon.midrange.com
As an Amazon Associate we earn from qualifying purchases.
This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].
Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.
midrange.com
