Re: Question about processing interesting "UTF-8" characters from XML -- RPG400-L

Hi Doug

Good thoughts overall - and I agree as to how useful this is - but we are stuck with it now - further comment would be unwise.

As to replacing with a text equivalent - you have to understand how many of these things there are. So far someone has managed to get the "skull" and the "key" - on an iPhone, if you type "key" you get an option above the keyboard to press and get the emoji of a key - sheesh!

It's just not practical - my plan is to record the position where the offending byies are - XML-SAX tells us that - then check the bytes against a table - HEY WAIT! I __could__ put an alternate text in there, couldn't I?

I'll have to get some kind of sanction, but this could work!

Thanks
Vern

On 6/8/2017 11:48 AM, Douglas Dunn wrote:

What an absurd problem! Before I say anything, I know you already said that
"currently you have no limitations on the entry of emoji", so I assume that
is a matter of company policy. The curiosity is consuming me though - emoji
and EBCDIC? I'm not sure those words have ever been in the same sentence
before.

Of course, I do not know what your application is, but I really can't see
any case where emoji, em dash, or ellipsis are considered "useful" input. I
personally would try to get that requirement changed, and delete characters
that are not on codepage 37. It just seems like a lot of work, for what
reason exactly?

If that is not practical, another option might be to translate emoji to
their "text" equivalent: the "smiley" becomes "=D", etc. All of those
should be EBCDIC characters from there, but of course 1 character longer.
Em dash becomes regular dash, ellipse becomes "...".

Just my thoughts!

On Thu, Jun 8, 2017 at 6:57 AM, Henrik Rützou <hr@xxxxxxxxxxxx> wrote:

Hi Vernon,

what is it that you receive in the XML-file originally, is it UTF-8?

On Thu, Jun 8, 2017 at 2:15 PM, Vernon Hamberg <vhamberg@xxxxxxxxxxxxxxx>
wrote:

Yáll

We get XML files from our field associates who use iPhones to enter
service information. That data is sent up to the IBM i in XML files.

We are using XML-SAX to process these files. But the process stops when

it

can't parse the XML, and at this time someone goes into the XML and

cleans

up the problem.

I am to find a way to eliminate as many parsing issues as possible.

Here's

what I've done so far, with help from Barbara Morris.

The things that are failing include emojis (some values are free text
entry and can contain anything available on an iPhone keyboard - and

there

are currently no limitations on this - don't ask!!)

Another thing that fails are things like an ellipsis or an em dash -

these

do not exist in EBCDIC 37.

The former use of XML-SAX did not include the ccsid option, so it tried

to

bring XML values into a CCSID 37 variable - and that can't be done, hence
the 351 parsing error status code.

I've changed this so that the option parameter is "doc=file ccsid=ucs2" -
we are at 7.1, so 1208 (UTF-8) is not an option.

So the values are all returned to RPG in UCS-2, and this is working OK

for

things like the ellipsis and em dash - more on this in a moment. The

emojis

still don't parse, because they are 4-byte entities in UTF-8 and don't
exist in UCS-2. I have a plan to take care of those, based on the offset
into the XML file that XML-SAX tells us in the event of an exception.

Back to the horizontal ellipsis - in UTF-8 this is a 3-byte sequence, in
hex, X'E280A6' - I see that in the XML file in the IFS that is tagged as
CCSID 1208 - that's required.

In the UCS-2 value, this is a 2-byte sequence, x2026.

In the program I assign the UCS-2 value to a column in a PF that is CCSID
37 - it appears there as X'0E447f0F' when I use DSPPFM on the PF..

At first I was not sure what this was - did a google on it and was led to
a site with IBM937 info - finally I understood this is a DBCS character
with shift-in and shift-out characters.

When I write this to the PF and use DSPPFM, it looks like " àÉ ".

I can accept that in the short term but wonder if we can do better - I
believe that the ellipsis prints as a blank at the customer, so that

would

be a suitable option, to get, perhaps, the X'3F' unprintable byte in that
place.

Or is there a way that I can take advantage of the DBCS-OPEN (is that the
term?) and actually print the ellipsis? I've never dealt with DBCS.

Or should I use iconv instead of just the default RPG character

conversion

when assigning the UCS-2 value to a CCSID-37 variable?

Thanks much
Vern
--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L)
mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our affiliate
link: http://amzn.to/2dEadiD

--
Regards,
Henrik Rützou

http://powerEXT.com <http://powerext.com/>
--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L)
mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our affiliate
link: http://amzn.to/2dEadiD