Question about processing interesting "UTF-8" characters from XML

Yáll

We get XML files from our field associates who use iPhones to enter service information. That data is sent up to the IBM i in XML files.

We are using XML-SAX to process these files. But the process stops when it can't parse the XML, and at this time someone goes into the XML and cleans up the problem.

I am to find a way to eliminate as many parsing issues as possible. Here's what I've done so far, with help from Barbara Morris.

The things that are failing include emojis (some values are free text entry and can contain anything available on an iPhone keyboard - and there are currently no limitations on this - don't ask!!)

Another thing that fails are things like an ellipsis or an em dash - these do not exist in EBCDIC 37.

The former use of XML-SAX did not include the ccsid option, so it tried to bring XML values into a CCSID 37 variable - and that can't be done, hence the 351 parsing error status code.

I've changed this so that the option parameter is "doc=file ccsid=ucs2" - we are at 7.1, so 1208 (UTF-8) is not an option.

So the values are all returned to RPG in UCS-2, and this is working OK for things like the ellipsis and em dash - more on this in a moment. The emojis still don't parse, because they are 4-byte entities in UTF-8 and don't exist in UCS-2. I have a plan to take care of those, based on the offset into the XML file that XML-SAX tells us in the event of an exception.

Back to the horizontal ellipsis - in UTF-8 this is a 3-byte sequence, in hex, X'E280A6' - I see that in the XML file in the IFS that is tagged as CCSID 1208 - that's required.

In the UCS-2 value, this is a 2-byte sequence, x2026.

In the program I assign the UCS-2 value to a column in a PF that is CCSID 37 - it appears there as X'0E447f0F' when I use DSPPFM on the PF..

At first I was not sure what this was - did a google on it and was led to a site with IBM937 info - finally I understood this is a DBCS character with shift-in and shift-out characters.

When I write this to the PF and use DSPPFM, it looks like " àÉ ".

I can accept that in the short term but wonder if we can do better - I believe that the ellipsis prints as a blank at the customer, so that would be a suitable option, to get, perhaps, the X'3F' unprintable byte in that place.

Or is there a way that I can take advantage of the DBCS-OPEN (is that the term?) and actually print the ellipsis? I've never dealt with DBCS.

Or should I use iconv instead of just the default RPG character conversion when assigning the UCS-2 value to a CCSID-37 variable?

Thanks much
Vern