Re: Can you select replacement character in translation from unicode (UTF-16)

On 18 Apr 2013 07:11, Åke Olsson wrote:

Problem is as follows:

A file with text field in Unicode (UTF-16) is read by a job running
under CCSID 870 (Czech).

The text field has a character value that is valid in many language
locales but not in Czech.

As a result these characters are replaced with X'3F' in the
conversion process between Unicode and EBCDIC. 3F causes lots of
problem when it is passed on later in the chain.

Question: Is there anywhere where you can configure the system to
replace non-existing characters with some other hex code (preferably
X'40' since that will not cause problems later in the chain)?

I am not aware of any ability to override the 0x3F as the EBCDIC substitution character. I expect that given a database file field for which the data will not properly convert into the desired CCSID, the condition will need to be addressed after-the-fact. Basically, what Bruce said. But FWiW, even if the OP already is aware, for the archives:

I believe native\RLA informs of the possibility of substitution characters upon the request to Open a file, but I do not recall what specific feedback occurs for any particular row. I think maybe a data mapping error CPF5029 occurs, and a preceding error message CPF5035 RC21 diagnoses "21: The data has been converted from one CCSID to another. However, substitution characters were used for characters which could not be converted." to inform of the condition on any particular field(s) for that record; but the data had been copied to the buffer, with the substitution character.

The SQL informs of the condition with a warning that can be used to condition a /recovery action/ such as another SQL request to perform the following [or similar]:
Set :stringVar REPLACE(:stringVar, x'3F', x'40')

http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/topic/nls/rbagssqlconversrules.htm
_i SQL conversion rules i_
"When two strings are compared, one of the strings is first converted, if necessary, to the coded character set of the other string.
...
A warning occurs if a character of the string is converted to the substitution character.
..."

http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/topic/rzala/rzalaccl.htm
_i Listing of SQLSTATE values i_
"The tables in this topic provide descriptions of SQLSTATE codes that can be returned to applications...
...
Table 2. Class Code 01: Warning
...
SQLSTATE value: 01517
Meaning: A character that could not be converted was replaced with a substitute character.
SQLCODE Value: +335