Re: UTF-8 or UTF-16

Aaron,

I can't say my answers are real crisp, but...

1 - If you are required to store Asian characters, what CCSID do you use?

I didn't catch where you mentioned where you are storing the data. As you
know there are a number of possible CCSIDs that can get the job done. As
you asked "what CCSID do you use" my answer is 1200 (UTF-16) assuming we're
talking DB2 tables/files. If you're talking IFS then I've used both 1200
(UTF-16) and 1208 (UTF-8). And if we go far enough back, 13488 (UCS-2).

2 - Are there IBM i server-side advantages to using UTF-16? (i.e. is text
searching quicker for Asian languages because they have two bytes in

UTF-16

vs. 3 in UTF-8?)

Assuming DB2 storage some consider the associated data types to be either
advantages or disadvantages. UTF-8 is implemented using the character data
type while UTF-16 is implemented using the graphic data type. With graphic
what you read is what you get, what you write is what gets written (well,
depending on the interface used to DB2). With character job CCSID can come
into play. From my answer to #1 above you can safely guess that I consider
conversion to/from the job CCSID to be a disadvantage.

If you are really going to be doing searches through the Chinese text then
having 50% more bytes to fetch and compare could have an impact (though I
don't know how much data you're talking about, so this could just be a nit).

3 - If you've used both UTF-8 and UTF-16, have you found one to be more
advantageous than the other?

Yes, but it usually is driven by the other application/platform. In general
I use UTF-16 when possible, but that really is just a personal preference.

Bruce



On Fri, Nov 22, 2013 at 3:14 PM, Aaron Bartell <aaronbartell@xxxxxxxxx>wrote:

My team has been looking at the advantages of using UTF-8 vs. UTF-16 if the
majority of the data you store is double-byte in nature. A number of
advantages and disadvantages are listed at this
link<http://en.wikipedia.org/wiki/UTF-8#Compared_to_UTF-16>:
http://en.wikipedia.org/wiki/UTF-8#Compared_to_UTF-16

One thing that stuck out to us on that link was this:

*Characters U+0800 through U+FFFF use three bytes in UTF-8, but only two in
UTF-16. As a result, text in (for example) Chinese, Japanese or Hindi could
take more space in UTF-8 if there are more of these characters than there
are ASCII characters. This happens for pure text[34] but rarely for HTML
documents or documents in XML based formats such as .docx or .odt. For
example, both the Japanese UTF-8 and the Hindi Unicode articles on
Wikipedia take more space in UTF-16 than in UTF-8.*

To sum it up, many (most?) Asian characters will be 3 to 4 bytes in UTF-8
and only 2 bytes in UTF-16. The majority of a webpage is markup, not
actual information from your database or text literals. Many (most?) new
applications convey data to the browser vs. staying on only on the machine.

*I now have 3 questions:*

1 - If you are required to store Asian characters, what CCSID do you use?

2 - Are there IBM i server-side advantages to using UTF-16? (i.e. is text
searching quicker for Asian languages because they have two bytes in UTF-16
vs. 3 in UTF-8?)

3 - If you've used both UTF-8 and UTF-16, have you found one to be more
advantageous than the other?

Aaron Bartell
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.