|
midrange.com
I don't know that I'm any expert either, but I'll give it a shot for a few
definitions and examples (which I hope come through):
Character set: A finite set of different graphic or control characters that
is complete for a given purpose. Each character is represented with a
character identifer such as Latin capital letter A being LA020000. One
example of a character set would be Character Set 640 - the Syntactic
Character Set
(Embedded image moved to file: pic01072.gif)An illustration of character
set 00640
Another character set would be Character Set 697 - the Country Extended
Character Set
(Embedded image moved to file: pic13559.gif)Country extended character set
00697
Graphic character: A visual representation of a character that is normally
produced by writing, printing, or displaying. For instance the number sign
(SM010000) is #.
Control character: A character whose occurrence specifies a control
function. For instance in EBCDIC DBCS environments the Shift Out and Shift
In control characters to begin and end a DBCS sequence.
Code point: A unique bit pattern that can serve as an element of a code
page to which a character can be assigned. The element is associated with
a binary value. The assignment of a character to an element of a code page
determines the binary value that will be used to represent each occurrence
of the character in a character string. Code points are one or more bytes
long. For instance in EBCDIC environments x'C1' is the Latin capital
letter A.
Code page: A set of assignments, each of which assigns a code point to a
character. Each code page has a unique name or identifier. Within a given
code page, a code point is assigned to one character. More than one
character set can be assigned code points from the same code page. For
example in Code Page 37 (EBCDIC USA/Canada)
(Embedded image moved to file: pic15607.gif) REQTEXT
As an example of more than one character set being assigned to a given code
page note that Character Set 640 and Character Set 697 are mapped onto Code
Page 37 (640 is a proper subset of 697).
CCSID: A number identifying a specific set of identifiers for encoding
scheme, character set, code page, and additional coding-related required
information (ACRI). For example CCSID 37 represents EBCDIC encoding, Code
Page 37, Character Set 697, and no special ACRI. CCSID 5035 represents
EBCDIC Mixed byte encoding, Code Page 1027, Character Set 1172, Code Page
300, and Character Set 370.
One problem that is often encountered is that for differenct CCSIDs the
same graphic character is represented by different code point values. If
for example we look at CCSID 500 (EBCDIC, code page 500, character set 697)
(Embedded image moved to file: pic10705.gif) REQTEXT
And CCSID 273 (EBCDIC, Code Page 273, Character Set 697)
(Embedded image moved to file: pic32130.gif) REQTEXT
We notice that the commercial at sign @ (SM050000) is x'7C' in CCSID 500
and x'B5' in CCSID 273. If the system is to accurately represent/process
SM050000 then it is rather important for the system to know what CCSID a
given piece of character data is in. The ideal is that if we were to
convert a string containing SM050000 to CCSID 277
(Embedded image moved to file: pic19113.gif) REQTEXT
from either CCSID 500 or CCSID 273 that we would end up with x'80' as that
is the correct code point value for SM050000 (@) in CCSID 277.
The iSeries will correctly perform these conversions so long as the data is
correctly identified by the CCSID currently in use to represent the data.
All too often though when a problem is encounted the data being processed
is not in the CCSID you might think it is. This can be due to variables
such as the terminal you are entering data from (they have their own
Character Set and Code Page values), the CCSID the job is running in, and
the CCSID conversion options in effect for your job. For this reason it is
often best/easiest to display the hex values of your data rather than
trusting the graphic character you see on your terminal as the terminal
will simply show you a graphic character based on the assumption that the
data is in the terminal configured code page.
qsrvbas@netscape.
net (Tom Liotta)
Sent by: To
midrange-l-bounce midrange-l@xxxxxxxxxxxx
s@xxxxxxxxxxxx cc
Subject
12/23/2004 07:26 RE: Another QtmmSendMail problem?
PM
Please respond to
Midrange Systems
Technical
Discussion
>> Due to all the possible knobs it may be easiest to simply
>> use debug and
>> look at the hex value of the address on their system
>> prior to the
>> conversion to confirm their @ is indeed x'7C'
This piece of Bruce's advice seems to be the one that gets missed the most.
Not necessarily for this problem; I wouldn't expect Brad to skip over it
lightly.
But confusions often seem to arise simply because we look at the
_characters_ that get displayed or printed and we forget (or never realize)
that the _important_ aspect is the actual bit pattern. The shape of the
displayed or printed character can change simply because we use a different
terminal or printer or emulator. The same data can look like a '@' on one
terminal and like a '§' on another just because the devices interpret the
hex values into different character sets.
Many of us learned years ago to use *CAT or *BCAT instead of the special
characters in CL programs for this same reason. The 'characters' would
change depending on the device settings regardless of the hex values of the
data.
I keep hoping we can work out a decent FAQ coverage of what this all means,
but nobody seems comfortable enough with all the concepts to get it right.
I know how I interpret things, but I'm totally unclear whether I'm very
close to understanding. So...
I'm going to ramble on about how I understand it all just to get something
going for the archives. Maybe some real experts will respond with
corrections, enhancements, clarifications or whatever; and a real FAQ will
start to emerge.
Character set -- The name for the things we see.
To me, a 'character set' represents a list of shapes of characters. Each
'character' in the set is what a device draws when it sees a particular bit
pattern from a given code page.
Code page -- The name for collating sequence of characters.
Different languages have different characters. German uses a lot of umlauts
for example. You can have the letter 'o' and an umlaut-'o' in German. If
you want to sort words that have those letters, how do you choose which has
the highest collating sequence? When converting to a different code page,
what happens to the collating sequence?
I've kind of pictured the problem of reconciling collating sequences
between different languages as partly being a computer design problem.
Somewhere in the hardware, there's a way to compare one character against
another and get a result that says "greater", "lesser" or "equal". But the
frequency of characters in words in a language also seems to need to be
addressed. This seems partly addressed by converting code pages from
computer to computer.
Because character comparisons happen so often, it should be addressed at a
low level. I'd imagine that code pages are designed to help minimize what
it takes to do comparisons of words and I would expect that conversions
between two code pages would take that into account. If I convert stuff
from Cyrillic to English, maybe I shouldn't expect exact bit-for-bit
matches. Maybe the difference would show up simply because computers that
normally operate on English data can be more efficient with a different
code page than computers that regularly operate on Cyrillic data.
So, the combination of code pages and character sets account for the
differences of collating sequences and visual representations. During code
page conversions, efficiencies are maybe maintained to some degree.
CCSID -- The name for a particular combination of code page with a
character set. For a given code page, a change in character set means it
needs to be a different CCSID.
Rather than keeping track of code pages and character sets separately, we
just use one symbol that stands for the two of them together, the Coded
Character Set IDentifier, CCSID.
Those are basically the vague, nebulous meanings I've come to use for those
three concepts. I'd love it if someone could say they're right, or correct
them where they're wrong. (Ideally, someone should also clear up how
Unicode fits in with CCSID since it almost seems as if Unicode sometimes
encompasses all of "CCSID" within almost a single CCSID. Or something like
that.)
Tom Liotta
midrange-l-request@xxxxxxxxxxxx wrote:
> 7. Re: Another QtmmSendMail problem? (Brad Stone)
>
>On Tue, 21 Dec 2004 12:49:56 -0600
>
>Things are still as they were... the funny thing is,
>SNDDST seems to send emails just fine when they use the @
>sign.
>
>When they use § and the email api, it works fine, shows up
>as @ in MIME file header, but § in a PF record that is used
>to log emails that are sent out.
>
>So to me, something is surely converting wrong in this
>particular case.
>
>They did some searching and found that the § and @
>characters are reversed in the different CCSIDs, though.
>
>I'm not sure where else to look.
>
> Bruce Vining <bvining@xxxxxxxxxx> wrote:
>>
>> A job CCSID of 870 and the resulting conversions to 500
>> would be the same
>> on both systems. What might be different though are the
>> code points being
>> generated by your keyboards (assuming the address is
>> being provided
>> interactively) and subsequently being processed in the
>> job so that the
>> actual inputs to the conversion are not the same.
>>
>> Due to all the possible knobs it may be easiest to simply
>> use debug and
>> look at the hex value of the address on their system
>> prior to the
>> conversion to confirm their @ is indeed x'7C' (and that
>> yours in test is
>> also). If debug isn't possible then we would need to
>> know the
>> configuration (CHRID and KBDTYPE) of their workstation
>> (and hope that this
>> does represent what the workstation really is sending,
>> which with all the
>> emulators available is always a big IF these days...);
>> the CCSID, DFTCCSID,
>> and CHRIDCTL of their job; and the CHRID value for their
>> *DSPF (or
>> *PNLGRP).
--
Tom Liotta
The PowerTech Group, Inc.
19426 68th Avenue South
Kent, WA 98032
Phone 253-872-7788 x313
Fax 253-872-7904
http://www.powertech.com
__________________________________________________________________
Switch to Netscape Internet Service.
As low as $9.95 a month -- Sign up today at
http://isp.netscape.com/register
Netscape. Just the Net You Need.
New! Netscape Toolbar for Internet Explorer
Search from anywhere on the Web and block those annoying pop-ups.
Download now at http://channels.netscape.com/ns/search/install.jsp
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
As an Amazon Associate we earn from qualifying purchases.
This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].
Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.
