Re: CPYTOIMPF and accented characters

[Justin Dearing <zippy1981@xxxxxxxxx>]

Mohammad,

You really need a hex editor when fighting encodings. There are two basic
facts to consider. What is the actual byte in that stream at that point in
the file, and what is the encoding the text editor is either reading from
the BOM or assuming.

It looks like CPYTOIMPF does not add a BOM to the begining of the file, but
its not terrible hard to do, but a little hacky.
http://www-01.ibm.com/support/docview.wss?uid=nas8N1011066

But get a hex editor plugin for notepad++. Thats the best way to debug
unicode issues.

BTW, if you want a text editor that can read EBCDIC, I recommend the one
built into the free and open source orthodox file manager farmanager. Its
developed by russians, but the english internationalization is great.

Justin Dearing

On Wed, Nov 2, 2016 at 11:39 AM Mohammad Tanveer <surgum@xxxxxxxxx> wrote:

> IBM I version 5770SS1 V7R2M0
>
> tried multiple combinations
> CCSID of the DB2 Column (tried with 37, 278, 1143)
> CCSID of the IFS File (278, 1251, 1143)
> CCSID of the job is 65535
>
>
> On Wed, Nov 2, 2016 at 8:23 AM, Mohammad Tanveer <surgum@xxxxxxxxx> wrote:
>
> > I tried FTP, SFTP and simple drag and drop of IFS file to PC.  same
> > behavior all the time.
> >
> > On Wed, Nov 2, 2016 at 6:52 AM, Rob Berendt <rob@xxxxxxxxx> wrote:
> >
> > > How are you "moving" the file?  Some methods are more friendly than
> others
> > > in preserving such characters.  FTP is generally recommended as the
> > > safest.  I suspect you're doing that because you mentioned ASC and BIN.
> > >
> > >
> > > Rob Berendt
> > > --
> > > IBM Certified System Administrator - IBM i 6.1
> > > Group Dekko
> > > Dept 1600
> > > Mail to:  2505 Dekko Drive
> > >           Garrett, IN 46738
> > > Ship to:  Dock 108
> > >           6928N 400E
> > >           Kendallville, IN 46755
> > > http://www.dekko.com
> > >
> > >
> > >
> > >
> > >
> > > From:   Mohammad Tanveer <surgum@xxxxxxxxx>
> > > To:     Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxx>
> > > Date:   11/02/2016 09:48 AM
> > > Subject:        Re: CPYTOIMPF and accented characters
> > > Sent by:        "MIDRANGE-L" <midrange-l-bounces@xxxxxxxxxxxx>
> > >
> > >
> > >
> > > I know that on IFS file has correct character code as I can see the data
> > > on
> > > AS/400 correctly.  Problem is only if I move the CSV file to PC accented
> > > character translation is getting lost somehow...
> > >
> > > On Wed, Nov 2, 2016 at 6:45 AM, Mohammad Tanveer <surgum@xxxxxxxxx>
> > > wrote:
> > >
> > > > I tried both.. in Notepad++ I see special codes in front of almost all
> > > > SWE/DEN characters when I switch the encoding to UTF-8 but not for
> > > accented
> > > > characters that's why I am thinking for some reason accented
> characters
> > > are
> > > > not converted properly or translation is not happening.
> > > >
> > > > On Wed, Nov 2, 2016 at 6:34 AM, Luis Rodriguez <luisro58@xxxxxxxxx>
> > > wrote:
> > > > > Are you using the "standard" Windows Notepad? What happens if yu open
> > > that
> > > > > file using an app like Notepad++ or Word? It could be a problem in
> how
> > > > > Notepad is opening the file...
> > > > >
> > > > > HTH,
> > > > >
> > > > > Luis
> > > > >
> > > > >
> > > > > On Wed, Nov 2, 2016 at 9:29 AM Mohammad Tanveer <surgum@xxxxxxxxx>
> > > wrote:
> > > > > > First I tried CCSID 277 it didn't work then I tried 278 and then a
> > > > > number
> > > > > > of others..  nothing works... as soon as I download the CSV to PC
> > > from
> > > > > IFS
> > > > > > I don't see accented characters appearing in notepad.
> > > > > >
> > > > > > Regards
> > > > > > .
> > > > > >
> > > > > > On Wed, Nov 2, 2016 at 6:05 AM, Luis Rodriguez <luisro58@xxxxxxxxx
> >
> > > > > wrote:
> > > > > > > Are you sure about the CCSID? A quick Google search gave me 277
> for
> > > > > > > Danish...
> > > > > > > Regards,
> > > > > > >
> > > > > > > Luis
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Nov 2, 2016 at 9:00 AM Mohammad Tanveer <
> surgum@xxxxxxxxx>
> > > > > > wrote:
> > > > > > > > After creating a CSV file using CPYTOIMPF and CCSID 278 or
> > > *PCASCII.
> > > > > > > > It looks like accented characters (Danish characters like A
> with
> > > > > circle
> > > > > > > on
> > > > > > > > top *Å*) do show up in 5250 emulation when I use EDTF on IFS
> file
> > > > > but
> > > > > > > they
> > > > > > > > don't show up when I copy the file to PC and open it in text
> > > editor
> > > > > > with
> > > > > > > > UTF-8 encoding.  It shows as $ sign.  Most of other Danish
> > > > > characters
> > > > > > > show
> > > > > > > > up correctly in text file. I tried copying the file to PC in
> > > binary
> > > > > > > format
> > > > > > > > and ASCII but no luck.
> > > > > > > >
> > > > > > > > Is it a bug or I cannot convert Accented characters using
> > > CPYTOIMPF
> > > > > for
> > > > > > > > PC?  Other CCSID 278 characters are converting fine.
> > > > > > > >
> > > > > > > > Following output is in EDTF after changing the emulation
> setting
> > > to
> > > > > > host
> > > > > > > > code 037.  Line 5 shows $ instead of *Å*
> > > > > > > >
> > > > > > > >  Browse :
> > > > > > > > /swe/SL600O.csv
> > > > > > > >
> > > > > > > >  Record :      18   of    1905 by  18
> Column
> > > :
> > > > > > 1
> > > > > > > > 225 by 131
> > > > > > > >  Control
> > > > > > > > :
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ....+....1....+....2....+....3....+....4....+....5....+....
> > > > > > > 6....+....7....+....8....+....9....+....0....+....1....+....2..
> > > > > > > > 1908932044     ,460000643478,MALENE HESSELDAL
> > > > > > > > ,GUNDERSEN                     ,JERICHAUSGADE 3  ST. TV
> > > > > > > > 0311732102     ,460000601288,KARINA
> > > > > > > > ,AKSTRUP                       ,LILLEVANG 145
> > > > > > > > 2805742472     ,460000741415,MAJ
> > > > > > > > ,ANDKJÆR                       ,SOLSIKKEVÆNGET 22
> > > > > > > > 9203064069 <(920)%20306-4069>     ,460000690554,EMILY
> > > > > > > > ,MIKKELSEN                     ,ROTTARPVEJ 32
> > > > > > > > 300368-0028    ,460000015491,GITTE
> > > > > > > > ,PEDERSEN                      *,M$GEVEJ 44   *
> > > > > > > > 110871-1228    ,460000503509,LOTTE
> > > > > > > > ,ERRI                          ,KÆRBØLVEJ 52 A
> > > > > > > > 0304712252     ,460000695645,METTE
> > > > > > > > ,AUNBIRK                       ,HOLMELINSVEJ 3
> > > > > > > > 1008762682     ,460000713579,NICOLE
> > > > > > > > ,FOGET                         ,SOLVANGSVEJ 5B
> > > > > > > > 0508691254     ,460000705806,FREYA CHALOTTE
> > > > > > > > ,HOLM                          ,ØSTERGADE 9
> > > > > > > > 0411742024     ,460000707761,PIA VIBEKE
> > > > > > > > ,HAUSTED                       ,KAMPESMOSE 23
> > > > > > > > 2203781132 <(220)%20378-1132>     ,460000673722,MARIE
> > > > > > > > ,KJÆR                          ,TATERBAKKEVEJ 3
> > > > > > > > 240259-2020    ,460000012598,LONE
> > > > > > > > ,MARQUARDSEN                   ,FORSYTHIAVEJ 38
> > > > > > > > 1204762526     ,460000699274,MALENE DALGAARD
> > > > > > > > ,KROGH                         ,CENTRALG$RDSVEJ 121
> > > > > > > > 2103572528 <(210)%20357-2528>     ,460000518039,DORRIT
> > > > > > > > ,KRISTENSEN                    ,SKJOLDAGERVEJ 92  2 TH
> > > > > > > > 0507380840     ,460000539850,KARIN
> > > > > > > > ,SIMONSEN                      ,STRANDGADE 28 N HOU
> > > > > > > > 1208520144     ,460000639066,ANNIE
> > > > > > > > ,BEJART                        ,PRÆSTEGAARDSVEJ 5
> > > > > > > > 2403741356 <(240)%20374-1356>     ,460000729054,KAREN
> > > > > > > > ,EBBESEN                       ,BELLAHOJVEJ 14
> > > > > > > > 1903802622     ,460000675467,VINNI MARIE         ,S.
> > > > > > > > MADSEN                     ,ØSTKÆRVEJ 27
> > > > > > > > 2604712478 <(260)%20471-2478>     ,460000701355,METTE
> > > > > > > > ,OLESEN                        ,BYAGEREN 25
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > This is after switching the emulation to Host Code Page to 1143
> > > you
> > > > > can
> > > > > > > see
> > > > > > > > line 5 show correctly in EDT.. and I can copy paste this data
> in
> > > > > > notepad
> > > > > > > as
> > > > > > > > you can see PC shows it correctly.  Which means PC can
> understand
> > > > > this
> > > > > > > > character in notepad... so why CPYTOIMPF is not able to convert
> > > it
> > > > > to
> > > > > > > plain
> > > > > > > > PC text using CCSID?
> > > > > > > >
> > > > > > > >  Browse :
> > > > > > > > /swe/SL600O.csv
> > > > > > > >
> > > > > > > >  Record :      18   of    1905 by  18
> Column
> > > :
> > > > > > 1
> > > > > > > > 225 by 131
> > > > > > > >  Control
> > > > > > > > :
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ....+....1....+....2....+....3....+....4....+....5....+....
> > > > > > > 6....+....7....+....8....+....9....+....0....+....1....+....
> > > > > 2....+....3.
> > > > > > > > 1908932044     ,460000643478,MALENE HESSELDAL
> > > > > > > > ,GUNDERSEN                     ,JERICHAUSGADE 3  ST.
> > > > > > > > TV
> > > > > > > > 0311732102     ,460000601288,KARINA
> > > > > > > > ,AKSTRUP                       ,LILLEVANG
> > > > > > > > 145
> > > > > > > > 2805742472     ,460000741415,MAJ
> > > > > > > > ,ANDKJÆR                       ,SOLSIKKEVÆNGET
> > > > > > > > 22
> > > > > > > > 9203064069 <(920)%20306-4069>     ,460000690554,EMILY
> > > > > > > > ,MIKKELSEN                     ,ROTTARPVEJ
> > > > > > > > 32
> > > > > > > > 300368-0028    ,460000015491,GITTE
> > > > > > > > ,PEDERSEN                      *,MÅGEVEJ 44   *
> > > > > > > >
> > > > > > > > 110871-1228    ,460000503509,LOTTE
> > > > > > > > ,ERRI                          ,KÆRBØLVEJ 52
> > > > > > > > A
> > > > > > > > 0304712252     ,460000695645,METTE
> > > > > > > > ,AUNBIRK                       ,HOLMELINSVEJ
> > > > > > > > 3
> > > > > > > > 1008762682     ,460000713579,NICOLE
> > > > > > > > ,FOGET                         ,SOLVANGSVEJ
> > > > > > > > 5B
> > > > > > > > 0508691254     ,460000705806,FREYA CHALOTTE
> > > > > > > > ,HOLM                          ,ØSTERGADE
> > > > > > > > 9
> > > > > > > > 0411742024     ,460000707761,PIA VIBEKE
> > > > > > > > ,HAUSTED                       ,KAMPESMOSE
> > > > > > > > 23
> > > > > > > > 2203781132 <(220)%20378-1132>     ,460000673722,MARIE
> > > > > > > > ,KJÆR                          ,TATERBAKKEVEJ
> > > > > > > > 3
> > > > > > > > 240259-2020    ,460000012598,LONE
> > > > > > > > ,MARQUARDSEN                   ,FORSYTHIAVEJ
> > > > > > > > 38
> > > > > > > > 1204762526     ,460000699274,MALENE DALGAARD
> > > > > > > > ,KROGH                         ,CENTRALGÅRDSVEJ
> > > > > > > > 121
> > > > > > > > 2103572528 <(210)%20357-2528>     ,460000518039,DORRIT
> > > > > > > > ,KRISTENSEN                    ,SKJOLDAGERVEJ 92  2
> > > > > > > > TH
> > > > > > > > 0507380840     ,460000539850,KARIN
> > > > > > > > ,SIMONSEN                      ,STRANDGADE 28 N
> > > > > > > > HOU
> > > > > > > > 1208520144     ,460000639066,ANNIE
> > > > > > > > ,BEJART                        ,PRÆSTEGAARDSVEJ
> > > > > > > > 5
> > > > > > > > 2403741356 <(240)%20374-1356>     ,460000729054,KAREN
> > > > > > > > ,EBBESEN                       ,BELLAHOJVEJ
> > > > > > > > 14
> > > > > > > > 1903802622     ,460000675467,VINNI MARIE         ,S.
> > > > > > > > MADSEN                     ,ØSTKÆRVEJ
> > > > > > > > 27
> > > > > > > > 2604712478 <(260)%20471-2478>     ,460000701355,METTE
> > > > > > > > ,OLESEN                        ,BYAGEREN
> > > > > > > > 25
> > > > > > > >
> > > > > > > >
> > > > > > > >  F3=Exit   F10=Display Hex   F12=Cancel   F15=Services
> F16=Repeat
> > > > > > find
> > > > > > > > F19=Left   F20=Right
> > > > > > > >
> > > > > > > > Any idea/help ?
> > > > > > > > --
> > > > > > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L)
> > > > > mailing
> > > > > > > list
> > > > > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > > > > To subscribe, unsubscribe, or change list options,
> > > > > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > > > > Before posting, please take a moment to review the archives
> > > > > > > > at http://archive.midrange.com/midrange-l.
> > > > > > > >
> > > > > > > > Please contact support@xxxxxxxxxxxx for any subscription
> related
> > > > > > > > questions.
> > > > > > > --
> > > > > > >
> > > > > > > Luis Rodriguez
> > > > > > > --
> > > > > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L)
> > > mailing
> > > > > > list
> > > > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > > > To subscribe, unsubscribe, or change list options,
> > > > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > > > Before posting, please take a moment to review the archives
> > > > > > > at http://archive.midrange.com/midrange-l.
> > > > > > >
> > > > > > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > > > > > questions.
> > > > > > --
> > > > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L)
> > > mailing
> > > > > list
> > > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > > To subscribe, unsubscribe, or change list options,
> > > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > > Before posting, please take a moment to review the archives
> > > > > > at http://archive.midrange.com/midrange-l.
> > > > > >
> > > > > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > > > > questions.
> > > > > --
> > > > >
> > > > > Luis Rodriguez
> > > > > --
> > > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L)
> mailing
> > > > > list
> > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > To subscribe, unsubscribe, or change list options,
> > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > Before posting, please take a moment to review the archives
> > > > > at http://archive.midrange.com/midrange-l.
> > > > >
> > > > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > > > questions.
> > > --
> > > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
> > > list
> > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > To subscribe, unsubscribe, or change list options,
> > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > Before posting, please take a moment to review the archives
> > > at http://archive.midrange.com/midrange-l.
> > >
> > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > questions.
> > >
> > >
> > > --
> > > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
> > > list
> > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > To subscribe, unsubscribe, or change list options,
> > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > Before posting, please take a moment to review the archives
> > > at http://archive.midrange.com/midrange-l.
> > >
> > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > questions.
> --
> This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
> To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> or email: MIDRANGE-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/midrange-l.
>
> Please contact support@xxxxxxxxxxxx for any subscription related
> questions.