Re: [IBMiOSS] 65535 CCSID

Ugh, dealing with other people's data always sucks... :(


----- Original message -----
From: Jeremy Meyer <JMeyer@xxxxxxxxxxxxx>
Sent by: "OpenSource" <opensource-bounces@xxxxxxxxxxxx>
To: IBMi Open Source Roundtable <opensource@xxxxxxxxxxxx>
Cc:
Subject: Re: [IBMiOSS] 65535 CCSID
Date: Tue, Dec 19, 2017 5:08 PM

Amen on that last option. Unfortunately it is 3rd party and I have no
control over that.

-----Original Message-----
From: OpenSource [[1]mailto:opensource-bounces@xxxxxxxxxxxx] On Behalf
Of Kevin Adler
Sent: Tuesday, December 19, 2017 5:06 PM
To: opensource@xxxxxxxxxxxx
Cc: opensource@xxxxxxxxxxxx
Subject: Re: [IBMiOSS] 65535 CCSID

The e-mail below is from an external source. Please do not open
attachments or click links from an unknown or suspicious origin.

Job CCSID doesn't matter because your data is not marked in job
ccsid, it
is marked 65535 which is hex/binary. This works mostly fine as
character
data for ILE applications because everything is EBCDIC and as long as
you
don't start mixing EBCDIC code pages, you're usually ok. Where it
gets
tricky is when you get outside of ILE. The version of Python shipped
in
OPS runs in PASE, which is an ASCII-based environment and so in order
for
Python to understand the data as a string of characters the data must
be
converted to ASCII/Unicode. There's no way to convert 65535 to ASCII,
Unicode, or any other encoding because it is binary and CLI (the API
ibm_db uses to communicate to the database) does not have a way to
tell it
what CCSID to use instead.

You have three options:

- keep casting the data in the query
- use Buck's suggestion and create a view
- or my suggestion: actually set the proper CCSID on your fields

----- Original message -----
From: Jeremy Meyer <JMeyer@xxxxxxxxxxxxx>
Sent by: "OpenSource" <opensource-bounces@xxxxxxxxxxxx>
To: IBMi Open Source Roundtable <opensource@xxxxxxxxxxxx>
Cc:
Subject: Re: [IBMiOSS] 65535 CCSID
Date: Tue, Dec 19, 2017 4:15 PM

Python 3 locally on the IBM i. Anytime I query a 65535 field I get
what
appears to be hex. I notice when the python job spawns it is CCSID
37
instead of 65535 our default. Interactively I tried issuing a
chgjob to
ccsid 37 via qsys2.qcmdexc and when I query and print the result
list
the 65535 fields look the same.

-----Original Message-----
From: OpenSource [[1][2]mailto:opensource-bounces@xxxxxxxxxxxx] On
Behalf
Of John Yeung
Sent: Tuesday, December 19, 2017 3:57 PM
To: IBMi Open Source Roundtable <opensource@xxxxxxxxxxxx>
Subject: Re: [IBMiOSS] 65535 CCSID

The e-mail below is from an external source. Please do not open
attachments or click links from an unknown or suspicious origin.

On Tue, Dec 19, 2017 at 2:57 PM, Jeremy Meyer
<JMeyer@xxxxxxxxxxxxx>
wrote:
> I have been using Python and ibm_db_dbi for automating some
tasks. The
problem I have is our CMS system stores its data in 65535 CCSID
fields.
Is there an easier way to translate that than to cast them to 37 on
each
field?

Dealing with encoding issues is one thing which I think is
significantly
easier in Python than in SQL.

Which Python are you using, and on which platform?

If you just query the data normally, without explicitly handling
CCSID,
what do you get?

John Y.
--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To
post a message email: OpenSource@xxxxxxxxxxxx To subscribe,
unsubscribe,
or change list options,
visit:

[2][3]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
or email: OpenSource-request@xxxxxxxxxxxx Before posting, please
take a
moment to review the archives at

[3][4]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=.

NOTICE: This electronic mail message and any files transmitted with
it
are intended
exclusively for the individual or entity to which it is addressed.
The
message,
together with any attachment, may contain confidential and/or
privileged
information.
Any unauthorized review, use, printing, saving, copying, disclosure
or
distribution
is strictly prohibited. If you have received this message in error,
please
immediately advise the sender by reply email and delete all copies.

--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit:

[4][5]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at

[5][6]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=.

References

Visible links
1. [7]mailto:opensource-bounces@xxxxxxxxxxxx
2.
[8]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
3.
[9]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=
4.
[10]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
5.
[11]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=
--
This is the IBMi Open Source Roundtable (OpenSource) mailing list To
post a message email: OpenSource@xxxxxxxxxxxx To subscribe, unsubscribe,
or change list options,
visit:
[12]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=EpAMkgyPmFMBF3j1XbXKHdCEBThJfiAEetqlvRtFTQI&e=
or email: OpenSource-request@xxxxxxxxxxxx Before posting, please take a
moment to review the archives at
[13]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=xORXIwE7LNW5klfVc2nEofLNBPh1bfsNQvZWN3myFW8&e=.

NOTICE: This electronic mail message and any files transmitted with it
are intended
exclusively for the individual or entity to which it is addressed. The
message,
together with any attachment, may contain confidential and/or privileged
information.
Any unauthorized review, use, printing, saving, copying, disclosure or
distribution
is strictly prohibited. If you have received this message in error,
please
immediately advise the sender by reply email and delete all copies.

--
This is the IBMi Open Source Roundtable (OpenSource) mailing list
To post a message email: OpenSource@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit:
[14]https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=EpAMkgyPmFMBF3j1XbXKHdCEBThJfiAEetqlvRtFTQI&e=
or email: OpenSource-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at
[15]https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=xORXIwE7LNW5klfVc2nEofLNBPh1bfsNQvZWN3myFW8&e=.




References

Visible links
1. mailto:opensource-bounces@xxxxxxxxxxxx
2. mailto:opensource-bounces@xxxxxxxxxxxx
3. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
4. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=
5. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
6. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=
7. mailto:opensource-bounces@xxxxxxxxxxxx
8. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
9. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=
10. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=ZnuxGhaq_GaaRUEopxGOJONk25ANj99WSeRhAxgIYNE&e=
11. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=yrwzLhGJ_LvvrUEetCL1vKw7uJD7W9XUzl3mUrOImew&s=7T8p4RZv_XlnmDYrEiWOeubOaFVPgWHVasye1IAqkp4&e=
12. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=EpAMkgyPmFMBF3j1XbXKHdCEBThJfiAEetqlvRtFTQI&e=
13. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=xORXIwE7LNW5klfVc2nEofLNBPh1bfsNQvZWN3myFW8&e=
14. https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.midrange.com_mailman_listinfo_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=EpAMkgyPmFMBF3j1XbXKHdCEBThJfiAEetqlvRtFTQI&e=
15. https://urldefense.proofpoint.com/v2/url?u=https-3A__archive.midrange.com_opensource&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2aXRESXNvBGuLbIQEybCPNLc7OK0UqjwsZsoCq7Jz6o&m=Ves8UJ4Ri_hqrqlXeOBNH0jHaLONFvNMKGiqNbnL8RA&s=xORXIwE7LNW5klfVc2nEofLNBPh1bfsNQvZWN3myFW8&e=