Re: using RDP to edit UTF-8 documents on the IFS

On 7/10/2013 6:52 AM, whatt sson wrote:

Follow up.

I changed the "default transfer type for .xml docs" from text to binary.
Now the file has been translated correctly - on the IFS - to UTF-8.

For the archives, this setting is found in
Windows > Preferences > Remote Systems > Files

I think this is strange behaviour. My IBM i CCSID is 37, my PC is US
English. The test XML file is CCSID(1208). If I open it in RDP 8.5.1
and edit it with Latin 1 or Latin 2 characters in it, and save it back
to the IFS, all the UTF-8 characters are preserved. I would have
thought that the translation would be preserved if the file were
considered text rather than binary.

After running a few more tests, I found that when I use the Web
Perspective the XML files seem to be translated properly. I used
Project Explorer and created a new project. Then I used Import... to
bring the XML files in, edit, etc. Once I was happy with those files I
used Export... to push the changes back to the IFS.

Now, instead of more "live parsing" which is really neat but not very
helpful if the cursor keeps hanging and hanging, it would be nice for IBM
to squash some bugs (like selecting all text sometimes which is in there
for more than 10 years already), or a nice feature (it's RD... for "i"
therefore you pay 800 bucks) it would read the CCSID setting on the IFS.
But that isn;t a nice feature to show off on some demo i suppose.

I myself am an RPG programmer. I don't spend much time with Java or web
things. I write procedures for service programs and often encapsulate
them in SQL stored procedures so our web group can interact with our
data through a clearly defined interface. There are some things I wish
for that would make my RPG programming easier, and I finally created an
RFE for that.
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=31305
This isn't my birthday party where I get every present I ask for :-)
but it is a way to let the RDi developers know what the community wants,
and it helps them justify spending time on the features we actually want
as opposed to features the sales guys tell them to create.

I strongly suggest that you create an RFE for this CCSID issue. Post
the URL here on this list and I bet many will vote for it. Because an
IDE intended for use with IBM i really should be CCSID-aware without
having to tinker with the setup.

End of ranting. Sorry about that,.

Use the RFE process to ask for new functionality, and use the PMR
process to report a bug. For instance, I am not familiar with a bug in
Select All. Maybe IBM isn't either... It's better to take action, I
think, than to be frustrated. It is clear that RDi is IBM's intended
direction; unlike almost every other IBM software, we have a real chance
to help steer it in the direction we want it to go.

On Wed, Jul 10, 2013 at 12:19 PM, whatt sson <whattssonn@xxxxxxxxx> wrote:

Buck,
Thanks for testing.

I create a file with EDTF, or from RDi.
I change the CCSID to 1208 with option 13.
Etc

I didn't see the "properties" window where you can change the encoding of
the file.
When i go to RSE and right-click on the file i get a properties window
only showing some attributes such as encoding but i can't change it.
However, when i choose File->Properties the i get the full window where i
ca change the encoding. But i seem to get this window not always.
Sometimes it's disabled, sometimes a just get the small properties window.

Very strange. I always see the exact same window whether I use File >
Properties or right click > Properties. In fact, the same shortcut key
is used for both (Alt + Enter). Grasping at straws here: is the IFS
file read-only? Is the authority correct? That's using RSE and the IBM
i Editing perspective. I see a different window when using the Web
perspective, but it doesn't change whether I use File > Properties or
right click > Properties.

Pff

However, i'm trying all things but it keep saying to me "just use another
tool for this".
Howver, 800 bucks a year is a lot of money for something you can't even
use to edit xml docs.

I'm not a web guy, so I don't know what reason I have to edit XML
documents. That's not saying there is no good reason to do so, it's
saying I myself am ignorant. Normally, I don't alter XML; I look at it
to make sure I generated the right stuff, or that a vendor sent what I
expected. For me, then, the Design tab isn't at all useful. I can't
search and I can't see the details the way my brain wants to see them.
I tend to use Notepad++ for this because it's integrated into Windows
Explorer, but maybe I should try RDi more so I can report issues (if I
find them with my limited needs).

The price. Sigh. WDSC 7 was included at no extra charge with the
compilers, and it had web perspectives included. Apparently, too many
people complained that it was bloated. So IBM stripped out a lot of the
web stuff for 8.5 in order to make them happy. But then IBM also broke
out the IDE as an additional charge product, with one price for the
lighter footprint (RDP) and another price for the heavier footprint
(RBD). That's where we are now; I guess if we need to do webby stuff,
we should be buying the bigger IDE.
--buck

The locally cached file is encoded ok in UTF-8, after i save the document.
For example ř is translated to C5 99.
But the file on the IFS is not, although it's CCSID is 1208.

This also happens when i create the file from RDP.
Another file with another name, no avail, still having the same problems.

Clearing the cache does not help.
I clear the cache regularly because i had some problems before that RDP
wouldn't load the - NEWER - remote source member but instead loaded the old
source in the cache.


So nothing helps, local file is encoded ok, but remote is not, no matter
what i do.
I can't even reproduce the situation from yesterday that i create a new
file (with EDTF file or whatever) and let it save explicitly to UTF-8.
Local file is ok but remote file is not.
The ř is translated to C5 99.
But the file on the IFS is not, although it's CCSID is 1208.
It's translated to a question mark (3F).
And the other characters (ö and ä which have a representation in Cp1252)
are translated to EF BF BD which is the UTF-8 replacement character.
So it seems after translating correctly to UTF-8 locally, when it uploads
the file it then translates to characters in the 437 codepage but with
UTF-8 representation. So if a character does not exist it becomes a
question mark or replecament character.
Complicate all this... yes.. but thats the situation.

But now comes a surprise. Whe i create a file on the IFS from RDP 0 but
WITHOUT .xml extension, then it does encode correctly in UTF-8 on the IFS.

So maybe there some config setting associated with .xml but all settings
specify UTF-8.

I give up.

I use another free xml tool to create them on the PC and then upload
binary to the IFS.
That will work.

But what a shitty product, RDP, really.




I did some more testing but it's really annoying.





On Tue, Jul 9, 2013 at 7:10 PM, Buck Calabro <kc2hiz@xxxxxxxxx> wrote:

Replies inline.

On 7/9/2013 11:49 AM, whatt sson wrote:

I'm trying to edit an xml document on the ifs using RDP (8.5) but it

seems

to have some trouble converting to/from UTF-8.

I have created the file on the IFS with CCSID 1208.

How? I used EDTF to create a file with a blank line, save it, exit
EDTF, re-run EDTF, F15 option 3 CCSID 1208, then paste <?xml
version="1.0" encoding="utf-8" ?> into the first line and saved it.
WRKLNK shows CCSID 1208.

Then i edit the file in RDP and put some text in it with latin-1
characters, such as ä.
Then i save the file but RDP converts to 1252 instead of 1208.

I opened my test.xml in RDP 8.5.1, changed to Source view and added
this: <test>Hélène says bonjour.</test> I am in the US, with a US
keyboard, so I typed the accented letters by holding down the Alt key
and pressing the numbers on the number pad: Alt+130 for é and Alt+138
for è. Then I saved the file. No warnings in RDP.

WRKLNK still shows CCSID 1208. EDTF shows a warning:
CPF9897 1 INVALID CHARCTERS FOR CCSID 01208 ENCOUNTERED IN DISPLAY
RECORD 3. IF THE RECORD IS CHANGED, THEY WILL BE REPLACED WITH
BLANKS.
Cause . . . . . : No additional online help information is available.
Exit EDTF.

Back to RDP, in RSE, right click the XML file -> Properties. I see that
file encoding is set to Default(Cp1252). I set it to Other UTF-8.
Apply, OK. I see funny characters in my RDP editor where my accents
used to be. I fix them and save.

WRKLNK shows 1208 and EDTF shows the accented characters properly. In
hex they are:
<test>
3C746573743E
Hélène
48C3A96CC3A86E65

When i put a latin-2 character in the text such as ř and then save to

text

RDP complains and asks to convert it to UTF-8 (which i want it to do).

When I paste that character into my RDP editor and save, I see no
warnings in RDP. WRKLNK shows 1208, EDTF shows a non-displayable
character on my US English 5250 (CCSID 37) session. In hex it is:

<test2>
3C74657374323E
ř
C599
</test2>
3C2F74657374323E

How can i set RDP to always save in UTF-8 format.
Or even better, to respect to CCSID attribute of the IFS file.

I did not see a preference for this in regular RDP 8.5.1. Maybe it's
because RDP is not the web developer version - I think that is Rational
Business Developer? Just a guess.

In the general settings i have set the default CCSID to 1208 for XML

files

(*.xml).
But this does not have any effect it seems.

General -> Appearance -> Content types. My XML type was set to UTF-8
(which is the default).

XML -> XML Files has what to do when /creating/ an XML file. Mine was
set to UTF-8 (which is the default).

RDP also behaves rather quirky with this.
When i explicitly choose to save to UTF-8 (when asked for it) it

converts

incorrectly to the existing file (CCSID 1208).
When i create a new file with CCSID 1208 and save the same text then it
does convert to UTF-8.

RDP (or RDi) seems to be completely broken on this.

I'm not sure what is different between our setups. In my case, I must
remember to change the properties to UTF-8. That's clumsy but I
wouldn't call it completely broken. I hope this was some help; if only
to show where RDP ends and RBD begins.
--buck