Re: CGIDEV2 file upload data conversion

Thanks Scott.

Unfortunately, we have a lot of production development working through this http setup. Changing it may potentially mean a lot of rework/testing etc.

I was thinking I would create another web instance, configured for binary, and direct the file uploads through this new instance. I'll give that a go.



-----Original Message-----
From: WEB400 [mailto:web400-bounces@xxxxxxxxxxxx] On Behalf Of Scott Klement
Sent: 02 December 2016 05:48
To: Web Enabling the IBM i (AS/400 and iSeries) <web400@xxxxxxxxxxxx>
Subject: Re: [WEB400] CGIDEV2 file upload data conversion

Sean,

I, too, find the various CgiConvMode settings unintuitive. I'm always amazed when the HTTP server does something that, to me, makes no sense, like, encoding non-HTML data with URL encoding or sometimes escaping
into HTML entities, seemingly at random. I suspect that the internal
code in the HTTP server is fairly convoluted for the purposes of backward compatibilty? Because it doesn't make much sense some of the things it does.

One way to work around the problem is to use "CgiConvMode binary" (which think is the same as %%BINARY/BINARY%%, but I try to avoid the old %% types unless the newer ones don't work -- but for binary, I'm at a loss as to what they could do differently, so probably 'binary' and "%%binary/binary%%" are the same... but I digress..)

Yes, this might break things because now the HTTP server is no longer translating to EBCDIC when it sends the data to your program. So you'll have to modify whatever utilities you're using to understand the data coming in UTF-8.

Once you fix that up to understand the native web encodings, however, you don't have to worry about the HTTP server doing weird things ;-)

Not sure if anyone has added that type of support to CGIDEV2. But, it is an open source tool, so you can add that and contribute it back to the project. However, for what you're doing, I wonder how CGIDEV2 is really helping you?

I can't wait until everyone has moved up to IBM i 7.2 finally (yes, it'll be a few more years). There, RPG natively understands UTF-8, so we can eliminate all this nasty EBCDIC conversion that has caused so many problems!

On 12/1/2016 9:03 AM, McGovern, Sean wrote:

I'm finding this very confusing. I seem to still have a problem uploading files via the web using CGIDEV2.

I create a txt file using Notepad, containing text ABC. I transfer this using NetServer to the IFS. In the IFS, if I display the hex values, I get 41 42 43, as expected (in CCSID 819).

If I then upload this same txt file via my web process, I get the correct result in my database table. The hex values are 41 42 43.

However, the data appears to be converted if I upload a XLSX file, for example.

I create a dummy XLSX file. I transfer this using NetServer to the IFS. In the IFS, the first 20 hex characters show...

504B0304 14000600 08000000 210062EE 9D685E01

...I assume this is correct. I can open the Excel from the IFS without issue.

If I then upload this same XLSX file via my web process, I don't get the same hex values in my database table. I get...

504B0304 1400060 008000000 210062C3 AEC29D68

If I debug service program CGISRVPGM2, at the point where it saves the file data to the temporary IFS file, the first hex values (program variable outds) are...

504B0304 1400060 008000000 210062C3 AEC29D68

If I understand correctly, this data is not being manipulated/converted by CGIDEV2, it is just the data provided through the web.

Somehow, the data is being converted/corrupted. I assume this is due to a httpd.conf setting? I have...

CGIConvMode %%EBCDIC/EBCDIC%%
DefaultFsCCSID 937
AddCharset UTF-8 .html .pgm
DefaultNetCCSID 1208

Tried %%BINARY/EBCDIC%% but no joy. Gave different hex values, but not correct. Tried %%BINARY/BINARY%%, but my web application doesn't work at all with that setting.

Ideas?

-----Original Message-----
From: McGovern, Sean
Sent: 01 December 2016 08:44
To: Web Enabling the IBM i (AS/400 and iSeries) <web400@xxxxxxxxxxxx>
Subject: RE: [WEB400] CGIDEV2 file upload data conversion

I'm using this technique to return a BLOB to a browser...

http://iprodeveloper.com/application-development/return-blob-browser

As I'm allowing any file type to be uploaded, when the user downloads, I'm specifying...

Content-Type: application/octet-stream

However, even if I specify the correct content type, I still seem to have an issue with PDF, DOC, XLS etc., the downloaded file is corrupt and will not open.


-----Original Message-----
From: WEB400 [mailto:web400-bounces@xxxxxxxxxxxx] On Behalf Of Bradley
Stone
Sent: 30 November 2016 18:33
To: Web Enabling the IBM i (AS/400 and iSeries) <web400@xxxxxxxxxxxx>
Subject: Re: [WEB400] CGIDEV2 file upload data conversion

I believe you need to set the content type for each type of file that will be downloaded in your Apache Config.

The content-type you're getting now is a default Apache uses when it doesn't know the content type.

Here's a link:
https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_si
te/Configuring_server_MIME_types

This may help. I only did a quick scan. I'm not sure it has a list of extensions and their proper mime types, but you should be able to find that using Google.

Brad
www.bvstools.com

On Wed, Nov 30, 2016 at 9:41 AM, McGovern, Sean
<sean.mcgovern@xxxxxxxxxxxxx

wrote:

Thanks.

I've now stopped the base64 encoding on the upload. The results look good.
I seem to be getting the correct hex values in the DB2 table.

I'm giving the users the ability to upload any type of file. When the
users wish to view the uploaded files, files are downloaded, specifying...

Content-Type: application/octet-stream
Content-Disposition: attachment; filename="xxx";

Working fine for XML, TXT, JPEG files etc., but have problems with
PDF, DOC, XLS.

When I use Notepad to compare the original uploaded file to the
downloaded file, no differences. Except, if I perform SaveAs in
Notepad, I see the original file has ANSI encoding specified, whereas
the uploaded/downloaded file has Unicode encoding specified.

Any ideas?




-----Original Message-----
From: WEB400 [mailto:web400-bounces@xxxxxxxxxxxx] On Behalf Of
Bradley Stone
Sent: 29 November 2016 18:13
To: Web Enabling the IBM i (AS/400 and iSeries) <web400@xxxxxxxxxxxx>
Subject: Re: [WEB400] CGIDEV2 file upload data conversion

Hi, Sean.

It looks like the data that "worked" was plain ascii and the other
was base
64 encoded (but still ASCII).

There should be no conversion of the data from ASCII to EBCDIC in
your case since the content type is application/[whatever].

I'd double check the headers on one that doesn't base64 encode the data.
And double check what you're using for the upload option to see if
one
base64 encoded and one didn't.

Or, you may need to read the headers and if it is base64 encoded,
decode it before storing it in your IFS.

Brad
www.bvstools.com.

On Tue, Nov 29, 2016 at 9:55 AM, McGovern, Sean <
sean.mcgovern@xxxxxxxxxxxxx

wrote:

About a month ago, I installed the latest version of CGIDEV2, and
developed functionality that allows users to upload file attachments
to DB2 table via the web, with the ability to download from the DB2
table. In this way, files can be attached to 'orders' and users can
view these files when they wish. All was working well (even made a
video demonstrating the process to users).

Now paperwork has been approved for me to promote this development
to a QA environment. I tested the functionality again and hit an
issue that I've been unable to resolve. I'm unaware of any changes
that have been made in the past month.

Now, when the file attachment data reaches CGISRVPGM2, it appears to
have been converted and I don't know why.

I have specified...

CGIConvMode %%EBCDIC/EBCDIC%%
DefaultFsCCSID 937
DefaultNetCCSID 1208

If, for example, I attempt to upload a PDF document, I can see in
browser debug, that the post contains...

Content-Disposition: form-data; name="Filedata"; filename="Imp 4991.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: base64

JVBERi0xLjYN...

The above base64 data translates to "%PDF-1.6", which is what I see
when I open the PDF document in Notepad. I'm unsure whether it
should be base64 encoded, but I believe this was working this way previously.

When I did this a month ago, I got the correct hex values in the DB2
table...

255044462D312E31
%PDF-1.6
(CCSID 819)

Now, I get...

4A56424552693078

Debugging, I see this is the data (in hex) that is read and written
to the temporary filename in CGISRVPGM2. CGISRVPGM2 doesn't appear
to be converting this data.

I don't know why I'm now getting 4A564...whereas previously I got
2550444...??

Thanks for any help.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list To post a message email: WEB400@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at http://archive.midrange.com/web400.