Re: CGIDEV2 file upload data conversion -- WEB400

When uploading a file attachment through the browser, I think CGIDEV2 just dumps the received binary data to a temporary IFS file. This is then available to my development, where I write this IFS file to a BLOB column in my DB2 table.

I did notice that CGIDEV2 is setting this IFS file as CCSID 819, and wondered whether this would be better as 1208 or whatever is defined in DefaultNetCCSID. However, I think the first issue is that when debugging CGIDEV2, I see that the hex values of the data being written to the IFS file are already corrupted before being written to the IFS. I don't think this data is being manipulated/converted by CGIDEV2 prior to being written to the IFS file.

It appears, to me, that it is the httpd.conf settings that are causing the data corruption/conversion. If I change CGIConvMode, I see different hex values coming through from the browser. However, I can't seem to get the settings right, whatever I try, the resulting hex values are never the same as when I just copy the XLSX to the IFS using NetServer.

-----Original Message-----
From: WEB400 [mailto:web400-bounces@xxxxxxxxxxxx] On Behalf Of Eric
Sent: 01 December 2016 15:34
To: Web Enabling the IBM i (AS/400 and iSeries) <web400@xxxxxxxxxxxx>
Subject: Re: [WEB400] CGIDEV2 file upload data conversion

I'm not the expert here but I don't think the upload corrupts the CCSID of the uploaded file.
Whatever you upload from the browser, should be uploaded as it is.

It could be the write into the IFS that is messing it up.

Where are you storing it? What is the CCSID of the directory you created to store this file?

On 1 December 2016 at 15:03, McGovern, Sean <sean.mcgovern@xxxxxxxxxxxxx>
wrote:

I'm finding this very confusing. I seem to still have a problem
uploading files via the web using CGIDEV2.

I create a txt file using Notepad, containing text ABC. I transfer
this using NetServer to the IFS. In the IFS, if I display the hex
values, I get
41 42 43, as expected (in CCSID 819).

If I then upload this same txt file via my web process, I get the
correct result in my database table. The hex values are 41 42 43.

However, the data appears to be converted if I upload a XLSX file, for
example.

I create a dummy XLSX file. I transfer this using NetServer to the
IFS. In the IFS, the first 20 hex characters show...

504B0304 14000600 08000000 210062EE 9D685E01

...I assume this is correct. I can open the Excel from the IFS without
issue.

If I then upload this same XLSX file via my web process, I don't get
the same hex values in my database table. I get...

504B0304 1400060 008000000 210062C3 AEC29D68

If I debug service program CGISRVPGM2, at the point where it saves the
file data to the temporary IFS file, the first hex values (program
variable
outds) are...

504B0304 1400060 008000000 210062C3 AEC29D68

If I understand correctly, this data is not being
manipulated/converted by CGIDEV2, it is just the data provided through the web.

Somehow, the data is being converted/corrupted. I assume this is due
to a httpd.conf setting? I have...

CGIConvMode %%EBCDIC/EBCDIC%%
DefaultFsCCSID 937
AddCharset UTF-8 .html .pgm
DefaultNetCCSID 1208

Tried %%BINARY/EBCDIC%% but no joy. Gave different hex values, but not
correct. Tried %%BINARY/BINARY%%, but my web application doesn't work
at all with that setting.

Ideas?

-----Original Message-----
From: McGovern, Sean
Sent: 01 December 2016 08:44
To: Web Enabling the IBM i (AS/400 and iSeries) <web400@xxxxxxxxxxxx>
Subject: RE: [WEB400] CGIDEV2 file upload data conversion

I'm using this technique to return a BLOB to a browser...

http://iprodeveloper.com/application-development/return-blob-browser

As I'm allowing any file type to be uploaded, when the user downloads,
I'm specifying...

Content-Type: application/octet-stream

However, even if I specify the correct content type, I still seem to
have an issue with PDF, DOC, XLS etc., the downloaded file is corrupt
and will not open.

-----Original Message-----
From: WEB400 [mailto:web400-bounces@xxxxxxxxxxxx] On Behalf Of Bradley
Stone
Sent: 30 November 2016 18:33
To: Web Enabling the IBM i (AS/400 and iSeries) <web400@xxxxxxxxxxxx>
Subject: Re: [WEB400] CGIDEV2 file upload data conversion

I believe you need to set the content type for each type of file that
will be downloaded in your Apache Config.

The content-type you're getting now is a default Apache uses when it
doesn't know the content type.

Here's a link:
https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_si
te/
Configuring_server_MIME_types

This may help. I only did a quick scan. I'm not sure it has a list
of extensions and their proper mime types, but you should be able to
find that using Google.

Brad
www.bvstools.com

On Wed, Nov 30, 2016 at 9:41 AM, McGovern, Sean <
sean.mcgovern@xxxxxxxxxxxxx

wrote:

Thanks.

I've now stopped the base64 encoding on the upload. The results look

good.

I seem to be getting the correct hex values in the DB2 table.

I'm giving the users the ability to upload any type of file. When
the users wish to view the uploaded files, files are downloaded,

specifying...

Content-Type: application/octet-stream
Content-Disposition: attachment; filename="xxx";

Working fine for XML, TXT, JPEG files etc., but have problems with
PDF, DOC, XLS.

When I use Notepad to compare the original uploaded file to the
downloaded file, no differences. Except, if I perform SaveAs in
Notepad, I see the original file has ANSI encoding specified,
whereas the uploaded/downloaded file has Unicode encoding specified.

Any ideas?

-----Original Message-----
From: WEB400 [mailto:web400-bounces@xxxxxxxxxxxx] On Behalf Of
Bradley Stone
Sent: 29 November 2016 18:13
To: Web Enabling the IBM i (AS/400 and iSeries)
<web400@xxxxxxxxxxxx>
Subject: Re: [WEB400] CGIDEV2 file upload data conversion

Hi, Sean.

It looks like the data that "worked" was plain ascii and the other
was base
64 encoded (but still ASCII).

There should be no conversion of the data from ASCII to EBCDIC in
your case since the content type is application/[whatever].

I'd double check the headers on one that doesn't base64 encode the data.
And double check what you're using for the upload option to see if
one
base64 encoded and one didn't.

Or, you may need to read the headers and if it is base64 encoded,
decode it before storing it in your IFS.

Brad
www.bvstools.com.

On Tue, Nov 29, 2016 at 9:55 AM, McGovern, Sean <
sean.mcgovern@xxxxxxxxxxxxx

wrote:

About a month ago, I installed the latest version of CGIDEV2, and
developed functionality that allows users to upload file
attachments to DB2 table via the web, with the ability to download
from the DB2 table. In this way, files can be attached to 'orders'
and users can view these files when they wish. All was working
well (even made a video demonstrating the process to users).

Now paperwork has been approved for me to promote this development
to a QA environment. I tested the functionality again and hit an
issue that I've been unable to resolve. I'm unaware of any changes
that have been made in the past month.

Now, when the file attachment data reaches CGISRVPGM2, it appears
to have been converted and I don't know why.

I have specified...

CGIConvMode %%EBCDIC/EBCDIC%%
DefaultFsCCSID 937
DefaultNetCCSID 1208

If, for example, I attempt to upload a PDF document, I can see in
browser debug, that the post contains...

Content-Disposition: form-data; name="Filedata"; filename="Imp

4991.pdf"

Content-Type: application/pdf
Content-Transfer-Encoding: base64

JVBERi0xLjYN...

The above base64 data translates to "%PDF-1.6", which is what I
see when I open the PDF document in Notepad. I'm unsure whether it
should be base64 encoded, but I believe this was working this way

previously.

When I did this a month ago, I got the correct hex values in the
DB2 table...

255044462D312E31
%PDF-1.6
(CCSID 819)

Now, I get...

4A56424552693078

Debugging, I see this is the data (in hex) that is read and
written to the temporary filename in CGISRVPGM2. CGISRVPGM2
doesn't appear to be converting this data.

I don't know why I'm now getting 4A564...whereas previously I got
2550444...??

Thanks for any help.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400)
mailing list To post a message email: WEB400@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx Before posting, please take
a moment to review the archives at http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400)
mailing list To post a message email: WEB400@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx Before posting, please take a
moment to review the archives at http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400)
mailing list To post a message email: WEB400@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx Before posting, please take a
moment to review the archives at http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400)
mailing list To post a message email: WEB400@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at
http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400)
mailing list To post a message email: WEB400@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at
http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list To post a message email: WEB400@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at http://archive.midrange.com/web400.