Re: C400-L Digest, Vol 9, Issue 22

Hi, Boris
I've recently had the same problem - exporting 12,000,000 records table to a CSV file in IFS. It was a daily process and took ca 50 minutes - just a plain CPYTOIMPF. All my attempts to tune block sizes gave nothing. It consumed 50% CPU on our 2-core (I think) box.
Then I wrote a specific program to export the table in exactly the same format as CPYTOIMPF did. It takes 5 minutes now and it also consumes 50% CPU, that is, if I'm not mistaken, the same close to 100% you are worrying about.
So to solve your psychological problem - try to do the same using CPYTOIMPF, then switch back and be happy with you program as you will probably also see the difference.

Here's what my program does:

/*---------------------------------------------------------*/
/* Must be compiled with options */
/* SYSIFCOPT(*IFSIO *IFS64IO) */
/*---------------------------------------------------------*/
#include <stdio.h>
#include <stdlib.h>
#include <recio.h>
#include <string.h>
#include <decimal.h>
#include <ctype.h>
#include <iconv.h>

#pragma mapinc("ompfe",......
#include "ompfe"
#define ompfe ......

int main(int argc, char *argv[])
{
_RFILE *fm ;
_RIOFB_T *rfm ;
FILE *OM_Exp ;
ompfe Movement ;
char szTemp[160], szTempOut[160], szTmp1[32] ;
char FromCode[32] = "IBMCCSID000370000000";
char ToCode[32] = "IBMCCSID01252";
char *InpBuf, *OutBuf ;
size_t i, InpLen, OutLen ;
iconv_t cd;

fm = _Ropen("EXPORT/OMPF1", "rr") ;
............
OM_Exp = fopen(argv[1], "wb") ;
............
cd = iconv_open( ToCode, FromCode ) ;

rfm = _Rreadf(fm, &Movement, sizeof(ompfe), __DFT) ;
while (rfm->num_bytes == sizeof(ompfe))
{
// forming szTemp from Movement........
InpBuf = szTemp ;
OutBuf = szTempOut ;
iconv( cd, &InpBuf, &InpLen, &OutBuf, &OutLen );
szTempOut[140] = 0x0D ;
szTempOut[141] = 0x0A ;
szTempOut[142] = 0x00 ;
fputs(szTempOut, OM_Exp) ;
rfm = _Rreadn(fm, &Movement, sizeof(ompfe), __DFT) ;
}
iconv_close (cd) ;

_Rclose(fm) ;
fclose(OM_Exp) ;
return(0) ;
}

My first version was without CCSID conversion but adding it had close to no effect on the timing (that is additional 12 million calls to iconv).
The only difference I see is that you use "fwrite" and I use "puts".
Try to change - may be it will have some effect.
But once again, if timing is OK, do not understand why you should care about CPU load...

HTH,
Jevgeni.



-----Original Message-----
From: c400-l-bounces+jevgeni.astanovski=sampopank.ee@xxxxxxxxxxxx [mailto:c400-l-bounces+jevgeni.astanovski=sampopank.ee@xxxxxxxxxxxx] On Behalf Of Boris
Sent: 24. mai 2011. a. 6:21
To: c400-l@xxxxxxxxxxxx
Subject: Re: [C400-L] C400-L Digest, Vol 9, Issue 22

Thanks for the quick response.
I'm probably missing on some basics here.
Here is a sample code:


for (i=0;i<=2000;i++)
buf[i]='c';
out = fopen("fileout.txt","w");
/*out = fopen("fileout.txt","wb,type=record,lrecl=2000");*/

for (i=0;i<=900000;i++)
{
num = fwrite (buf, 1, sizeof(buf), out );
}

fclose(out);

I removed the read function. It was populating the buf.
This program takes about 90% CPU.
I also tried:
num = fwrite (buf, sizeof(buf), 1, out );
It did reduce the CPU to about 85%. Then I increased the size of the buf to
10000 assuming it would further reduce the CPU but it instead jumped to 94%.


How do I make the program write by records/blocks? The commented line
doesn't work - fopen returns NULL.

Thanks.
Boris.