RE: Garbage characters received by socket

Scott,

I think you may have nailed it.

The programmer responsible for the socket has written a few before but they 
have had this same issue.  The 'fix' was to increase the buffer size on the 
sending Windows system.

I'll pass this along and hopefully we can improve our socket process.

BTW, we are using TCP-IP for the communications.

Thanks,

Rick 

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On 
Behalf Of Scott Klement
Sent: Tuesday, March 13, 2007 3:00 PM
To: RPG programming on the AS400 / iSeries
Subject: Re: Garbage characters received by socket

Hi Rick,

We have an application that monitors a socket for application 
information in an XML format.  At varying places in the transmission 
we receive a string of '@'.

I've seen this before, but it's ALWAYS an error in the RPG coding. 
Usually a programmer who is learning sockets for the first time, and didn't do 
a very good job of debugging their code.

To illustrate what can cause this... consider the following code:

D InputData       s            100a
    .
    .
   len = recv(mySock: %addr(InputData): %size(InputData): 0);
   QDCXLATE( %size(InputData): InputData: 'QTCPEBC');


How many bytes will the recv() API get?  It might receive 100 bytes -- if the 
network speed is fast enough to deliver 100 bytes in one call. 
But that certainly cannot be counted on.  The recv() API does not know that the 
RPG programmer's world revolves around fixed-length fields in fixed-length 
records.

It thinks it's reading a network stream.  It can return as few as 1 byte per 
call, or as many as the size I specified (100 in this example) and every time I 
call it, it can be a different number.

Look at the QDCXLATE API that I call after recv().  Note that I'm translating 
the WHOLE RPG variable, NOT just the part that I received. 
See the mistake?

In the preceding example, if I received 80 bytes, then the 80 bytes would look 
correct, but there'd be 20 @ symbols after those 80 bytes, to 
fill up the rest of the field!   Those @ symbols didn't come from the 
socket, they came from the blanks that were in the last 20 bytes of my field!

You see, in EBCDIC, x'40' = BLANK.   However, I've told QDCXLATE that my 
field isn't EBCDIC, it's ASCII!!  And in ASCII, x'40' is the @ symbol. 
So when it translates the blank characters, it THINKS they're @ symbols, 
because that's what they'd be if the field was an ASCII field.   When it 
translates it to EBCDIC, it therefore converts the x'40' characters into the 
EBCDIC code for @, which is x'7C'.

So the socket never received the @ symbol, it was my fault for translating the 
extra spaces at the end of my field, despite that they were never received on 
the socket!

The issue is that we sometimes don't receive the entire XML string.
A co-worker is responsible for this process and tells me that there is 
a 32k buffer restriction.  He thinks the '@' string sometimes pushes 
the data beyond the buffer size and we lose it.

You didn't tell us which network protocol you're using with your socket. 
    However, I can't think of any network protocol where the preceding 
paragraph makes sense.

With TCP, if the buffer fills up, the sender gets stuck blocking on the
send() API until you receive part of the buffer and make space for more data to 
come in.

With UDP, data could be lost, but you'd never get anywhere close to 32k!!  
You'd start losing data around the 500 byte mark, or maybe 1000, or maybe 1500, 
depending on the MTU of your underlying IP stacks.

And Unix-Domain sockets work very much like TCP... the sender blocks...

The only thing I can think of is that you may be using non-blocking sockets, 
and only part of the data is sent because of buffer space.  If this is the 
case, your programs are written incorrectly.  Don't just assume that there will 
always be enough buffer space!!  Non-blocking sockets MUST take care to re-send 
anything that wasn't sent the first time.  It has nothing to do with a 32k 
buffer, either.  Depending on the link speed, there may be previous data in the 
buffer, so the available space may be much smaller than the total size of the 
buffer.

When the programs are properly written, there's no practical limit on the 
amount of data that can be received over a socket.  Just make sure you 
understand that data is received as a STREAM, not a fixed-length record.  If 
the sender calls send() with 500 bytes, there's no reason to expect that the 
receiver can call recv() and get 500 bytes in one call. 
  He may get 123 bytes on the first call, then 209 on the 2nd call, then
169 on the 3rd call... they add up to 500, and all 500 will be received with 
nothing lost, but it's not reasonable to assume that they'll all appear in one 
call to recv()!!  That's just not how TCP works.

Though, perhaps you're not using TCP...  you didn't say.

Does this sound right?  Has anyone else encountered this type of 
behavior using sockets in the iSeries?

Absolutely.  However, it's always been a programmer error.


Privileged and Confidential.  This e-mail, and any attachments there to, is 
intended only for use by the addressee(s) named herein and may contain 
privileged or confidential information.  If you have received this e-mail in 
error, please notify me immediately by a return e-mail and delete this e-mail.  
You are hereby notified that any dissemination, distribution or copying of this 
e-mail and/or any attachments thereto, is strictly prohibited.