Re: XML performance issue

On 9/14/2012 6:45 AM, Åke Olsson wrote:

Each XML message is from 15Kto 30K characters in length. Characters not bytes - since the messages are in UTF-16 (CCSID-1200) the message length in bytes is double that.

Generating and sending messages for 64K products took just over 6 and half hours. This to me seems like a lot.

I just spent the last several months putting together an infrastructure for a very similar project. Unfortunately, your numbers aren't that far off.

First, the generation. You can generate XML into a local buffer and send that buffer to MQ, or you can write the XML to an IFS file and then send the IFS file. The second part adds the overhead of reading the file back from the IFS, but it allows you to look at what you sent, which is pretty important for debugging. I ended up with a compromise: I always sent from an internal buffer but I had a switch that would optionally write the message to an IFS file. It's saved me countless hours of debugging.

Next, the transmission. A lot of your overhead will depend on your MQ configuration. I am NOT an MQ expert by any means, but one thing to look into is your logging level. If your queues are logged, then MQ will be writing the messages to a disk file in addition to sending them on. In a multi-platform environment, that means writing to the IFS for each platform. I found that once we added logging (required for the package I was installing), message processing jumped into the tens or even hundreds of milliseconds per message. If I do the back of the envelope math for you, you're getting throughput of about 300ms per message. Unfortunately, at least with logging, that's not unexpected.

The first thing I would do would be to segregate the generation and the sending. Write code to just generate the messages without sending to MQ and see how long it takes. Compare that time to your total time; any additional time is overhead from MQ.

Joe