Fw: Java Remote Command Failure

Hi All,

Our QA department is implementing new remote commands initiated from a WebSphere (AIX) Application Server and they are reporting that 20%-30% of the remote command connections are failing with the response code:


<ResponseInfo code="6003" description="Connection Exception" message="Established connection failed to component:AS400_CMD:216.58.195.132" source="AS400_CMD"/>


I asked for a java log showing the error and received the following:

"2016-05-02 00:14:35.527 [ERROR] [WebContainer : 192] [1079c481-7768-4f48-a159-00bf23a09c8d] com.ABC.soa.msg.AppMsgImpl@d010db90[code=6003,description=Connection Exception,message=Established connection failed to component:AS400_CMD:216.58.195.132,system=AS400_CMD]
com.ABC.soa.exception.SystemException: com.ABC.soa.msg.AppMsgImpl@d010db90[code=6003,description=Connection Exception,message=Established connection failed to component:AS400_CMD:216.58.195.132,system=AS400_CMD]

at com.ABC.soa.icom.common.As400CommandCall.execute(As400CommandCall.java:289) ~[soa-icom-commons.jar:na]
at com.ABC.soa.icom.dao.AccountsReceivableDao2Impl.getActualDaysPastDue(AccountsReceivableDao2Impl.java:137) ~[soa-icom-commons.jar:na]
at com.ABC.soa.billing.dao.IcomsRetrieveBillingInfoV1x2DaoImpl.retrieveBillingInfo(IcomsRetrieveBillingInfoV1x2DaoImpl.java:173) ~[soa-billing-services.jar:na]
at com.ABC.soa.billing.dao.IcomsRetrieveBillingInfoV1x2DaoImpl.getBilling(IcomsRetrieveBillingInfoV1x2DaoImpl.java:142) ~[soa-billing-services.jar:na]
at com.ABC.soa.billing.service.RetrieveBillingInfoV1x2ServiceImpl.getBilling(RetrieveBillingInfoV1x2ServiceImpl.java:85) ~[soa-billing-services.jar:na]
at com.ABC.soa.billing.endpoint.RetrieveBillingInfoV1x2EndPoint.retrieveBilling(RetrieveBillingInfoV1x2EndPoint.java:64) ~[soa-billing-services.jar:na]
at sun.reflect.GeneratedMethodAccessor240.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) ~[na:1.6.0]
at java.lang.reflect.Method.invoke(Method.java:611) ~[na:1.6.0]
at org.springframework.ws.server.endpoint.MethodEndpoint.invoke(MethodEndpoint.java:134) ~[spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.ws.server.endpoint.adapter.MarshallingMethodEndpointAdapter.invokeInternal(MarshallingMethodEndpointAdapter.java:140) ~[spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.ws.server.endpoint.adapter.AbstractMethodEndpointAdapter.invoke(AbstractMethodEndpointAdapter.java:53) ~[spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.ws.server.MessageDispatcher.dispatch(MessageDispatcher.java:233) [spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.ws.server.MessageDispatcher.receive(MessageDispatcher.java:173) [spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:88) [spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:59) [spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:239) [spring-ws-2.1.3.RELEASE-all.jar:2.1.3.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936) [spring-webmvc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:838) [spring-webmvc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) [javax.j2ee.servlet.jar:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812) [spring-webmvc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) [javax.j2ee.servlet.jar:na]
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:914) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200) [com.ibm.ws.webcontainer.jar:na]
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453) [com.ibm.ws.runtime.jar:na]
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515) [com.ibm.ws.runtime.jar:na]
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306) [com.ibm.ws.runtime.jar:na]
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:277) [com.ibm.ws.runtime.jar:na]
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214) [na:CCX.CF [o1330.02]]
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113) [na:CCX.CF [o1330.02]]
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175) [com.ibm.ws.runtime.jar:na]
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) [com.ibm.ws.runtime.jar:na]
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205) [com.ibm.ws.runtime.jar:na]
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1702) [com.ibm.ws.runtime.jar:na]
com.ibm.as400.access.ConnectionDroppedException: Disconnect request received, connection terminated.
at com.ibm.as400.access.AS400ThreadedServer.forceDisconnect(AS400ThreadedServer.java:237) ~[jt400.jar:JTOpen 7.4]
at com.ibm.as400.access.AS400ImplRemote.disconnectServer(AS400ImplRemote.java:453) ~[jt400.jar:JTOpen 7.4]
at com.ibm.as400.access.AS400ImplRemote.disconnect(AS400ImplRemote.java:425) ~[jt400.jar:JTOpen 7.4]
at com.ibm.as400.access.AS400.disconnectAllServices(AS400.java:1203) ~[jt400.jar:JTOpen 7.4]
at com.ibm.as400.access.ConnectionList.removeExpiredConnections(ConnectionList.java:648) ~[jt400.jar:JTOpen 7.4]
at com.ibm.as400.access.AS400ConnectionPool.cleanupConnections(AS400ConnectionPool.java:137) ~[jt400.jar:JTOpen 7.4]
at com.ibm.as400.access.PoolMaintenance.run(PoolMaintenance.java:86) ~[jt400.jar:JTOpen 7.4]"


looking at the classes in in the java logs, it appears that WAS makes a new connection request from the connection pool but immediately the ConnectionDroppedException class is invoked and then the iSeries cleans up the disconnected connection from the connection pool.


I have an open PMR with IBM on both sides but we are finding this difficult to troubleshoot. Can anyone give any advice for gathering more detailed insight into digging into this error?


Jim