Socket close() and then re connect()

I'm encounterring a problem that I couldn't quite understand. The
problem is related to closing the socket connection and then reconnect
again. When I attempt to reconnect, I'll, sometime, get the "A remote
host refused an attempted connect operation." error. It almost seems
like the close didn't really drop the connection on the server side and
make the server port become available again.
I did some googling and found this:
When the close() system call is initiated by an application, only the
interface to the socket is destroyed, not the socket itself. It is the
kernel's responsibility to destroy the socket internally. Sometimes, a
socket may enter a TIME_WAIT state, on the server side, for up to 4
minutes.

Would what I found above be the cause of my reconnect problem?

Thanks