You're right that INADDR_ANY allows connections from any adapter. (Which is what you want 95% of the time...)
Instead of INADDR_ANY, you simply set it to the IP address (binary -- not dotted) of the adapter you want to bind to.
bind_addr = inet_addr("127.0.0.1");
// fixme: do error checking here.
localAddr.sin_addr.s_addr = bind_addr;