Try sending scope message (QMHSNDSM) API. If your program crashes you can perform cleanup routines. Also, check for the record lock timeout values, so even if the program crashes no one lock can keep others in a stalemate. Finally, if that's the only reason you're using C for this, you might consider RPG and SQL. I rarely use C for strict DB file handling, it's not worth the trouble with SQL and RPG around. I don't see where a file open is an issue unless it's an exclusive lock. If the file is in QTEMP there's no reason to hold an exclusive lock(if you are) which would solve the re-entry problem.
However, I noticed you stated that a user space is an option. Then you could use a stream file (flat file). Again, same thing just don't open it exclusive and if the program bombs you can re-create it repeatedly with no locks. It is possible that the record handling code is causing storage issues as suggested so maybe removing that code or moving it to RPG may help.
But aside from all that, I would use queues for what you're doing it's a much more elegant solution and very robust on the i. It doesn't make sense to use files or user spaces for message passing and IPC.
Stay in touch when you're away with Windows Live Messenger.