Hello Alain,
Am 25.10.2021 um 23:50 schrieb Alan Campin <alan0307d@xxxxxxxxx>:
I have done all of this in C for an open source project that I was working
on. I had pre-start jobs running in my own subsystem and I would spawn a
job to perform programming work that came from a socket program.
Do I get this correct? You had a socket listener running (as PJ?), and dynamically spawn workers?
It would perform the work which could be of any length but what I ended up
doing was using a User Queue to return a result which worked fine.
I don't really understand what the queue is meant to do. A deposit for communication between the socket listener and the peer application?
Received the request, looked up to see if it had Job running and created
one if it did not and then sent that request to a pre-start job and got
back the response through a user queue.
Ah, now I get it. This is different than I'm used to from Linux/Unix. There you usually have one (rarely more) "main" listener process(es). If a TCP session request comes in, the listener will copy itself (do a fork call) and finish the TCP connection. This one will do the work until the socket is closed or whatever condition should make the child process end. Meanwhile, the main listener is free to handle additional requests, because the socket accept() call is done in the worker.
This is working very fine for a long time, but despite all kernel-side tricks like copy on write and whatnot, a fork call is expensive. And the slower the machine, the more time it takes to fork a new worker process. (Inetd is doing likewise and was mainly thought as lessening a system's memory usage when a lot of tcp listeners were running independently.)
Another example: The Apache Web Server on Linux had the "prefork" process model available for years, to alleviate the time and CPU load needed for a fork, trading speed for higher memory requirements. At system boot time it preforks a configurable amount of worker children, handing incoming requests over, opening files in the file system as needed, reading them, and pushing data over the tcp connection. Apparently the old IBM http server does likewise. (I'm ignoring CGIs for the sake of simplicity here.)
What I have in mind is to mimic this behavior, but not with truly running workers, instead "having PJ jobs available", and triggering them active by whatever means is needed by the socket listener "handing over" a connection request. Since I also want to do the accept() in the worker, the tcp socket listener is unaffected and continues to dispatch incoming request to workers.
Terms I've put into quotes are still something I need to learn about. Especially considering that OS/400 has no fork() but spawn(), I'm not sure if I can duplicate the behavior of a fork with spawn. I will re-read the UNIX C Applications Porting to AS/400 (
https://ibmdocs.pocnet.net/SG24-4438-00.pdf and
https://ibmdocs.pocnet.net/SG24-4938-00.pdf) to learn more about that topic.
This "accept in child process"-approach doesn't require queues, because the socket communication and file system access stuff is done completely within one task/thread/process. Still, some details remain with question marks.
In parallel, I'll re-read Chris Hird's "Let's C" blog posts to understand how he dealt with that handover to a worker process.
Anyway I would be willing to share some code if you are interested.
I'm interested! In addition, if you don't mind, what about publishing your OpenSource code on Github, so everybody interested can see and learn?
One more point I also did the same thing for CGI jobs. I wrote a single
program that was called for all CGI requests. That program then dispatched
to one of pre-start job or started one. The result was sent back and
returned to the caller.
I see. But exactly that complete spawning of a new process is what inetd currently does and what takes 2-3 seconds. This delay is what I want to eliminate. And I'm curious how to exactly do this "dispatching to a PJ".
I should mention that when I said, "I have heard that data queues are very
slow" I was referring to communicating from the outside world through ACS.
Not relevant to me. :-)
Data queue inside the IBM i are fast although User Queues a lot quicker.
Probably also not relevant.
:wq! PoC
midrange.com
