On 17-Dec-2013 07:29 -0800, Michael Schutte wrote:
Every month we are getting an error from SNDDST saying that it cannot
be ran in jobs with multiple threads. I get that this command cannot
run with multiple threads. But I am curious on why we are getting
multiple threads in an interactive session.
The /system/ can create additional threads, system threads,
irrespective the Allow Multiple Threads (ALWMLTTHD) attribute for the
job. Only user threads are prevented from being created in the job
contrary to the threading limitations.
The program trying to use this command is vendor code. All of their
code is written in RPG3 and CLP running in the default activation
group. We now own the code and as we make changes to the code, we
convert to RPGLE. We have created service programs, some with named
activation groups, others with named activation groups.
Some with named ActGrp and some with ¿caller? or ¿new?
We have also introduced SQLRPGLE into some programs (FYI, when we
make changes to the vendor code, we move to a new library higher in
the LIBL, we leave the original code alone, just our decision to do
that). Side note, all programs called interactively is called by a
menu program written in RPG3 by the same vendor, as we convert the
called programs from RPG3 to RPGLE we sometimes add H-SPEC
actgrp(*CALLER) other times we do not. When using *CALLER, it's my
understanding that the program run in the default activation group.
Not sure if this matters to anyone trying help me with this
situation.
In the *CALLER scenario for an SQLRPGLE called from the DftActGrp, a
reusable query will not close [Close SQL Cursor (CLSSQLCSR) parameter],
so the associated threads can remain active, awaiting more work to
complete the query activity. Similarly any open\active query
implemented with multiple threads that has not been full-closed could
presumably cause the same issue.
Depending on the origin for the multiple threads, if by query
activity, the Change Query Attributes (CHGQRYA) to assign a Parallel
Processing Degree (DEGREE) of *NONE can suppress the use of multiple
threads for a query. If however the threads are due to User Defined
Function (UDF) invocation versus either I/O parallel or SMP processing,
then the change to the Query Attributes [AFaIK] does not apply.
I've been reading articles on the web about multithreading (new
concept to me). I remember once reading that to start a new thread,
the program has to specifically call the spawn routine. I'm certain
that none of us at this shop has done this in any programs (lack of
understanding being the biggest hurdle). I've also read that Only
batch immediate jobs and prestart jobs provide multithread-capable
support here:
<http://publib.boulder.ibm.com/html/as400/v4r5/ic2924/index.htm?info/RZAHWOVEPO.HTM>
Warning: The above link froze my browsers for several minutes
The above link is further described by the following:
v4r5 InfoCenter -> Programming -> Programming Support -> Multithreaded
Applications
_i Multithreaded applications i_
"
_What is a thread_?
...
_Thread basics_:
...
_Programming with threads_:
...
_Code snippets that show how to use threads_:
..."
With that information, I found the effective equivalent v7r1 link is:
<
http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/topic/rzahw/rzahwovepo.htm>
wherein there is a link to:
<
http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/topic/rzahw/rzahwprint.htm>
_PDF file for Multithreaded applications_
The statement about the spawn is that "The spawn() API is the only
programming method that can start a batch immediate or prestart job that
is capable of supporting multiple threads." However earlier a statement
notes that "In the i5/OS® kernel threads support, only a subset of the
supported job types can create threads. Interactive and communication
jobs do not provide multithread-capable support." And that the "OS
examines all job types except communications jobs and interactive jobs
for the ALWMLTTHD parameter." So while the spawn() API may effect a
multithreaded job type of BCI or PJ, that does not suggest other job
types are precluded from multi-threading per use of ALWMLTTHD; i.e.
allowing multiple threads, beyond just system-initiated /system
threads/. For example even a batch job started with Submit Job (SBMJOB)
using either the Allow Multiple Threads (ALWNLTTHD) parameter to *YES,
or instead deferring to a Job Description (*JOBD) set to allow threads,
will start a multithreaded job type of BCH; the threads within the job
may not be started with the spawn() API, but other thread APIs such as
pthread_create() (a POSIX thread API). So anyhow, hopefully to clarify:
<
http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/topic/apis/293735.htm>
_Running threaded programs_
"When you run a threaded program, the job that runs a threaded program
must be specially initialized by the system to support threads.
Currently, several mechanisms allow you to start a job that is capable
of creating multiple kernel threads:
..."
But that's documentation for V4R5, we are currently on V7R1. But if
this is still the case why are our interactive jobs spawning multiple
threads.
Again, most likely system-initiated threads; i.e. system-threads that
are allowed to disregard the ALWMLTTHD(*NO) specification of an
interactive job.
Unfortunately, I do not have any job logs to include in this email,
however as I recall the error was message CPFA0A8 "Operation not
allowed in a job running multiple threads."
When the error occurs, the output from WRKJOB OPTION(*ALL)
OUTPUT(*PRINT) SLTTHD(*ALL) I expect should produce output that includes
both the *THREAD information and the thread stacks. With the details
from the threads, perhaps what created those other threads will be
obvious. There is however an allusion to a potential restriction for
the above WRKJOB invocation, so the OPTION(*PGMSTK) might have to be
specified; i.e. the docs suggest a [but not what specific] failure, if
OPTION(*ALL) is invoked "in a job that allows multiple threads" versus
being invoked in a job that _has_ multiple threads. Another more likely
issue, is that the system threads for the Query Engine will sit idle for
an open query not actively running, such that the threads have no active
stack, pending receipt of further work.
The Dump Job (DMPJOB) with Job Threads (JOBTHD) parameter as *YES
asking for thread information might be required to get the necessary
information, because the threads may have no active stack. While I am
not aware of any system vs user nor any identification of what initiated
the thread visible from Work With Job (WRKJOB) output, perhaps something
in the DMPJOB gives more details; e.g.:
DMPJOB PGM(*NONE) JOBARA(*NONE) ADROBJ(*NO) JOBTHD(*YES)
A job trace [Start Job Trace (STRTRC)] started at the point of the
failure is potentially helpful, but the tracing must catch and include
in its output, future activity of those other threads, even if only the
termination of the threads.
I also came across this website. Multithreaded programs in Java
<http://pic.dhe.ibm.com/infocenter/wmqv7/v7r1/topic/com.ibm.mq.doc/ja11160_.htm>
"The Java™ runtime environment is inherently multithreaded."
While essentially true, though for a different reason, the reference
to MQM docs is probably not the best reference for the issue at hand.
More appropriately the following doc reference; see my emphasis with
asterisks for the JVM:
<
http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/topic/rzahw/rzahwjavco.htm>
IBM i 7.1 Information Center -> Programming -> Multithreaded
applications -> Language access and threads
_i Threads considerations for Java language i_
"Java™ threads operate on top of the i5/OS® kernel threads model using
the java.lang.Thread class. Each Java thread is one of the many tasks
that run in the process.
You can do all of the activities that are listed in the Threads
Management section.
The Java virtual machine (JVM) *always creates several threads* to
perform services such as Java garbage collection. The system uses these
threads; applications should not use them.
You can use native methods to access system functions that are not
available in Java. Native methods are not *PGM objects. They are
procedures that are exported from Integrated Language Environment® (ILE)
service programs (*SRVPGM). These *native methods always run* *in
multithreaded processes*; therefore, they must be threadsafe. The ILE
COBOL, RPG IV, CL, C, and C++ compilers are threadsafe.
..."
So I can see where this could be our issue. Some new emails that we
send out are being sent using RPGMAIL. RPGMAIL uses Java. So if the
user did something to send out an email, I can see this being our
culprit. But there's only one program that I recall that gets sent
using RPGMAIL and I know that it hasn't been sent because I am copied
on it. All other RPGMAIL emails are sent from batch job that are
submitted from the Job scheduler and therefore wouldn't have an
effect on any one interactive session.
Perhaps then, JAVA has been excluded.?
This doesn't happen for all users, just a select few. The ones that
get the error are different from month to month.
Maybe review the PRTSQLINF of the SQLRPGLE [service] programs to look
for UDF utilization and any parallel implementations.
The boss started asking why has this just started. The truth is we
really don't know if it just now started to happen as users used to
ignore these kind of messages. As far as they were concerned the
program completed successfully because accounts receivables had
generated an invoice (but no email was sent).
Perhaps changing the code in reaction to the failure of the SNDDST to
do something that the user can not merely ignore; performing something
that [unlike the SNDDST] does not fail when multiple threads are active
in the process.
Others, in discussions found on the web for a matching failure,
suggest using something other than SNDDST to avoid the restriction; e.g.
either use a command [or API\program-call] to effect the send of the
message\mail that is not restricted to a process with only the
one\initial thread, or run the SNDDST in another process such as in a
submitted job [per SBMJOB] or off-load to a job awaiting an entry on a
queue that notifies of the SNDDST work to be done.
But if this is new what could be spawning the multiple threads? Is it
the SQL, Activation Groups, or is the Java?
I am betting some SQL, and the ActGrp choice may play a role, due to
the effect on the [lack of] closure of cursors. Possibly the query
implementation for certain\specific inquiries is done using parallel
processing, such that only those jobs later becoming the /incident/ with
the failing symptom; or perhaps they always implement with parallel, but
only sometimes does a user end up taking that particular path through
the code.
The open-files list [WRKJOB OPTION(*OPNF)] might help to infer some
query activity, and the activation group for the open. If the job
remains active with multiple threads, and the user can be allowed to
issue a SNDDST for testing, then servicing [STRSRVJOB] and debug
[STRDBG] could be issued against the job. After verifying the SNDDST
still fails, reclaim each of the activations [RCLACTGRP] and test again
if the SNDDST still fails [and\or the WRKJOB OPTION(*THREAD) to see if
the threads went away]; a SQL cursor getting closed as a side effect
will appear in the joblog as a query debug message. That could help to
find the program\query responsible for the extra threads.
midrange.com
