On 21-Oct-2014 13:40 -0500, Gary Thompson wrote:
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote
server. If an FTP connection can't be made to the remote, the ASN
document is copied to a member file WAIT_Q, and the ASN program
continues with the next ASN as our warehouse completes the next
shipment.
We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been
sent from members in a file I call SEND_Q. A separate file is used
to avoid contention with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process
and looks to copy any members from WAIT_Q to SEND_Q and then remove
all _copied_ members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?
The most conspicuous solution would be to *not* move them. Of course
they are not actually being _moved_, they are actually copied; that they
are copied instead of moved complicates the issue, per likely performing
a clearly non-atomic operation that should appear atomic to
participants, to avoid any requirement for the alluded external
synchronization. Avoiding yet another _copy_ is also a benefit.
The FTP_ASN_2 job could process only members in WAIT_Q added *prior*
to that deviation [away from the SEND_Q file], and could even operate
directly against the WAIT_Q file; send the member data, and then remove
that member when sent. If the extra _copy_ was required, that same
algorithm [using FIFO] should prevent collisions as well. The Retrieve
Member Description (RTVMBRD) enables traversing the member list by
date-order [FIFO], as well the Display File Description (DSPFD) for the
Type Of Information (TYPE) of *MBR when output is directed to an Output
File (OUTFILE).
My ideas have so far included:
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.
As a resource external-to\separate-from the actual resource being
copied, there is a level of indirection; implicitly adding specific
complexity. However as a single entity, that might be utilized with a
lower cost, both in coding and run-time.
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q
with ALCOBJ.
The Allocate Object (ALCOBJ) can only exclusively lock the data of a
*specific member*, not the file; the file and member objects will have
an *SHRRD lock, and only the data [the dataspace actually] will have an
*EXCL lock. The methods available to lock exclusively the actual
database *FILE object, are incompatible with also enabling effective
atomicity of the multi-step process; i.e. the processing would be unable
to effect Remove Member (RMVM) while that lock is held, regardless the
lock is held within the same thread\process.
However the move\copy process could lock each member [the dataspace]
separately, exclusively, and then release those locks implicitly by the
eventual RMVM request, only after all of the members had been copied.
Thus somewhat more complicated than locking a third-party object, but
the mutex is established directly on each entity\resource rather than
something elsewhere.
Looking for any advice or comments about design pitfalls . . .
The idea of using members for logical separation of the data is often
a poor choice when there are concerns for concurrency\synchronization;
database members, by design, are best treated as something to persist.
As Buck suggests, the use of data _records_ is a much better approach;
logical separation by a key. Note: while keeping the multi-member
implementation, the mutex could be implemented on the first record of
data or all rows of data in any one member being processed, and could
include commitment control for an easier means to backout [rollback]
work; deleting all of the records could serve as a semaphore for
allowance of removing the member.
Given the Stream File (STMF) can similarly be both processed as data
[possibly as /records/ or as just a image\binary] by the FTP, and
processed as /objects/ operated against by the system with effective
atomicity, the STMF might be a better choice; e.g. one might expect the
operations like move and rename should appear atomic to participants
accessing the STMF. That eliminates the travails with accomplishing
what is an effect atomic member\data copy and removal of the original.
I can not vouch for the efficacy of, with regard to the effective
atomicity of, the Move Object (MOV) [aka MOVE] command, but that would
be theoretically the more appropriate choice than a copy. That feature
is implemented with a copy [effectively Copy File (CPYF); just what
might be the choice for a user data copy], but ideally would be
implemented such that concurrent operations were unable to interfere
with the /from-objects/. And even if the individual object /moved/ is
seemingly atomic, and while there is the ability to move a generic [i.e.
effectively *ALL], there is no capability to perform the multi-object
operation under commitment control, so the issue with a partially
complete operation would exist if using that function. IIRC the
documentation [help text] for the MOV command says there is no support
for /moving database members/, but my recollection was that the feature
worked as long as the CPYF feature could operate on the files, and
specifically to operate having specified *NONE for the Format Option
(FMTOPT) parameter; i.e. the record formats were identical.
As an Amazon Associate we earn from qualifying purchases.