× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.



Thank you Chuck,
I learned several things I'd no/little idea of, (happens often reading your replies).
I'm now building/testing code based on a data area that FTP_ASN_1 and FTP_ASN_2 use to avoid contention for WAIT_Q.
The member choice is typical of the many programs in our house that transfer data using FTP, so a solution using data records may come later.
I really like your MOV suggestion, especially doing something similar with STMF on the IFS, which, likewise, may come later.
Your insights into the issues with the copy/delete of members makes believe it's important to avoid contention for WAIT_Q members
between FTP_ASN_1 and FTP_ASN_2.
At one time I had the thought that the right member naming convention could avoid the need for two files . . .
Thanks again!

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of CRPence
Sent: Wednesday, October 22, 2014 1:53 PM
To: midrange-l@xxxxxxxxxxxx
Subject: Re: How to avoid file corruption transferring members from one file to another

On 21-Oct-2014 13:40 -0500, Gary Thompson wrote:
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote
server. If an FTP connection can't be made to the remote, the ASN
document is copied to a member file WAIT_Q, and the ASN program
continues with the next ASN as our warehouse completes the next
shipment.

We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.

I have program FTP_ASN_2 pulling ASN documents that have not yet been
sent from members in a file I call SEND_Q. A separate file is used to
avoid contention with FTP_ASN_1.

When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and
looks to copy any members from WAIT_Q to SEND_Q and then remove all
_copied_ members from SEND_Q.

My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?

The most conspicuous solution would be to *not* move them. Of course they are not actually being _moved_, they are actually copied; that they are copied instead of moved complicates the issue, per likely performing a clearly non-atomic operation that should appear atomic to participants, to avoid any requirement for the alluded external synchronization. Avoiding yet another _copy_ is also a benefit.

The FTP_ASN_2 job could process only members in WAIT_Q added *prior* to that deviation [away from the SEND_Q file], and could even operate directly against the WAIT_Q file; send the member data, and then remove that member when sent. If the extra _copy_ was required, that same algorithm [using FIFO] should prevent collisions as well. The Retrieve Member Description (RTVMBRD) enables traversing the member list by date-order [FIFO], as well the Display File Description (DSPFD) for the Type Of Information (TYPE) of *MBR when output is directed to an Output File (OUTFILE).

My ideas have so far included:

Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.

As a resource external-to\separate-from the actual resource being copied, there is a level of indirection; implicitly adding specific complexity. However as a single entity, that might be utilized with a lower cost, both in coding and run-time.

Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q
with ALCOBJ.

The Allocate Object (ALCOBJ) can only exclusively lock the data of a *specific member*, not the file; the file and member objects will have an *SHRRD lock, and only the data [the dataspace actually] will have an *EXCL lock. The methods available to lock exclusively the actual database *FILE object, are incompatible with also enabling effective atomicity of the multi-step process; i.e. the processing would be unable to effect Remove Member (RMVM) while that lock is held, regardless the lock is held within the same thread\process.

However the move\copy process could lock each member [the dataspace] separately, exclusively, and then release those locks implicitly by the eventual RMVM request, only after all of the members had been copied.
Thus somewhat more complicated than locking a third-party object, but the mutex is established directly on each entity\resource rather than something elsewhere.

Looking for any advice or comments about design pitfalls . . .


The idea of using members for logical separation of the data is often a poor choice when there are concerns for concurrency\synchronization; database members, by design, are best treated as something to persist.
As Buck suggests, the use of data _records_ is a much better approach; logical separation by a key. Note: while keeping the multi-member implementation, the mutex could be implemented on the first record of data or all rows of data in any one member being processed, and could include commitment control for an easier means to backout [rollback] work; deleting all of the records could serve as a semaphore for allowance of removing the member.

Given the Stream File (STMF) can similarly be both processed as data [possibly as /records/ or as just a image\binary] by the FTP, and processed as /objects/ operated against by the system with effective atomicity, the STMF might be a better choice; e.g. one might expect the operations like move and rename should appear atomic to participants accessing the STMF. That eliminates the travails with accomplishing what is an effect atomic member\data copy and removal of the original.

I can not vouch for the efficacy of, with regard to the effective atomicity of, the Move Object (MOV) [aka MOVE] command, but that would be theoretically the more appropriate choice than a copy. That feature is implemented with a copy [effectively Copy File (CPYF); just what might be the choice for a user data copy], but ideally would be implemented such that concurrent operations were unable to interfere with the /from-objects/. And even if the individual object /moved/ is seemingly atomic, and while there is the ability to move a generic [i.e.
effectively *ALL], there is no capability to perform the multi-object operation under commitment control, so the issue with a partially complete operation would exist if using that function. IIRC the documentation [help text] for the MOV command says there is no support for /moving database members/, but my recollection was that the feature worked as long as the CPYF feature could operate on the files, and specifically to operate having specified *NONE for the Format Option
(FMTOPT) parameter; i.e. the record formats were identical.

--
Regards, Chuck
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.


As an Amazon Associate we earn from qualifying purchases.

This thread ...

Replies:

Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.