RE: How to avoid file corruption transferring members from one file to another

Thanks Alan,
I want JOB_2 to copy all members from file WAIT_Q to SEND_Q.

The copy all members step is the "point of contention/collision" I want to control.

Each WAIT_Q member has a document JOB_1 was not able to FTP to remote server MOSTLY_UP, which may not be available for minutes or hours.

My current plan uses data area SYNC_DA that both jobs must lock/read/update/release.

JOB_2 must open SYNC_DA, write "COPY_MBRS" and then close SYNC_DA before copying WAIT_Q members to file SEND_Q.

After a WAIT_Q member is copied, JOB_2 removes it from WAIT_Q.
After all WAIT_Q members are copied/removed, JOB_2 writes "ADD_MBRS" to SYNC_DA.

At this point, JOB_2 tries to FTP any/all SEND_Q members to MOSTLY_UP, removing SEND_Q members after successful FTP.

JOB_1 waits for documents to be created and when received, attempts to FTP to MOSTLY_UP.
If FTP works, JOB_1 waits for the next document.
If FTP faild, JOB_1 must read SYNC_DA, looking for "ADD_MBRS".
If "ADD_MBRS" is found, JOB_1 determines the next member name for WAIT_Q, copies the document to the next WAIT_Q member, then releases SYNC_DA.
If JOB_1 does not find "ADD_MBRS", it releases SYNC_DA, and tries later.

Problems like JOB_1 not able to add to WAIT_Q or JOB_2 not able to FTP SEND_Q members will be handled by setting limits and having each job SOS to QSYSOPR.
This problem is a big issue for my company and this project is maybe a general model to automate a problem that has, to date, been manual,
or, in the case of EDI 856 ASN, not handled because the ASN is very time-critical.

Again, thanks to Alan and all on this list (especially Bruce, Doug and Ron for APIs at work, and Jamief for example RPGLE from CODE400 ) .


-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Alan Campin
Sent: Tuesday, October 21, 2014 4:47 PM
To: Midrange Systems Technical Discussion
Subject: Re: How to avoid file corruption transferring members from one file to another

Not sure I see the problem. If you have an exclusive lock on the member when you writing to it how can the other job get to it?

The other thing you might so is name the member by some name like T_xxxxx and never transfer anything with a T_ name. When you finished writing, rename to a regular name.

On Tue, Oct 21, 2014 at 4:03 PM, Gary Thompson <gthompson@xxxxxxxxxxx>
wrote:

Thanks Darryl,
I can see how the trailer record could help in that case, but,
thankfully, I've not yet had that problem.

I'm trying to sync separate programs in separate jobs so file members
made by job 1 can be transferred to job 2 without corrupted data or
dead-locks.

Each job has its 'own file' where members are stored until they can be
processed, but at some point file 1 members must get copied to file 2
and that is the point where I want to avoid contention/corruption.

My first try will be to use a data area each program reads/writes so
each program 'sees' when the other needs exclusive use of the WAIT_Q
file.

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of
Darryl Freinkel
Sent: Tuesday, October 21, 2014 2:41 PM
To: Midrange Systems Technical Discussion
Subject: Re: How to avoid file corruption transferring members from
one file to another

I don't remember how I did this at other sites, but you could write a
trailer file to the target system folder when you have completed
sending the file. The target system could monitor for the file and
then start processing the file when its done.

Sometimes the target system is too quick for the sending system to
complete it processing and the file may not be done by the time the
target reads it and as a result misses a bunch of records.

On Tue, Oct 21, 2014 at 2:40 PM, Gary Thompson <gthompson@xxxxxxxxxxx>
wrote:

I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote

server.

If an FTP connection can't be made to the remote, the ASN document
is copied to a member file WAIT_Q, and the ASN program continues
with the next ASN as our warehouse completes the next shipment.

We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.

I have program FTP_ASN_2 pulling ASN documents that have not yet
been

sent

from members in a file I call SEND_Q. A separate file is used to avoid
contention
with FTP_ASN_1.

When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process
and looks to copy any members from WAIT_Q to SEND_Q and then remove
all _copied_ members from SEND_Q.

My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?

My ideas have so far included:

Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.

Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q
with ALCOBJ.

Looking for any advice or comments about design pitfalls . . .

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
http://archive.midrange.com/midrange-l.

--
Darryl Freinkel
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take
a moment to review the archives at http://archive.midrange.com/midrange-l.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take
a moment to review the archives at
http://archive.midrange.com/midrange-l.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.