Re: Recommendations for a newcomer?

Thanks for that insightful post, Randy. I recall a tour of Bank of America's check clearing data center in Los Angeles. A Huge facility where they process some astronomical number of checks through their readers each day, generating batches of transactions at the same time. The idea of Big Data goes back a ways.

-Nathan



----- Original Message -----
From: Randy Mangham <rmangham@xxxxxxxxx>
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxx>
Cc:
Sent: Tuesday, May 14, 2013 11:17 PM
Subject: Re: Recommendations for a newcomer?

I rarely jump into these "philosophical" discussions but in this case if
we're making recommendations to a "newcomer" I feel compelled to use my
more than 30 plus years of experience (and NOT 1 year of experience times
30) to educate. I've been a business analyst for over 10 years and spent
the remaining 20 plus before that actively designing and coding systems.
There are business needs that simply CANNOT be logically or efficiently
managed in anything OTHER than a BATCH process. Permit me to illustrate.

Without going into overly complex explanations let me point out that my
most recent employer processes thousands of orders a day from branded
restaurants you'd easily recognize at 18 distribution centers across the
USA which are delivered by 18-wheeler tractor/trailer rigs on scheduled
routes with 10 to 20 individual drops per route.  We take individual orders
in our call center and via our web site and, from larger restaurant
operators, files sent from customer central order systems sometimes with
hundreds or thousands of orders in a file. It would be logically impossible
to accurately allocate inventory (very complex rules in our case), generate
picking information that utilizes time and labor efficiently in the DC's
and determine how much product (by volume and weight) can be loaded on an
individual truck (as well as the order in which the products should be
loaded in each of the frozen, refrigerated and dry compartments) in
anything other than a BATCH process.  Nor could we generate load documents
for all the drivers in the correct delivery sequence in anything OTHER than
a BATCH process until ALL the orders have been received for all the routes
since they're coming in over a 36 to 48 hour period. Trying to do such a
thing in an interactive order-by-order process would be absolutely
impossible. Score one for BATCH processing.

In a previous position in one of the world's largest banking institutions
there were internal audit security processes that were run once a day in
BATCH after ALL the transactions for the business day were posted. Those
audit processes read millions of account records and hundreds of millions
of individual transaction records every night to verify the integrity of
the financial data. We were checking to be sure that, as one example, the
difference between opening and closing balances matched to the transactions
associated with each account to be sure that nobody internally was
committing defalcations by manipulating transactions between accounts (but
which didn't force the books out of balance) along with other processes
which my NDA forbids me to mention. Again, it would have been logically
impossible for those audit processes to be executed after every ATM, wire
transfer or EFT transaction (millions of which were being executed during
the business day). Can you imagine, after processing each transaction,
firing off the audit security processes EVERY SINGLE TIME? Of course not!
Only a BATCH process makes sense. Score two for BATCH processing.

Another banking example is that your bank doesn't keep printer files open
for the monthly statements for thousands or millions of accounts and write
each transaction you execute during a month to the individual statement
printer file as it occurs. They generate your statement all at once in a
big BATCH process from the transaction history. It would be impossible to
generate statements any other way. Score three for BATCH processing.

By the way, even when you deposit a check at the ATM, the transaction may
be immediately visible on the bank's web site but unless you've got some
special privilege that I've never seen a bank extend to even its largest
business clients, the money will show on your BOOK balance but isn't part
of your AVAILABLE balance until the bank sends the data extracted from that
check image in a BATCH  file to the bank on which the check is drawn (or to
an intermediary bank that clears checks for smaller institutions) and waits
for the other bank to determine if there's money to cover the check BEFORE
they credit all the money to you. There are very valid reasons for not
sending that check data immediately to the other bank as an individual
transaction and receiving back a "paid" or "returned unpaid" status. What
if the person or business that gave you the check hasn't gotten to the bank
with the matching deposit but will do so before the end of the banking day?
Another example would be If your employer hands you a check, let's say for
expense reimbursement, where it's entirely possible the check is drawn on
an account that is kept at a zero balance to reduce the potential for
fraud. Many businesses send a Positive Pay file to the bank before the end
of the business day for any checks issued that day so that the bank will
know which checks are valid and should be paid. The bank then "sweeps" the
money out of some other account as each check is presented. It would be
seriously stupid for a company's A/P system to send an individual Positive
Pay transaction to the bank as each check is issued second-by-second during
the payment run (which is most often also done in a BATCH process). Score
four for BATCH processing.

I've architected all sorts of applications that performed some serious
processing behind an interactive transaction (such as immediately
determining on hand inventory on a supplier's system, getting credit card
authorizations in retail environments or processing billing data from SABRE
as individual reservations are ticketed to clients) so I'm well acquainted
with such systems where immediate processing is a business need. But just
as it's incumbent upon system architects and software developers to use the
right tools (languages, utilities, etc.) for a given project it's ALSO
incumbent to recognize when implementing business rules in software is best
expressed in a batch vs. an interactive process. We ARE supposed to be
doing what is best for the business and not what WE think is most fun and
challenging to develop.