RE: Event files and business integration (was Re: Monitoring a file forchanges from Java?)

["John Taylor" <lists@xxxxxxxxxxxxxxxxxxxx>]

Hi James,

I've built a commercial product which handles this type of event processing,
so I have a bit of experience which might help you.

To begin with, you need to think of an Event in more generic terms. It is a
data entity in itself. The data entity "Event" has at least the following
basic attributes:

- Event Instance ID
- Event Type ID
- Event Creation Timestamp
- Event Data

The event processing framework typically provides one or more servers (aka
"engines") which are responsible for the capture, storage, routing,
dispatch, audit, error reporting and cleanup of all events. These engines do
not know or care about PF record formats, keys, or anything else in your
business domain. All they understand and care about are Events.

Business information such as PF keys, PF record images, "Order Number",
"Customer Account" and so on, are conveyed between Event Producers and Event
Consumers. These producers and consumers are not part of the framework
themselves, but are rather more like "plug-ins". The Event Framework exposes
interfaces for both the generation and notification of events. 

Specific business information is passed from Producer to Consumer via the
"Event Data" attribute of the Event Instance. In some cases, this will be an
entire record image, but most often it is a specific data structure which
provides only the information required to process that specific event.

It's not practical to try and decide beforehand whether this event data
should consist of record images or key data only. Such decisions should only
be made when defining a particular event type. An event which defines a
"Sales Order Placed On Credit Hold", is quite different from an event which
defines "Printer Waiting For Forms Change". 

A useful implementation of such a framework does typically generate a
sizable data store because those events must have persistance for audit
control, error reporting, recovery processing etc. The framework should
provide a means of automatically pruning it's datastore; for example by
periodically deleting all events over a given age.

I hope I've been able to provide you with some helpful suggestions to think
about.


Regards,

John Taylor


> -----Original Message-----
> From: java400-l-bounces@xxxxxxxxxxxx 
> [mailto:java400-l-bounces@xxxxxxxxxxxx] On Behalf Of James H H Lampert
> Sent: Monday, December 13, 2004 12:26 PM
> To: java400-l@xxxxxxxxxxxx
> Subject: Event files and business integration (was Re: 
> Monitoring a file forchanges from Java?)
> 
> Does anybody here have any business integration experience?
> 
> My file monitoring experiment has an ultimate goal of 
> developing part of a business integration adaptor, the part 
> that handles outgoing events.
> 
> I'm faced with a decision, though:
> 
> If I put complete record images into the event file, the 
> event file records could become quite large. As in multiple 
> kilobytes apiece.
> Especially if, for whatever reason, we keep both before and 
> after images.
> And even after the event has been passed on to the business 
> integration server, we're still stuck with a huge record in 
> the event archive. But if I put record keys in, instead, I 
> end up with the following potential
> scenario:
> 
> Suppose the outgoing-event-processing thread is handling a 
> flurry of events. Event 23 involves a change in field FOO of 
> record 47 from a value of BAR to a value of BAZ, while Event 
> 42 changes FOO of record 47 from BAZ to QUX. Suppose further 
> that Event 42 occurs before the thread has gotten to Event 
> 23. If we have a complete record image in the event file, 
> then we capture both the BAR-to-BAZ transition from Event 23, 
> and the BAZ-to-QUX transition from Event 42. But if we only 
> have keys in the event file, then we end up seeing the 
> results of Event 42 twice.
> 
> Am I straining at gnats here?
> 
> --
> JHHL
>