I have been reading the excellent Distributed Event Based Systems (lets call it DEPS for brevity) from Springer. These guys publish some of the absolute best cutting edge computer science work in the world. 2 of my favorite references that I have used for work in the past:

Recent Advances in Memetic Algorithms
Evolutionary Computation in Data Mining

These were phenomenal references while implementing serious cutting edge software. The academic research into optimal algorithms and strategies for computing complex ‘things’ is pivotal in implementing a world class solution. We are in the information age and we have SO many resources available to us. So much research, so much published material etc. It is silly not to use these resources for our real world software solutions!

Anyhow - onto my main point - the basic nuts and bolts of a content filtering event processing system - as stated in DEPS (somewhat paraphrased of course).

The Simple Operations

• Publisher - Publish( a message )
• Subscriber - Subscribe( a subscription )
• Subscriber - Unsubscribe( a subscription )
• Event Broker - Notify( a subscriber, a message)
• Publisher/ Event Broker - Advertise( a message schema )
• Publisher/ Event Broker - Unadvertise( a message schema )

The Collaborating Parts

Subscription
-a subscriber
-a message filter

Message Filter
-some predicates that the message should satisfy

Event Broker
- the part that gets the messages, tests filters, and notifies qualified subscribers

Publisher
-any part that publishes a message to the event broker

Subscriber
-any part that subscribes to some message filter of messages

Thats it - all done…..

So - in essence, it is pub/ sub of some XML messages via a broker/ daemon process (sounds like tib rv eh). From there the ‘content based filtering’ comes into play. How do we sort out the relevant bits from the stream of XML messages. Whereas Tib Rv gives you topic based filtering - you have to receive the message and open it up to get any more filtering. The ‘content based’ part winds up being key.

The trick with all of this will be to make it scale out. Seems like a lot of XML parsing. The real magic will be in the broker layer(s). There are myriad topographies described in DEPS. Being able to correctly scale out the broker tiers is a task with many smaller parts.

• Covering algorithms to decide if a message matches a filter
• Matching algorthms to decide what messages match
• Routing and delivery algorithms
• Inter-Broker semantics (how to deal with cycles, etc)
• Event scoping
• Routing table management
• Efficent XML parsing

The list goes on and on. All of the rest is making it efficent. This is an interesting set of prospects. It seems there is a sufficent body of research and a number of academic implementations (Jedi, Rebecca, Sienna, and more). The commerical ESP folks seem to have taken it all a more (Stream) SQL route. All interesting. I am curious to see if any commerical products are out there for plain XML message ‘content based filtering’.

More Eventually;
Chris

2 Responses to “The basic parts of an XML message “content based filtering” event processing system”

1. chris' dev Tidbits, on October 3rd, 2006 at 1:44 pm, said:
   

   My Blogger A-List - a good start feed for .net Developers…

   So, time has come, that I also post my A-list. I just put them together because some of my colleagues…

2. 11p : Sunday, on October 4th, 2006 at 12:00 pm, said:
   

   [...] Chris on XML event processing system. Latest CD Purchases; [...]

Comment on this post below

You must be logged in to post a comment.

You can leave a response, or trackback from your own site.