For the past few years - I have been talking about/ working on what I have been calling ‘real time data mining’. Marc is back blogging again and he has an interesting post here - that lead me to this guys interesting blog. They are talking about ESP - event stream processing. In other places this is also called CEP - complex event processing. These things are an infrastructure technology for my ‘real time data mining’. I will be doing my best to check out the companies/ technologies in this space.
In the past - working on real time algorithmic trading systems - in particular working with depth-of-market data from the CME has shown me a few things:

• Real time data can be plentiful (several GB per day per instrument for depth of market data)
• Real time data can be fast (many coming ticks per second per instrument)
• Real time data can be hard to keep a current state (depth of market data is broadcast up to 20 levels out from the current bid/ ask - you need to keep aggregate calculations based on the current bid/ ask spread which means you need to update your aggregates, scaling, normalization, formulas etc rapidly)
• Get ready to deal with lots of threading

It sounds like having real companies that are able to enable good logical APIs that provide the needed capacity, performance etc. These enabling technologies will enable yet-another layer - the real time data mining layer. I have worked on this for real time trading - and I think this is/ will be a HUGH area for these technologies.

For a few years, I also worked on the 30th and 60th busiest sites on the web. These guys are basically micro-marketers. I was trying hard to drive them towards a real time data mining model. We did manage to get in place (thanks much to the efforts of my friend John) a good set of real time ‘decision logic’ - updateable rules based actions that react to real time and aggregated historic data. This is a step in the right direction. This area can however also be included in the group of companies that could use a decent set of standardized real time ESP/ CEP/ Real Time Data Mining- or let me coin ‘RTDM’. Since the BI space (business intelligence) space seems to have a zillion buzz words/ acronyms - I will add fuel to the fire.
In any case - cool stuff. I will keep working on my RTDM - (particularly optimization and classification)

-Chris

No Responses to “Real time data mining”

1. Chris Donnan : Programming - Brooklyn Style - Cantor Fitzgerald moves to real-time data analysis with Kx Systems, on September 19th, 2006 at 4:03 am, said:
   

   [...] As I mentioned in my post on Real Time Data Mining the ability to rapidly analyze lots of high speed incoming data is key.It looks like KX has done it again with Cantor. I know Danny from over at another of my favorite companies, Wombat spent some time over at Kx. I First became aware of Kx maybe a 18 months ago when a trader I had been working with sent me a link to their site. I have been following them and it seems they just continue to shine. These guys are my picks for huge winners in the financial IT services/ software sector. They are all in fact - already successes, but I think it is only the tip of the iceberg for them. I believe it is beyond question that these companies will be able to ride the exponential increases in the automated electronic market space. These players are best of breed. I cant wait to see how they all do! Kx Wombat Progress Apama 29 West [...]

Comment on this post below

You must be logged in to post a comment.

You can leave a response, or trackback from your own site.