Chris Donnan : Programming - Brooklyn Style

More CEP - Complex Event Processing2
chrisdrop posted in programming, ESP/ CEP on August 31st, 2006

I am seeing all the more focus on ESP/ CEP stuff these days. The need for a layer of abstraction on top of ‘lots of messages’ is strong. In technology we are already pretty good at ‘lots of data’ flying around, but watching  in a repeatable way with some good constructs is still in need. The need to translate some flying data into meaningful events is key. This is particularly relevant in regards to algorithmic trading I might add.
It seems that the general trend in this space is a sort of SQL like query for event flow.

You will see things like (Apama-esque):

or: (StreamBase-esque)
CREATE STREAM TickTriples AS
SELECT symbol, T1.price AS price1, T2.price AS price2, T3.price AS price3
FROM Ticks T1 -> Ticks T2 -> Ticks T3
WHERE T1.symbol = T2.symbol AND T2.symbol = T3.symbol;

SELECT  *
FROM TickTriples
WHERE price1 > 80 AND price2 < 80 AND price3 < 80 AND symbol = "IBM";
The Apama version reminds me of TradeStation which is sort of similar in that it consumes streams of events and it can do simlar event sequence operations. Obviously - TradeStation is specific to market data, it is older and not so advanced as the Aleris, StreamBases, Coral8s, etc… The ‘StreamSQL’ version has obvious SQL lineage.
To my amusement (and a few others I imagine)  - a StreamBase employee and an Apama employee have been ‘going at it’ on one of the CEP email forums. The quibble points are over bold statements about performance, leadership in industry etc. Classic vendor stuff. Long-story-short; the emerging point is that there seems to be no really good set of metrics/ standards by which to compare performance of ESP/ CEP products. That will come soon I bet.
Anyhow - there you have it - CEP/ ESP technology is building. Plenty of tech fun and work to be found here I bet!

-Chris

Bailey Island Maine Vacation Highlights0
chrisdrop posted in non-techinical on August 13th, 2006

Just got back from a most excellent family vacation. Here are a few of my ~120 fav pix (what a dad I have become!).

Real time data mining1
chrisdrop posted in trading, AI/ Machine Learning, programming on August 9th, 2006

For the past few years - I have been talking about/ working on what I have been calling ‘real time data mining’. Marc is back blogging again and he has an interesting post here - that lead me to this guys interesting blog. They are talking about ESP - event stream processing. In other places this is also called CEP - complex event processing. These things are an infrastructure technology for my ‘real time data mining’. I will be doing my best to check out the companies/ technologies in this space.
In the past - working on real time algorithmic trading systems - in particular working with depth-of-market data from the CME has shown me a few things:

• Real time data can be plentiful (several GB per day per instrument for depth of market data)
• Real time data can be fast (many coming ticks per second per instrument)
• Real time data can be hard to keep a current state (depth of market data is broadcast up to 20 levels out from the current bid/ ask - you need to keep aggregate calculations based on the current bid/ ask spread which means you need to update your aggregates, scaling, normalization, formulas etc rapidly)
• Get ready to deal with lots of threading

It sounds like having real companies that are able to enable good logical APIs that provide the needed capacity, performance etc. These enabling technologies will enable yet-another layer - the real time data mining layer. I have worked on this for real time trading - and I think this is/ will be a HUGH area for these technologies.

For a few years, I also worked on the 30th and 60th busiest sites on the web. These guys are basically micro-marketers. I was trying hard to drive them towards a real time data mining model. We did manage to get in place (thanks much to the efforts of my friend John) a good set of real time ‘decision logic’ - updateable rules based actions that react to real time and aggregated historic data. This is a step in the right direction. This area can however also be included in the group of companies that could use a decent set of standardized real time ESP/ CEP/ Real Time Data Mining- or let me coin ‘RTDM’. Since the BI space (business intelligence) space seems to have a zillion buzz words/ acronyms - I will add fuel to the fire.
In any case - cool stuff. I will keep working on my RTDM - (particularly optimization and classification)

-Chris

Does this guy work/ live in the same world that I do?0
chrisdrop posted in c#, programming on August 9th, 2006

C#: Is the Party Over?

Considering that we cannot hire enought C# developers and many Java developers that I know are now doing C# work… I must ask myself; does this guy work/ live in the same world that I do? I actually had to check the date at the bottom of the article to see if it was this year!?
C# is not C/C++, it is not Java. Those languages - and C# - are all execellent (along with many others). I am sure there could be MORE success for C# - but dead!? cmon’. When it dies - I will be happy to use whatever is next, or whatever old new thing is en-vogue again, but this article is just … well … dumb.
-Chris

PS - I am still sitting on my breezy porch in Maine - I love vacation ! I also love that my 2 lil’ ones still nap.

Test Driven Development and Mock Objects Developer Session Slides0
chrisdrop posted in java, Agile Development, User Interfaces, programming, .net on August 6th, 2006

Here are some items from the Test Driven Development and Mocks developer session I did a few weeks ago in NYC. I excerpeted lots of people -see the refs at the end of the slide deck. The main things I wanted to hilight were #1 - how important unit tests are. How much better they work when they are done with other core development practices (IoC, etc), How useful mocks are - and a core mock pattern to use.
BigFatGuyTests.cs
TickTests.java
TDD Dev Session.ppt

Enjoy;
Chris

Ode to Assertions0
chrisdrop posted in programming on August 4th, 2006

Oh assertions, how my love for you has grown over these past years.

I must admit (*sigh*) that I have been seeing less and less of my old friend - Debugger - much to her dismay. I believe it is because you and I have been together more lately.  She was a good friend - but so demanding of my time. She would reveal to me all my inconsistencies and inadequacies. She would point out all of my shortcomings. It would however take her a while to finally get to the point. I would have to push and pull, prod and poke to force her to get it out.

You on the other hand; AS SOON as I make a mistake - you are there - pointing it out for me. You waste no time helping me figure out what I did wrong. I am thankful for this. Even though I HATE making mistakes - I always will. You accept this and you are strong for me.

Thank you Assertions.

I fear however I have begun to obsess over you, just as I have in the past with my old companion - Debugger. The difference may however be disturbing to you - Assertions - my faithful companion. You see - Debugger - she used to use me - now - Assertion - I use you - everywhere. In the past, if I had an issue, Debugger would force me to spend hours with her. I would have to take long walks through the software, coerce and cajole her. Eventually - she would show me what I needed to see. Oh the memories.

Assertions - I have spread you all over my software. Anywhere I think I may make an error somewhere down the line - I put you. I NEED you to tell me what I am doing wrong. This keeps me away from my even more unhealthy relations with - Debugger. Sometimes - she sings her siren song - and calls me back. But usually - with your help - Assertions - I find my way out quickly. You have been my rock.

Oh Assertions - my perverse obsession has lead me to make entire applications dedicated to you - I call these “Unit Test Suites”. These applications consist entirely of you - Assertions. Oh how far gone it has become. You have been so steadfast for me - and yet- I use you to no end - forgive me.

The whole world has gone after you with these “Unit Tests”, but I have not forgotten where you have always served me. You still live in the heart of my software, not just in these “churches built to you” - “unit tests”. You can forever, faithfully be with me both in the heart of my software and in my tests.

Thank you for being such a part of my (code) life. I hope that someday I can repay you.

With love;
Chris

I will leave you with this:

Assert.IsTrue(Chris.HasLoveFor(Assertion));

Apache ActiveMQ0
chrisdrop posted in java, c#, C++ on August 2nd, 2006

Apache ActiveMQ is a fast open source JMS 1.1 Message Fabric which supports clustering, peer networks, discovery, TCP, SSL, multicast, persistence, XA and integrates seamlessly into J2EE 1.4 containers, light weight containers and any Java Virtual Machine together with having a host of Cross Language Clients. ActiveMQ is released under the Apache 2.0 License 

I have been playing on and off with Apache’s ActiveMQ open source message queue project. It was really easy to get the server configured and running. I am impressed by the cross language access - as a ‘real’ message queue  product should/ would support. So far - the java connections were easy as pie. I have been able to download and build the C#/ .net client implementation of their ‘open wire’ format. My 1st crack at getting the unit tests running were unsuccesful so far, but I believe this is because I am running the server instance on a linux machine and the C# stuff is running on a windows box. I have been unsuccesful in reconfiguring their test suite it would seem… so far We shall see how this progresses -  another hour or so and I bet it will work nicely. Hopefully I will be able to get a server instance up and running, send messages to and from using C++, C# and Java. Then I will have met my goal with this project.

You can see examples of connecting with C++ here

C# here

Java here

Theoretically in Ruby via Stomp , Python via stomp

I say that is not too bad - providing it works nicely

-Chris

Loads o’ tick data WITH depth of market !!! Woohoo !!!0
chrisdrop posted in trading on August 2nd, 2006

In the past - it has been very difficult to get good tick data for a breadth of products over long periods of time…. If tick data was hard - depth of market (level 2) data was IMPOSSIBLE!
Lo and behold..

Reuters launches tick history service

“Reuters DataScope Tick History covers global, intra-day time and sales, time and quotes, and market depth across asset classes, covering OTC and exchange-traded instruments. “

In the past - I had to build tools to record tick, level 2/ depth of market data. Then I could play it back and do trading system simulation./ optimization. There were some places to buy tick data from - but we COULD NOT EVEN BUY depth-of-market data.

I wish this was around a few years back! I am certain that some of my prior prop trader clients will jump at this service with all 4 paws, claws extended !

-Chris

JetBrains - TeamCity0
chrisdrop posted in programming on August 1st, 2006

Oh JetBrains how do I love thee - let me count the ways….

IntelliJ
Resharper
Even Omea Reader
and now TeamCity

I am a big believer in team dynamics, automation, tools, continuous integration, code coverage, static analysis, etc. I think if JetBrains does this TeamCity bit as I expect them to - it will be a huge win for teams. In the past I have come onto consulting engagements and tried to just set up this set of infrastructure. Now - I can just say … er - like - um .. buy TeamCity and I will go onto the more noteworthy task of actually building your software

I know a few folks that have had ‘bad feelings’ towards the MS ‘Team System’ - I am glad that someone is trying to put together a real team product.

Of interesting note - the ThoughtWorks folks have also done something interesting… Buildix Also a noteworthy team product. It is not as complete as the JetBrains product - but nice nonethelss - and open source. Throw in Clover, Fisheye and a few more odds and ends and we are getting there

-Chris