Chris Donnan : Programming - Brooklyn Style

The basic parts of an XML message “content based filtering” event processing system2
chrisdrop posted in ESP/ CEP on September 26th, 2006

I have been reading the excellent Distributed Event Based Systems (lets call it DEPS for brevity) from Springer. These guys publish some of the absolute best cutting edge computer science work in the world. 2 of my favorite references that I have used for work in the past:

Recent Advances in Memetic Algorithms
Evolutionary Computation in Data Mining

These were phenomenal references while implementing serious cutting edge software. The academic research into optimal algorithms and strategies for computing complex ‘things’ is pivotal in implementing a world class solution. We are in the information age and we have SO many resources available to us. So much research, so much published material etc. It is silly not to use these resources for our real world software solutions!

Anyhow - onto my main point - the basic nuts and bolts of a content filtering event processing system - as stated in DEPS (somewhat paraphrased of course).

The Simple Operations

• Publisher - Publish( a message )
• Subscriber - Subscribe( a subscription )
• Subscriber - Unsubscribe( a subscription )
• Event Broker - Notify( a subscriber, a message)
• Publisher/ Event Broker - Advertise( a message schema )
• Publisher/ Event Broker - Unadvertise( a message schema )

The Collaborating Parts

Subscription
-a subscriber
-a message filter

Message Filter
-some predicates that the message should satisfy

Event Broker
- the part that gets the messages, tests filters, and notifies qualified subscribers

Publisher
-any part that publishes a message to the event broker

Subscriber
-any part that subscribes to some message filter of messages

Thats it - all done…..

So - in essence, it is pub/ sub of some XML messages via a broker/ daemon process (sounds like tib rv eh). From there the ‘content based filtering’ comes into play. How do we sort out the relevant bits from the stream of XML messages. Whereas Tib Rv gives you topic based filtering - you have to receive the message and open it up to get any more filtering. The ‘content based’ part winds up being key.

The trick with all of this will be to make it scale out. Seems like a lot of XML parsing. The real magic will be in the broker layer(s). There are myriad topographies described in DEPS. Being able to correctly scale out the broker tiers is a task with many smaller parts.

• Covering algorithms to decide if a message matches a filter
• Matching algorthms to decide what messages match
• Routing and delivery algorithms
• Inter-Broker semantics (how to deal with cycles, etc)
• Event scoping
• Routing table management
• Efficent XML parsing

The list goes on and on. All of the rest is making it efficent. This is an interesting set of prospects. It seems there is a sufficent body of research and a number of academic implementations (Jedi, Rebecca, Sienna, and more). The commerical ESP folks seem to have taken it all a more (Stream) SQL route. All interesting. I am curious to see if any commerical products are out there for plain XML message ‘content based filtering’.

More Eventually;
Chris

Parallel and Distributed Programming using C++0
chrisdrop posted in programming on September 26th, 2006

I just wanted to give this book a ’shout out’. I have been plenty of reading on Safari and I this is one book that I really liked.

Parallel and Distributed Programming using C++

It is a great raw overview of processes, threads, pthreads, MPI and some other stuff. I am not really a big fan of PVM implementations - and CORBA is ..dead.. but - the rest of the book was really excellent. I recommend it to anyone trying to work with *nix and multithreading, interprocess communication, or full on distributed computing. It has a great bottom up approach to explaining it all. It has great code references and is written in a very understandable way. Check it out
-Enjoy-

Chris

Wombat pushing the envelope and SUSE Linux Enterprise Real Time2
chrisdrop posted in linux, trading on September 20th, 2006

I was doing some of my normal reading this evening. One of the more interesting things that I was reading was - again - regarding some news from Wombat today: Wombat Raises the Bar Again in High Performance Market Data Race on Intel, Novell, and Voltaire Technologies. Interesting - read the news - I found it while reading at another publication that I have read front to back since last night - AutomatedTrader.net. what a fantastic new publication - really!!! Anyhow I am off topic now. In short - Wombat is king in low latency data - enough said!
The Wombat speed records are awesome - but the real purpose of this post is to talk about SUSE Linux Enterprise Real Time. My main desktop has been Suse for ~the past year. I have had lots of attempts at making my MAIN machine linux over the past several years, but Suse was the 1st linux distro that let me make it my primary OS. I do not know how I missed it - but man is this real time Suse interesting! This statement sums up what they are shooting for here “SUSE Linux Enterprise Real Time, is the low-risk, high-performance, real-time Linux solution for complex financial services applications.”. Red Hat has such a huge enterprise finance market share - I am curious to see what happens with Suse Linux Enterprise Rea Time.
Some bullet points:

True Real-time SUSE Linux

Built on the SUSE Linux kernel, Real-Time Extensions is specifically engineered for real-time enterprise computing needs. Real-Time Extensions provides a single-kernel environment that directly controls all system operations. Complex real-time applications require that file I/O, networking and graphics be controlled deterministically together with real-time tasks. Only Real-Time Extensions ensures this determinism.

Guaranteed Response Time

In multiprocessing systems, Real-Time Extensions allows individual CPUs to be shielded from interrupt processing, daemons and other Linux tasks. Processor shielding provides a highly deterministic environment where interrupt response and transaction execution time is guaranteed. Real-Time Extensions offers a guaranteed kernel response time of less than 30 microseconds.

Scalable SMP and Processor Shielding

In tightly-coupled symmetric multiprocessing systems, SUSE Linux Enterprise Real Time allows individual CPUs to be shielded from interrupt processing, daemons, bottom halves, and other Linux tasks. RTE processor shielding provides a highly deterministic execution environment where interrupt response and the time it takes to execute a transaction can be guaranteed.

Multithreading and Preemption

The SUSE Linux Enterprise Real Time Linux kernel allows multiple processes to execute in the kernel simultaneously. The kernel protects key data structures and critical sections of code with semaphores and spin locks to preserve system integrity.

Processes executing in the RTE kernel can also be preempted, i.e. forced to relinquish a CPU involuntarily. The RTE kernel can transfer control from a lower priority process to a higher priority process except when the lower priority process is executing in a critical kernel section. To provide deterministic response, many critical sections of the kernel have been tuned and optimized to dramatically shorten non-preemptable conditions. These changes are key to allowing a high-priority process to respond immediately to an external event, even when the CPU is currently in use.

Semaphores internal to SUSE Linux Enterprise Real Time also support priority inheritance to prevent priority inversion when multiple threads of an application are competing for operating system resources.

Single Kernel Environment

SUSE Linux Enterprise Real Time is a standard Linux designed to fully support time-critical applications. RTE provides a true single-kernel programming environment that directly controls all system operation. Complex time-critical applications often require that high-speed file I/O, networking and graphics be performed deterministically together with real-time task scheduling. Only RTE’s single-kernel design ensures determinism and high-speed performance of all these features.

Frequency-Based Scheduler

SUSE Linux Enterprise Real Time’s Frequency-Based Scheduler (FBS) is a high-resolution task scheduler that enables the user to run processes in cyclical execution patterns. FBS controls the periodic execution of multiple, coordinated processes utilizing major and minor cycles with overrun detection. A performance monitor is also provided to view CPU utilization during each scheduled execution frame.

Real-Time Clock & Interrupt Module

SUSE Linux Enterprise Real Time supports Concurrent’s Real-Time Clock & Interrupt Module (RCIM), a multifunction PCI card designed for time- critical applications that require rapid response to external events. Eight programmable timers and twelve input and output external interrupt lines are available. Any interrupt source can be distributed to other systems for synchronizing multi-system applications. The RCIM also includes a high-resolution synchronized clock to provide a common time base across multiple systems. A GPS option allows system time to be synchronized with GPS standard time. High-stability crystal oscillator options provide for accurate time keeping without an external time source.

Needless to say - I will be following SUSE Linux Enterprise Real Time closely! Leave it to Wombat to be taking advantage of all the possible leverage points available. They are a great example of smart, motivated, experienced people - in the field - delivering empowering technology.

-Chris

Cantor Fitzgerald moves to real-time data analysis with Kx Systems0
chrisdrop posted in Finance, trading on September 19th, 2006

Cantor Fitzgerald moves to real-time data analysis with Kx Systems

As I mentioned in my post on Real Time Data Mining the ability to rapidly analyze lots of high speed incoming data is key.It looks like KX has done it again with Cantor. I know Danny from over at another of my favorite companies, Wombat spent some time over at Kx. I First became aware of Kx maybe a 18 months ago when a trader I had been working with sent me a link to their site. I have been following them and it seems they just continue to shine. These guys are my picks for huge winners in the financial IT services/ software sector. They are all in fact - already successes, but I think it is only the tip of the iceberg for them. I believe it is beyond question that these companies will be able to ride the exponential increases in the automated electronic market space. These players are best of breed. I cant wait to see how they all do!

Kx
Wombat
Progress Apama
29 West

I was also interested to see in the current issue of Dealing with Technology that IBM is planning on entering into these areas. It will be interesting to see how a huge player can function in an environment with relatively small companies that have been innovating for years already. My bet is on IBM paying to catch up by buying some market leaders.
-Chris

PS - cool magazine Automated Trader - Journal of Automated and Algorithmic Trading

Core Software Development Values1
chrisdrop posted in patterns and principals on September 19th, 2006

I am approaching my final day on my current project. I am leaving another software effort at an investment bank tomorrow. I was writing up some documentation for the people ‘picking up the torch’. The person I spent my time with today was great. He is bright and experienced. He asked great technical and business questions. The gentleman that I am refering to does not have the same experience set as I have obviously, so there needed to be some context set up. Much of the work that I have done was done out of my core software development values and beliefs about what is important in software to me.
One thing I find interesting is that I am often put in a position of validating my core software development values when working with new people. Thankfully, given time to put my money where my mouth is - they are eventually convinced . Why did I do A or B? Why do I have such and such a pattern or recurring theme in my work? Why were certain decisions made or evolved towards. Here is a quick list of items that we discussed. This is sort of a random list -but these are things that are worth valuing.

• Value delivering high quality software to the client rapidly
• Deliver iteratively - work closely with the clients/ users to make the software be what they want/ need
• Value common metaphors in your software design so that it is not all ‘ad hoc’ in structure.
• Remove dead code
• Value readability and understandability
• Value the DIP because of its ability to support the OCP (see Principles and Patterns)
• Boldly refactor the code as it is appropriate
• Make a repeatable logical seperation in your solution
• Give classes, methods and components single responsibilities
• Tolerate NO redundant code
• Start as simple as possible, complexify as needed and continually maintain with refactoring
• REALLY value composition over inheritance
• Inherit to be reused, not to reuse functionality from some super class

I think there are a few folks that have summed up core software development beliefs that I have come to share. Robert C Martin deserves much credit. I think one of the best papers out there for software developers is Principles and Patterns. I cannot say how many software developers I have refered this paper to and how many of them have come back to me in the future to tell me how much it affected them. Integrating these values into your beliefs about ‘what is important’ as a software developer - will better your ability to deliver effective software solutions - I promise. This paper will give a name to the some practices you have learned over the years and if you are lucky show you some things you have been missing.

These books:

Martin Fowler - Refactoring: Improving the Design of Existing Code
Eric Evans - Domain Driven Design
Joshua Kerievsky - Refactoring To Patterns
Andrew Hunt and David Thomas The Pragmatic Programmer
Mary and Tom Poppendieck Lean Software Development

These people have said it before me. There are plenty of other excellent books that I have talked about. I am a huge book lover. That said - these books are the ones for core values. Get the underlying themes for these books and you will be seeing software more deeply. Understanding the impetus for when to do what, when to apply what pattern is what experience as a software developer gives you. Patterns are all well and good, but understanding why they exist and the principals behind them is the link that many software developers are missing.

My advice - focus on core values and principals. When you understand core principals in software development - your solutions will have a new level of maturity. Applying patterns is easy. Typing code is easy. Making lsustainable solutions that scale and are maintainable is a whole different challenge. Building from core principals will make the difference.

-Chris

Python or Ruby in your C# application0
chrisdrop posted in ruby, dynamic languages, python on September 15th, 2006

I am looking forward to embedding some modern object oriented scripting in my upcoming applications. IronPython is the best candidate currently. Spring.net as an ‘expressions’ framework. It is basically an Antlr based executor of string C# code - see here. This comes with the runtime burdens (non-trivial sometimes) the code management issues of Antlr (C# code generation). It is also not a complete language; you cant use all of C#.  I have used the expressions stuff in certain conditions. I have also done a few variations of my own reflection based ‘mini-languages’. I have done my own operator precedence parsers, etc. Using a more complete - powerful scripting language like Ruby or IronPython seems great… Here is a brief article about how to embed the IronPython engine in your C# application.
I will also mention that Ruby CLR has come a long way. I have been following both the Ruby CLR that John Lam has been working on, as well as the Gardens Point Ruby CLR Compiler project. I would say that IronPython is certainly ahead, but they will be there soon (I bet Ruby CLR before Gardens Point).

I believe that embedding these languages in my larger .net applications at the approprate layers of abstraction will enable removal of plenty of, reflection code and XML configuration. These are often used in conjunction to allow some extension points. Expressions and my mini-languages have been useful here. Class types implementing an interface and the concretion type in some XML config file is SO common. Moving more of this to a LANGUAGE like Ruby or Python will be welcome!

-Chris

The Hull Book0
chrisdrop posted in Finance, Derivatives on September 13th, 2006

I have been getting into the oft sited John C Hull book  - Options, Futures And Other Derivatives.

This is basically the de-facto reference to … Options futures and other derivatives, as you may have guessed. Here are a bunch of great slides that cover lots of the book’s content - plus pretty pictures   Visual aides are always particularly useful. I am well versed in futures, many option types/ strategies, many credit instruments, and FX based futures and forwards. I can certainly beef up my knowledge about when exactly someone would really apply many of the more complex techniques. I mainly want to learn the more complex exotic options and composite multi-asset strategies. While this is viewed as an excellent ‘introductory text’, it is also viewed as a ‘keep it by your side’ tool. I am sure that it will come in handy in years to come.

-Enjoy

Chris

The future of the financial markets0
chrisdrop posted in Finance, trading on September 10th, 2006

Via Mostly I found a paper from IBM called “The trader is dead, long live the trader!”. What a great paper, I can’t believe I missed it to this point. I have for some years been working at auto-trading with a few clients/ partners. More specifically - I had been working on optimization software to train, test, verify and moreover tune auto-trading systems. This paper, and the other elaborate drama referenced in the Mostly post talk about the future of the financial markets. Of course - all predictions must be taken as that - but for years we have been talking about the ideas discussed in the IBM paper.

In essence - the crux of the matter is that once transparency and speed take over - all classes of investors are on an even playing field. The ability to make decisions based on real time analysis as opposed to knowledge gained is crucial. This means that all the more automated systems and decision making processes are key. Pre-trade analyitcs are key. Risk management is of course paramount. Read the IBM paper. Especially in the financial services IT area - it is important to see what the future growth and shrinkage areas are going to be (most likely).

I love this diagram (excerpted from the IBM paper)

So - all you technologists in the finance space - here is what you need to work on. Not only does the paper say that there will be increased spend on ‘vendor utility’ in the finance space, it refers to these as key items that ‘the players’ will be looking for. There are no real shockers here - but this is an excellent set of recommendations to follow. I will say that I agree with much of this speculation.

I was having a chat with a friend of a friend from a large investment bank the other day. He said to me “it is a war between the investment banks. Technology is now, and will be increasingly moreso, the differentator”. Wow - is that the calling of opportunity or what!

A great time to be doing technology in finance - at the cusp of a reniassance.

-Chris

Moving on, new gig, yet-another-investment-bank for me2
chrisdrop posted in programming on September 8th, 2006

I recently had a rash of excellent opportunities come my way. I have always said if you work hard, study your domain, and are passionate about your work; people notice. If you are a leader at heart - people will follow. By some stroke of luck, connections through my accountant, phone calls from old friends, etc, I was lucky enough to be presented with some excellent opportunities. It is amazing that with no effort - amazing work opportunities present themselves. I will say that Wll St is ripe with opportunities for qualified software developers.

I will be moving from consulting with Finetix to a full time role at a major investment bank here in New York as a VP in their Equity Derivatives area. I have spent some time in credit derivatives, fx derivatives, so I guess it is time for some equity derivatives. I will be working on a global desktop front office trading framework. There will ultimately be many desks using this desktop framework/ component set and there is a great opportunity to impact these trading desks and lots of software developers. This is all right up my alley. Needless to say - I am eager to get into it.

Making a move in jobs is always very hard. I am immensely thankful to everyone that was a part of the switch to my new role. I wish there were more copies of me as there were quite a few things I would like to do and would have been given the opportunity to do. Alas - I am only one and I had to choose.

I am also thankful to Finetix and the clients that I have been with during my time at Finetix. Several times I have said about software consulting; “sometimes you are the president, sometimes you are the janitor - either way you are compensated well”. I am sure that president is overstating it and janitor is also an exaggeration, but the sentiment holds true. This is the nature of consulting. A short while ago - I got an email from a former consulting client (a good engagement in the long run) about how their global credit derivatives trading application had rolled out to ~1000 desktops. This project was one of my pets so I am glad of course for it to be in the wild. From what a friend on that project says, the technical goals we shot for are met and tey are reaping the rewards and forthcoming versions of this software should be more stable, faster to deliver, less brittle, and simply more flexable Consulting, you will often miss these times (I had already moved on by the time this project had ‘rolled out’). I will be eager to evolve a portfolio of software over a period of years again. Working as a ‘perm’ at an investment bank will give me this opportunity again.

Anyhow - the details of the whole thing are long and boring - so, I will not get into it. Enough of my blabber.
There you have it;
Chris

Fun Finance Related Technology galore !0
chrisdrop posted in Finance, trading, programming on September 4th, 2006

A quick rundown of interesting tech stuff for the financially oriented software folks out there….

Interesting Article on Lehman’s Fixed Income Grid Implementation. They ultimately went with Platform’s Symphony grid solution after revewing offerings from Sun and the ever popular DataSynapse. Interesting read. Many people (like Matt) have talked about the trend for banks and the like to move from lots of small custom and/ or disparate grid solutions to ‘enterprise’ consolidated grid solutions. I am sure we will see lots of this…..

Here is a great article talking about a few of my favorite companies (hmm). This paper describes a solution that Wombat has put in place for Merrill Lynch in conjunction with 29West. In short - Wombat provides super low-latency data feed handlers for the exchanges of the world. They also provide an excellent set of APIs for the entire publishing, caching, etc of boat loads of market data. 29West’s Latency Busters Messaging product is “a fast, efficient, and lightweight messaging system” used for data broadcasting. Together they have been able to put together solution for ML that is worth reading about.

Marc chats up a topic near and dear to my heart here - the performance of desktop trading applications.

Here are just a few magnificent technologies that are worth using (or at least reading up on!)

Kdb+ (from those Kdb folks)
A more recent fav StreamBase
Been watching these folks for a while after a friend sent a ref to me last year Progress Apama Algorithmic Trading Platform
Vhayu - My same friend sent me a link on these guys last year too. Recent interest in Wombat stuff has brought them to my attention again.
More ESP suff from Aleri Labs

Go if you can - High Performance On Wall St

Subscribe if you can - Dealing With Technology

Enough blabber. I have just had all this stuff in a Firefox session and been reading it all for 2 weeks again and again.

FUN WITH FINANCE AND TECH!

-Chris