So – I am deep into Erlang (still learning for > 1.5 years), Getting into Clojure (new) and getting into Haskell (a few months now). I have decided to spurn the OCaml/ F# branch of the world at least for a little while…

Anyhow – Erlang is the functional language I have the most experience with, yet next to Haskell – the LANGUAGE feels weak. Erlang the language + OTP + the Erlang VM runtime is amazing but the Language itself is not as lovely as Haskell. When I am typing C# code, I am finding myself more and more functionally minded. For a few years now I have felt the move to functional programming on me. C# has had great improvements – but it still feels way too brittle and verbose next to Haskell.

I recently read these articles:

Haskell for C# Programmers Part 2: Understanding IO.

Haskell for C# Programmers Part 3: Visualizing Monads

I found them helpful understanding monads as someone with a large C# experience. I also found this one:

Monads as containers very helpful. I will admit to having read dozens of articles on monads and just now really getting to a point where I think I can actually perceive the need for writing one of my own.

Functional programming style has made great differences in my personal style in several languages. I like programming languages and I am literate in many. Seek the monad, but 1st seek to understand the primitive bits of functional programming and it will aide your programming in any language.

LAgent : an agent framework in F# – Part I – Workers and ParallelWorkers

LAgent : an agent framework in F# – Part II – Agents and control messages

LAgent: an agent framework in F# – Part III – Default error management

LAgent: an agent framework in F# – Part IV – Custom error management

LAgent: an agent framework in F# – Part V – Timeout management

LAgent: an agent framework in F# – Part VI – Hot swapping of code (and something silly)

I have worked for some years on genetic, evolutionary, memetic and other optimisation software – specifically in the context of automated/ systematic trading. I have also worked for some years on very multi-threaded software systems. I have been interested in the actor model for writing concurrent software.

Axum

Axum anincubation project at Microsoft implementing the actor model for the CLR. Others have posted on it – Matthew Podwysocki has two posts here and here that are worth a look. The Axum team blog is here. Have a read as you can, they are much more ‘on it’ wrt Axum than I.

Genetic Algorithms

Genetic algorithms are a class of algorithms from the school of ‘evolutionary computation’. I will not go into a study here, just google genetic algos. Needless to say – this example is a very naive example. I tried pretty hard to use any of my ‘real’ evolutionary algorithm libraries, but they just did not work easily enough with Axum. More on that later…

Caveats

Aside from being a really naive GA, the demo is also ver imperative and array/ for loop based. This is not good – but I was focused on getting Axum running. After I was unable to pull in my real evolutionary algo libs I just went for bare bones. We are also doing a ‘pure numeric’ optimization problem, this is a common test alog for testing multi-objective optimizer software. It was easy to convert to axum – so I used it. Perhaps for future examples I will try to apply a GA in a more interesting way (eg; not just a maths problem).

High level steps I followed to getting the demo running

1. Define the data elements flowing between actors. I 1st tried to use a class called ‘individual’ from my standard libs. Axum has a concept called schemas. Here is a post from the Axum team blog on schemas. Schemas are data transfer objects or ’schematized messages‘ that are immutable and used to travel in a type safe way between agents, across channels.
2. Next I defined the channels. Channels are aptly named, they are typed channels that your messages (those schematized object instances from above) travel through from one agent to another.
3. Next I defined the agents that received messages from a channel, and published messages back onto a channel. In the agents are any ‘real code’. the schemas and channels are just construct definitions to establish what can travel around.

Although a simple example, it was fun. I missed having Resharper also . I will likely continue on and try to get a more ‘real’ optimizer running using Axum and report back. Love the model, I am skeptical that much will make it into the CLR I hope some does…

In this segment, we will discuss pushing in singletons that you have already created and adding a simple generic hook point.

Here is our starting test. We want to create something outside the container, then add it as a singleton.

This is trivial, we simple give it a delegate that returns that instance when it is called.

Here our lambda is just going to return that instance. (PS – this is a C# lambda expression, google it.).

But wait, what is wrong here… was it so trivial? Well, how about disposal? Do we want to dispose this thing when our container is disposed? I do not think so… So we can create a policy that does not dispose.

Here is a test:

Let us split our SingletonCreationPolicy into 2 parts – 1 that is the same in both cases – the creation bit, and 2 – the disposal bit we can ‘add on’ in only the case we want it in. This results in:

In our Kontainer class:

 the basic singleton creation policy is broken out (back to):

And finally, the Disposable version of the policy for ‘container managed’ singletons:

OK, so that code allows us to push singletons into the Kontainer and allows them to be disposed correctly. (Note the use of protected auto-properties. Less code - I like it).

Hook points

Many 3rd party vendors have used events as hook points for some time. I think grid control products like DevExpress’s XtraGrid have gone far and wide using events as the primary ‘hook points’ to their APIs.

In order to give similar behavior to SpringFramework’s IObjectPostProcessor, we can add an event for a similar effect. Let us start with a test:

We want to hook onto the point when a user is ‘getting’ an object. and ‘do something’ before they actually get it. In this case, we are getting drastic; we are completely replacing what the container has internally. Most usages would be more benign most likely.

We change our Get method, add a GettingObject event and allow callee’s to each have a go at the returning object. So our Kontainer class changes like:

This will happily allow callers to modify the created object and return to the ultimate caller of Get the result. It id the event listener’s perogative to elect to process the created object based either on the object itself or on the label.

So; there you have a nice way to push instances of objects into the container and a cheap hook point to ‘do stuff’ when the user calls to Get.

More soon;

Chris

In this segment, we will make our stuff dispose nicely.

1st – let us look @ our creation policies in a bit more detail:

Here is the simplest possible one – a prototype creation policy. It will simply call the user’s func each time it is called, passing in the calling container for possible child dependency resolution :

The singleton creation policy is a bit more:

This policy uses the user supplied func to create a value 1x, then returns the value for all subsequent calls.

Now – why is this relevant? Well in order to dispose properly, we need to make our container disposable and implement that so – in normal form – here is a test and a little class to help test:

All we want to do is to dispose our container and ensure that any disposable items are disposed – for creation policies that want it as such.

So – we will add a dispose method to our container:

Our container has a field that is a dictionary of string to Definition. The list of values is a list of definition. If our definition has a disposable creation policy, we dispose it. This is using LINQ to select the values in an orderly fasion and a functional-esque call to dispose each in the ToList().ForEach(lambda) call.

We updated our singleton creation policy to look like this:

So – now our singleton creation policy will dispose singletons when the container disposes. If a caller supplies any disposable custom creation policy – it will also dispose as per their mechanism.

Now – I need to make sure that I throw exceptions ‘at the door’ when users provide bad data, or use the container incorrectly. After that, I need to make a pass at thread safety for the container…

In Part 1 of this short series, we outlined the basic behavior for the simplest possible DI container; what I’ll call Kontainer.

In this segment, we will write some code to make those tests pass. In order to allow us to make those tests pass, I wound up doing the following:

• Keep a list of definitions that are registered with the container
• Abstract the idea of an object definition
• Abstract the idea of a creation policy such that singleton and prototype are instantiations of a creation policy, but we could add other creation policies externally if we want.

This led to the following basic code:

The basic class to hold the definition:

The policy used to govern the creation of the objects in the user provided Function

A slightly modified version of the registry part of the container

and the basic bits of the core container

In order to test sub-property resolution, I also added the following test:

What is that all about? Well, the Function signature takes an argument for the calling container so that the class being built can resolve its own dependencies via the container.

This code was sufficient to make our basic round of tests pass. In addition, it was a little bit of relatively simple code. Furthermore, we can handle other creation policies aside from singleton and prototype. In order to prove that out, I added the following test:

 And the other test creation policy:

This is a ’singleton per thread creation policy’. This creation policy allows there to be a singleton created per thread. It uses primative locking style that we may change when we make the ‘whole shebang’ thread safe. Essentially – this will allow us to just prove out that we can supply alternate creation policies.

In the next part – we will sort our the disposal needs…

It is no secret to people who know me that am a HUGE DI proponent. When using a language like C# or Java certainly – you must do it to maintain a degree of flexibility, separation and manageability in their software.

I have used most of the DI containers. Last- I wrote one @ home over the weekend and brought it in to my last employer. In this series of posts – I am going to write a simple DI container, then go and make it threadsafe using some of Jeff Richter’s ’stuff’.

So how do I want this simple container to be?

• JUST DI, (not broad like Spring)
• Code based (no xml)
• Label based (not just type based)
• As little code as possible

Why those things??

Just DI 

I want it really, really simple. I have explained DI to people for some time and it is often obfuscated by whatever tool we are using. My ‘dumb simple’ explaination is usually something like:

When you are writing a class, and you need some other thing to help you carry out your behavior, in stead of newing it alive, depend on the appropriate abstraction and allow it to be injected. What do I mean by injected? I mean make a constructor arg or allow a settable property

Code based

I just want code – no XML, etc. It is a pain to debug etc.

Label based (below)

As little code as possible

I want AS LITTLE CODE AS POSSIBLE. I want to maintain NO CODE.

Constructor args or settable properties

Well – constructor args give you a ‘guaranteed fully baked class’ (as long as your arguments are ‘valid’ eg; not null, in the proper state, range etc). Changing constructor args can sometimes be a pain. They are also less amenable to odd circular dependencies (admittedly to be avoided anyhow). While I appreciate the value of constructor args – my general experience is that sometimes settable properties are more flexable. Take for example the case where you need to inject dependencies POST construction (in WinForms for example often the designer will ‘new’ your types).

Type based injection or label based injection?

Some friends of mine prefer type based injection. Some of the other current containers prefer type injection as well. Essentially the idea is that you ‘bind’ some abstraction to some single concrete implementation. For example:

// registration
diContainer.Bind(typeof(FooImpl))

// usage
IFoo f = diContainer.Resolve()

What I don’t like about this is that it forces you to have 1 implementation of IFoo in the system. This is OK in many cases, but it fights the ability to have N different implementations of a common interface. Take for example IPresenter or IView or something general like this. Binding a single implementation to something like this is prohibitive.

So – I prefer to have label based lookups. This has its own downside, mainly that you have to look for string names. With string based lookup – it is something like:


// registration
diContainer.Register("A", typeof(FooImpl)) // or some variation


// usage
IFoo f = diContainer.Resolve("A")


.. or something like that. The salient bit is the fact that you are asking for something by name, as opposed to just by type.

So I am comfortable using settable properties sometimes and names/ labels because:

• With tests, you can be a bit more dynamic
• It is generally a bit more flexable IMO
• With autoproperties, it is little code

So – what do I want the simple container to do?

• Allow me to register definitions for singletons using lambdas/ delegates
• Allow me to register definitions for prototypes (factory behavior)using lambdas/ delegates
• Allow me to register singletons that are already created
• Allow me to inject properties on instances of objects that already exist
• Allow me to get objects from the container by name
• Allow me to dispose objects when the container disposes
• Allow me to dispose objects when the container disposes
• Allow me to add behaviors externally via some hook points

Here are the basic starter tests then of Kontainer, my simple container:

With this set of tests, we get the following interfaces:

So – yesterday was my last day in Equity Linked Technology @ Merrill Lynch in New York. I have spent the past 2 years working on building their ‘next generation’ sales/ trading pre-trade trade lifecycle platform. To avoid any confusion or concern – no – I was not let go – I left on my own (it was a hard decision to go) for a great opportunity working at a hedge fund doing just what I want to do.

It has been full of fun, trials and hard work. At the end of the day, I will really miss certain people. Also, it was very hard for me to leave after only 2 years. It took a lot of hard thinking, introspection and consideration to decide to leave with only 2 years in. I am a very ‘goals oriented’ person and one of my goals going into this job was to have a 5 year run. That being said, goals are ordered, and my other goals have overridden this one – so… life is full of trade-offs.

I will not understate how much I will miss my friends from within Merrill. I will also miss a whole bunch of coworkers that I was only fortunate enough to get to know a bit. Many people made a good impression on me. I believe that I was fortunate enough to make a good impression on many people. The small emails I got in response to my goodbye email were in many cases very flattering and humbling. Thanks again to all of those people who thought well of me – your kind words and thoughts are appreciated.

Next…

Well – it has been a VERY long time since I have had 3 consecutive weeks off – but that is what I am doing 1st. I will be taking 1 week around my neighborhood (Cobble Hill Brooklyn). Since Gabe (my ~6 year old) has summer camp every day this coming weel, I will be spending a lot of time with Shannon and Micah (my littler 3.5 year old). I am looking forward to just taking lazy time this week.

After my week of relaxing at home – we are off to California for a bit of a vacation. We will have 2 weeks out there – some in San Francisco, some at Yosemite. We will be seeing many missed/ loved friends out there – so we are really looking forward to it.

Our kids will be seeing some of their long-time good kid friends, so we are excited for them. Each time we go on a family trip – they change a bit. This will be a ‘bigger’ trip for them – so we expect them to change quite a lot. Kids are great fun – they are a wonder. They are also of course quite a lot of work!

…and after vacation??

So – after vacation – I will be going to work for a hedge fund. I will be starting in their NYC office, then within the next 2-3 months (fingers crossed for the paperwork and all). We will be moving to their London office. I was supposed to go work for Merrill in London as well, so this was ‘in the works’ in either case – but this job is going to actually be it.

Me, and why did I move jobs?

Essentially the new job is more aligned with what I want to do. A former colleague called me and said he had something that was a good opportunity. I have known many recruiters for years, and sitting in a desk in a bank, everybody knows there is no shortage of incoming calls for ‘good opportunities’. Jobs coming from people you know and respect in the field however are about the only worthy source of REAL good opportunities of that ‘next level’ variety.

For me – I can either go up the ‘management of technology’ ladder, or go get closer and closer to the business of trading. I have had teams in the past, although only up to ~10 people. At Merrill, I had the opportunity to help move our larger group of equity derivs technology to more agile methods. I have helped many people run their development teams. I like to run dev teams. I am a ‘field marshal’ type according to the Myers-Briggs Type Indicator (see here, or here). It is in my nature to want to go accomplish something with a team. In my current world, I think I am more likely to lead ‘in the trenches’ in a very hands on tangible way. I do not really think this lends itself well to the investment bank hands off management of technology. My current opinion is there is too much politics to be a hands on field marshal leader in an investment bank, or at least at Merrill.

So – that leads me to the new fund. Essentially, I will be working with a very small team with – as far as I can judge – very talented people. The general mindset of ‘hiring few and excellent people’ as opposed to many many many people of varying skill levels rings with me.

Business Proximity

Sitting on the desk – it is important to me to have that immediate attachment to the business. I have spent years working with traders trying to automate parts of their work from your standard ‘enter these limit orders’ to automate an entire trading strategy, to flowing quote requests across global trading desks. I have often said that what I think I do, and what I think is valuable is executing strategic solutions incrementally bit by bit in those more tactical time-frames. In other words – you do not have to hack away day by day on the desk and grind to a halt at some point. You can have a strategy and execute within your own strategic directives for the software and deliver business value rapidly building the platform out bit by bit – ALL THE WHILE delivering business value.

Plinko

Prioritization. It all comes down to executing the business needs in EXPLICIT order of priority. All the while, dumping the answers (the code/ software) into the right strategic buckets. I have called this the Plinko model of software development.

So – that is plenty of chatter – I am off to play with the boys!

-Chris

I have had a good deal of looking at and playing with Coherence for (.net specifically). Several people have been talking about using it in areas near mine. It has a few features that are quite similar to a solution that I have worked on for some time. In any case, I am a fan so far. My question is; is it a good enough event processing engine for the majority of use cases?

Essentially, most use cases for stream processing are solved without things like sliding lookback windows etc. Certainly – those ARE useful features and there ARE LOTS of use cases for them, but I have seen a few classes of applications be solved 100% by types of stream processors that DO NOT do those types of temporal things.

Imagine for instance that you want to see;

All booked trades. This is usually easy enough. Maybe you have some XML on the wind, maybe it is a tib message of some sort (RV/ EMS), maybe you put an object on the dist’d cache. In all cases, you are essentially putting your data somewhere, and notifying existing parties. I think the old standard for this was essentially, map your object to the tib, and to the database. This means you can save, transport/ notify, etc.

Mapping layers

This is of course 2x mapping code Not so great. With Coherence you have to write the code to serialize your classes to the correct forms (similar in many ways to implementing microsofts IXmlSerializeable ala sax parsing looking stuff). In the more recent Coherence for .net, not only can you implement IPortableObject (the serialization methods to get/ put to the cache), but you can also register external serializers. This is excellent in my opinion as you can have a 100% non-invasive solution. You tell the cache engine what class will serialize your given types, and you are good to go.

This is all well and good, but – you are still essentially going to get the 2x mapping problem… 1x for the cache serialization, 1x for the long term persistence (likely a database). This is not so great .

Continuous Queries

The fact that Coherence lets you register a ContinuousQuery is great. The filter syntax is not so pretty:

Filter filter = new AndFilter(new EqualsFilter(“getTrader”, traderid),
new EqualsFilter(“getStatus”, Status.OPEN));

It will work, and it will lend itself to higher abstractions above that lower level AST-like abstraction. It will call you back when your cache gets the relevant insert, update, delete, etc. Good stuff. This is the essence of continuous queries. My basic thought is that is ‘good enough’ for many, many use cases. Sure, the temporal stuff is good as well, but it is really unneeded for many problems.

My least favorite part of this is the mapping layers. These are layers that really do not add any business value, they are insulation, infrastructure, etc. Another missing link with coherence is the history of changes. Each change is going to change the entity in the cache, but the incremental changes, those are not held onto, unless you write code to do so.

Entity Capabilities

I guess this gets me thinking on how important it is to have entities that are;

• diff’able
• merge’able
• support incremental change broadcasting (If you have a 1 mb entity, and you mod 1 field, you do not want to broadcast the whole thing.)

I am a fan – of this Coherence business, but there are still other bits of the puzzle that must be thought about. How are you going to keep an audit history? What will you do for long term persistence/ reporting on those entities? How can you apply a sort of bulk change set to your entity? Etc.

-Chris

CLINQ to NEsper – I want it ASAP.

I will be toying with this ASAP.

Now if you could just make it FAST CONCURRENT RUBY CONTINUOUS – I would be super satisfied… How do we get there?

Ruby and Python are great languages. I have spent several years now with ruby and I am just getting into python in the past year or so. I will still admit that I prefer ruby in every way, but I have come to like and appreciate python as well.

I have been working on building some very interesting global derivative trading applications – so how do python and ruby factor in?? Well – I could have ‘just used them’ in several places. That of course is not valuable, but I have found a few ways to use these tools to really add value to my projects.

Scripting for your existing .net application

I have been making an effort to make our plugin application suite scriptable. I have been able to get IronPython 2 running along with a syntax hi-lighting code editor. It has proved extremely useful. I have also tried to do the same for IronRuby, but with less useful success. Why you may ask – predominantly for 2 reasons:

IronPython vs IronRuby for scripting your existing applications

Basically, IronRuby does not let you do some things that are a must! You CANNOT unsub from events yet . The other big issue is calling existing .net APIs that need generic args. You can see Jon Lam’s response to my queries here. Jon also said:

Our .NET interop is fairly weak today. Because of our current push to get the language running in time for RailsConf, all .NET interop work is being pushed back into June on our schedule.

I understand, but this does make it virtually unusable for using as an existing scripting engine against my current .net applications.

IronPython 2 does a MUCH better job of .net interop. It basically just works as expected in 99% of the cases I have needed it for. This is just a joy I must say. Especially trying to use the early cuts of IronRuby on the DLR, it was painful to get anything interop based between existing .net and IronRuby really working.

Another thing I have done with IronPython is to support ‘calculated formula columns’ in a UI Grid control. We basically implemented this and it is currently ‘not supported’ in production, but in a future iteration we have planned (and requested from users), excel like formula summaries/ aggregations. We will continue this work with IronPython.

So – as you can see – IronPython made itself able to add value.

So how have I fit in Ruby in a meaningful way?

Essentially, we have a plugin framework so that several teams can concurrently develop ‘independent’ applications that can integrate. This means that there are several global teams developing plugins (think eclipse) that can integrate, use and extend one another. This is great, but it means there are all kinds of build, assembly versioning and other application packaging, deployment issues.

In any big bank, or other corporation I expect, there will be different ways of doing things. The continuous builds for differing teams may be different. Some are nant based, some are simple, some are complex, some are msbuild, some do a better job of CruiseControl setup than others, etc, etc. Now if you want to take the work from several teams like this, and build, package, test, deploy in a sane way, there is some orchestration needed. This is where I have used ruby to add real value.

Ruby, Rake/ Python, BuildBot

Rake (and plain old ruby, not j, iron, or another variety) have enabled me to have a single homogenous view of the otherwise complex world. It is basically my tool for integrating multiple teams that are doing things as is appropriate in their own area, but not necessarily what is the ‘best’ for use by everyone else in the world. Ruby has REALLY shined here. In stead of having dozens of XML, Nant, MSBuild scripts, MSBuild extensions, Nant extensions, cruisecontrol configs, bat files, etc – I have ‘owned’ them all with some rake build scripts. I have managed to do this with some ~200 lines of terse, easy to read ruby.

Interestingly, a smart colleague of mine has done something very similar with plain-old-python (and BuildBot). He has managed to take some 50 projects that can be put together in varying ways, built, deployed, etc. He has taken a very difficult problem (managing a large enterprise suite of .net of branched, versioned and VERY WIDELY UTILIZED software) and owned it with some 400-500 lines of python.

What did we get for our efforts with python and ruby?

Number one is just the joy of really getting into daily use of other languages. I have programmed several languages over the years from C/C++, to Java, to SQL, Javascript, Tradestation EL, to Python and Ruby. Working with new languages is a good experience – bottom line. I have had a lot more time with ruby than python. I always love ruby, and I have really enjoyed more real time with python in the past year. I think these languages have each made me a better programmer.

With regards to scripting applications, these languages, in particular IronPython have just made it too easy to add value fast. I have done the work with ANTLR, regex garbage languages, compiling C#, etc. These approaches ALL take much more effort to get similar rewards. I can see myself working in a mode of ‘core plumbing’ in C# and/ java and the higher level stuff in IronRuby JRuby.

That reminds me, a small note. Since we are java server/ .net client apps, I have used JRuby on several occasions to quickly do some send messages around the network, and otherwise mimic stuff coming from the server. I have had more effort on the client/ .net side of things, but I am sure if I was doing more server work, I would be using JRuby all the time.

Regarding build and integration – the big thing that we got IMO was a homogenous solution. Whereas before, there were the xml files, bat files, etc. – we each now have small bits of easy to manage code that does all we need. This is coupled to a small degree of ‘convention over configuration’, but essentially, we have gotten a lot of value for a little code and effort. We have also gotten easy to maintain solutions going forward.

Downsides??

The only real downside here is ‘adding more tools’. There is always resistance (appropriate more or less) to adding to the enterprise overhead of ‘more tech’. Developers also need to learn more to work in more ‘polygot programmer’ environment. So the ‘more tech is bad’ and ‘devs need to learn more’ arguments are valid arguments. That being said, I think the benefits are outweighing the downsides so far.

The future?

I see more coming in the same spaces. I will be working more on ‘higher level scripting’ of existing .net apps, and build/ integration etc. At some point, I can see doing ‘it all’ with these languages perhaps – but I think the current sweet spot in the stack is adding value at certain points of the abstraction. The right tools for the right job and all that.

I just had to mention this one- Rabbit MQ:

Built on the Erlang OTP
Implementation of AMQP
Java and .Net implementations
Open Source (similar to qpid)