03/02/2013
14/07/2011
13/07/2011
Some truly wacky looking C++ code (Boost 1.47.0)
I had to look at it a few times. Cool, sorta wacky.
try_
[
f(arg1)
]
.catch_<runtime_error>()
[
cout << val("caught runtime error or derived\n"),
throw_()
]
.catch_<exception>()
[
cout << val("caught exception or derived\n"),
throw_()
]
.catch_all
[
cout << val("caught some other type of exception\n"),
throw_(runtime_error("translated exception"))
]
26/03/2011
Facebook’s New Realtime Analytics System: HBase To Process 20 Billion Events Per Day
Facebook did it again. They’ve built another system capable of doing something useful with ginormous streams of realtime data. Last time we saw Facebook release their New Real-Time Messaging System: HBase To Store 135+ Billion Messages A Month. This time it’s a realtime analytics system handling over 20 billion events per day (200,000 events per second) with a lag of less than 30 seconds.
Alex Himel, Engineering Manager at Facebook, explains what they’ve built (video) and the scale required:
http://highscalability.com/blog/2011/3/22/facebooks-new-realtime-analytics-system-hbase-to-process-20.html
25/09/2010
11/11/2009
Google Go
Google’s Go: A New Programming Language That’s Python Meets C++
1st blush looks nice. Types good, C like performance good. Built for multi-core etc. I will admit I wish it looked more functionally inspired. I am loving Erlang these days (and to a lesser extent Haskell) so going more imperative again feels a step back…
I will give this one a *go* I think…
06/07/2009
LAgent: An Agent Framework in F# – YES
LAgent : an agent framework in F# – Part I – Workers and ParallelWorkers
LAgent : an agent framework in F# – Part II – Agents and control messages
LAgent: an agent framework in F# – Part III – Default error management
LAgent: an agent framework in F# – Part IV – Custom error management
LAgent: an agent framework in F# – Part V – Timeout management
LAgent: an agent framework in F# – Part VI – Hot swapping of code (and something silly)
05/07/2009
Good reads…
Been reading lots…
Finance, microstructure:
- A Dynamic Model of the Limit Order Book
- liquidity supply in a pure limit order book market
- Optimal liquidation against a Markovian limit order book
- Fluctuations and response in financial markets
Languages:
- Expert F# Hardback (Expert’s Voice in .Net)
- Programming Clojure (Pragmatic Programmers)
- Why Functional Programming Matters
Distributed computation:
Just good readin’
17/05/2009
Axum Genetic Algorithm Example
I have worked for some years on genetic, evolutionary, memetic and other optimisation software – specifically in the context of automated/ systematic trading. I have also worked for some years on very multi-threaded software systems. I have been interested in the actor model for writing concurrent software.
Axum
Axum anincubation project at Microsoft implementing the actor model for the CLR. Others have posted on it – Matthew Podwysocki has two posts here and here that are worth a look. The Axum team blog is here. Have a read as you can, they are much more ‘on it’ wrt Axum than I.
Genetic Algorithms
Genetic algorithms are a class of algorithms from the school of ‘evolutionary computation’. I will not go into a study here, just google genetic algos. Needless to say – this example is a very naive example. I tried pretty hard to use any of my ‘real’ evolutionary algorithm libraries, but they just did not work easily enough with Axum. More on that later…
Caveats
Aside from being a really naive GA, the demo is also ver imperative and array/ for loop based. This is not good – but I was focused on getting Axum running. After I was unable to pull in my real evolutionary algo libs I just went for bare bones. We are also doing a ‘pure numeric’ optimization problem, this is a common test alog for testing multi-objective optimizer software. It was easy to convert to axum – so I used it. Perhaps for future examples I will try to apply a GA in a more interesting way (eg; not just a maths problem).
High level steps I followed to getting the demo running
- Define the data elements flowing between actors. I 1st tried to use a class called ‘individual’ from my standard libs. Axum has a concept called schemas. Here is a post from the Axum team blog on schemas. Schemas are data transfer objects or ‘schematized messages‘ that are immutable and used to travel in a type safe way between agents, across channels.
- Next I defined the channels. Channels are aptly named, they are typed channels that your messages (those schematized object instances from above) travel through from one agent to another.
- Next I defined the agents that received messages from a channel, and published messages back onto a channel. In the agents are any ‘real code’. the schemas and channels are just construct definitions to establish what can travel around.
Here is my very simple Individual schema. Notice some things were commented out. There is a coerce function in Axum that will take a class that has the same shape as a schema and turn it into one. I had some difficulty with non-trivial objects coercing into their schema counterparts, so I just wound up dumbing down the object and bringing it ALL into the Axum demo. There is supposed to be the ability to share libs – but you must ‘behave correctly’ in your libs. No static data mutation, no side effects etc.
This is basically just 2 arrays of doubles 1 array represents the ‘genes’ the other array represents the ‘fitness’ along N dimensions. In our example, we are using 3 genes and 2 fitness objectives.
Here are our 2 channel defs:
One channel is used to send individuals to for evaluation (look @ the genes and assign fitness values). The other channel is used to take an entire population of individuls and produce the next generation. The interesting bit is that individuals can be evaluated completely on their own in whatever the evaluation enviroment is. Populations of individuals can also be evolved (crossbred, mutated etc) as a unit of work. The classic EA algos have many nice partitions where you can parallelize etc.
The agents
The agents are where all the work goes. The agents look like something from a classic message based system. They listen on a queue for incoming data and they send data out to a queue, the queues are channels in Axum. Here is our ‘evaluator’ agent. The job of this agent is to listen to the incoming individuals for evaluation channel, evaluate them, then send the evaluated individual back to the queue. Here is the simple start:
You can see that it is declared as an agent it also has the channel EvaluationChannel declaration. You can see then that this agent has as its ‘Primary Channel’. The agent goes into a loop in the constructor (sort of odd) then calls ‘receive’ to get messages from its channel. It calls its own Evaluate function then it sends back to the Evaluated port on its channel. The receive call is special in Axum; blocks until there are messages to consume. This is classical ‘producer consumer’ stuff here. The other call with the funny <– sends the data back to the port on the channel.
Note: this is the Kursawe MOP problem evaluator
Functions
Functions in axum are routines that do not cause side effects (commonly called procedures when you do cause side effects). Here are some functions from the above Kursawe Evaluator:
These are the function that evaluates the incoming individual. I have left out some code for simplicity’s sake. I just wanted to show a few things.
- Immutability of schema objects: I am not mutating the incoming individual to the Evaluate function, I am returning a new individual with the same genes as the incoming one and new fitness values that I assign.
- No side effects: Functions dealt only with local variables and returned data without causing side effects.
- Lots of odd casting business: Axum simply had strange problems calling external libs (like system.math in this case). I got many errors like “cannot cast double[] to double[]“. I saw some errors that showed that it seems Axum wraps all the primative types in an AxumSomethingOrOther<primative> type. Again, this just caused ugly code to deal with it + casting galore.
Here are a few more code examples of both the evolution bits (the thing that takes a whole population of indivuals and evolves them into a new population) and the Axum application that coordinates the whole thing.
I have ‘red boxed’ the Axium bits, channel declaration, the receive calls, etc. I have also yellow hilighed another ‘no side effects’ bit. I could not use any ‘standard’ random number generators. Even my custom ‘deterministic’ random number generator because it mod’d static variables. In any case – this random # generator returns an instance of a new RandomNumberGenerator with each call. This is back to immutability. Once I create one of these, it will set its own internal state and return another new rng. This may look strange in the yellow hilighted lines but that is the reason. I actually like that feel. I have been using BclExtras immutable collections lately and it is the same feel, goodness.
This class you can see is getting an input population from its channel, then it is making a new population of individuals to use as seeds for the next population. This is the really naive bit. We do a purely random selection and a purely random crossover of genes from each individual we have randomly selected. So – we randomly couple individuals and randomly select their genes for their offspring. There is no mutation in this example. My real optimizers – I trued to use an optimizer that was fit for solving MOPs (multi-objective problems – problems where you want to minimize/ maximize > 1 fitness/ utility function/ value). Axum just could not integrate too well with a big bit of preexising code.
This final bit is the agent that derives from AxumApplication – it is basically our main. You can see 2 calls to create each agent in a new domain (more next perhaps on that). You can also see the same pattern for using the RNG (immutability). Note that these agents have bunch of your normal mutable state, standard .net-ish stuff in them. They feel very much like normal C#.
Axum issues:
Axum had some general issues with .net CLR level. I think I was trying to program too much in a .net 3.5 way – I believe Axum is 3.0 compliant. Here are my little list of issues:
- Generally red herring error messages like ‘unable to recover’ with no more detail
- Poor handling of arrays (like my above casting issues) in many cases I had to copy values out of the arrays into new arrays.
- When newing a schema type, you can’t use () – just the new XXX {propName=xxx} if you put new XXX() you would get truly odd errors.
- Generally high difficulty integrating with other C# libs. I tried 1st to interop with a big lib, it barfed, I then tried to have another small C# project in the same solution, I got null pointer errors when adding ref’s to it, etc. I had all kinds of errors using external types -just too hard or not possible.
Closing thoughts
Although a simple example, it was fun. I missed having Resharper also 
. I will likely continue on and try to get a more ‘real’ optimizer running using Axum and report back. Love the model, I am skeptical that much will make it into the CLR 
I hope some does…
. I will likely continue on and try to get a more ‘real’ optimizer running using Axum and report back. Love the model, I am skeptical that much will make it into the CLR I hope some does…25/04/2009
03/03/2009
Semi-Random thoughts
I have talked with people for some time about work I had done in the past with copulas… today saw a good ref on Slashdot:
The Formula That Killed Wall Street
I do not like PowerShell.
I still love Rake for building/ running .net apps
Visual Studio crashes all the time when editing XAML (it hates custom attached properties I think)
Xceed’s latest release of their grid is replete with sucky bugs
I feel overly awake at 11:30, that is what happens when you go to the gym at night
The global economy sucks, everyone is feeling it. I hope the low is not so low we can’t all bear it. I really do not want my wife and kids to be truly effected by it.
I love my iPhone and my MacBook Air
I am pissed that my skateboard went missing during our move
I love living in London
I have been thinking a lot about WPF, what are good practices for larger apps wrt managing XAML/ templates/ styles etc
I want a good lock free (or at least low, low lock) hashtable for C# (not a bad place to start Julian M Bucknall)
Windbg rock
The latest Ants profiler is pretty impressive, well done
OK, that is enough
27/01/2009
Update
Well, it has been 55 days since I posted last – that is a record for me! As of my last post, my family and I moved to London. Since then, we have lived in Knightsbridge for a few weeks, gone on holiday to Austria, moved to Chelsea, unpacked our goods, furnished rooms, hooked up bills of all sorts… and so on.
It has been a whirlwind of activity, expense, fun and overall goodness. I have been ‘FaceBooking’ all my small, personal items regularly – it takes very little effort. Now that that business is all through, and life is getting back to ‘normal’, I will be blogging again about my normal technical fare.
Currently I am building some good software using WPF and the other MS stuff. I am also getting back into using Rake for my build and general scripting automation. All in all, things technically have been much the same. I plan to be elaborating more ASAP!
For now, here we are having fun in Austria 
-Chris-