Why blog about Erlang now?

For me writing blog posts is often a therapeutic way to work out puzzles and clarify my own understanding of a concept. I have been writing Erlang on and off for around 1.5 years now. As of late I have also been typing a lot more Haskell and some OCaml. This has all been great and I love these languages. I have decided to commit a bit more to Erlang lately and I thought I would start blogging again in earnest – mostly with small-ish Erlang examples just to force myself to be able to explain the concepts. I can program in Erlang (to some level at least) – now I want to be able to be good enough to explain Erlang software to people.

Essentially – I think that erlang the language is OK, I think that Erlang the runtime is amazing and I think that OTP – the common set of Erlang behaviors is very powerful. The entire ecosystem puts together a very compelling offering. You can achieve massively parallel, manageable bits of software in small enough bits of code that it is ownable. So – Erlang it is…

So in that spirit – I will start here with some very simple Erlang concepts.

Erlang is Dynamically Typed

Erlang is a dynamically typed language that to be honest – I found pretty ugly (syntactically) at first. I have done plenty of programming in dynamic languages – especially in Ruby (and to a slightly lesser extent – Python). I really do like dynamic languages but to be honest; I do believe that in larger software projects today that static typing can really by you a generally higher degree of assertable ‘correctness’. I prefer Haskell’s type system to many others – but my basic point is that I think types are powerful. In any case – I still think Erlang’s total offering is powerful enough to make a concede this point.

While Erlang is dynamically typed it does have a simple way to organize primitive data elements into composite types – this is Erlang records. You can also of course use tuples (a sort of anonymous type). In addition, Erlang code is organized into modules. So – while we may organize our code differently than in other languages – we have some good organizational concepts.

Modules

-module(exchange).

Records and Tuples

As I mentioned – erlang lets us organize primative elements into composite data types. The anonymous way of doing this is tuples. The ‘named’ way of doing this is records.

-record(order,{market, price, size, side}).

This is the declaration of a record in Erlang. You can declare an instance of a record like this:

#order{market=vodln,price=127.95,size=350,side=bid}

It is essentially a nice dress on top of a tuple which would look like this:

{vodln,127.95, 350, bid}

So – records are really just nice ways to put together composite types with names. Tuples are ways to assembly composite types anonymously. Both have their usages.

Object Orientation vs Functional Programming

Notice that these records/ tuple things have no methods ‘on’ them. Typically when you declare a type in OO languages – you put methods on them. This whole bundling up of methods and data IS OO programming. Functional Programming lets you have data and describe functionality. What this means is that it is usually easier to add more functions on a given data type in a functional language – but harder to change the data structures. This is because in an OO language – the data is WITH the functionality. In a functional language the data is apart from the functionality. It is all a tradeoff, but I think the functional paradigm is often much closer to the domains I have been working in (trading specifically). So – FP takes away the general encapsulation of data/ methods together. Note that you can still hide private functions inside a module. Eg; you implement a module and export 1 of 10 methods. This is still a good way to expose a minimal surface area of your software to the outside consumer of it.

Functions

Functions are – as you may have guessed at the heart of Erlang.

levelFor(Order, Exch) when Order#order.side == bid ->

levelForSide(Order, Exch, bid, fun(E) -> E#exch.bids end);

levelFor(Order, Exch) when Order#order.side == offer ->

levelForSide(Order, Exch, offer, fun(E) -> E#exch.offers end).

Here we have an example function that I will use as a demonstration for several points.

• You declare functions in erlang as a top level item. You do not put them inside of records, you put them in a module – but that is it.
• Values/ Parameters all start with upper case letters (I found this very odd at 1st).
• You can declare the same function N times each with different ‘guard clauses’ and the order of the declrations is the order in which the runtime attempts to bind to them.
• This example shows how to extract values from records.
• This example shows  simple funs- anonymous functions.

Declaring functions, Guard Clauses, Extracting data from records

“levelFor” is the name of our function. This function takes 2 arguments named Order and Exch. The arguments come after the name in the (…).

After that is a guard clause – this ‘when’ is evaluated by the runtime to see if the method should be executed. So – we can see in case 1 – if the side == bid – then we will execute the method. The actual method body comes after the ->.

You can extract values from records using this syntax ValueName#recordName.fieldName. So our example above has several of those.

Funs and multiple function Declarations

The bit after the -> is invoking another method and passing in as the last argument an anonymous function. That bit that starts with fun(E) and ends with “end”. This is simply an unnamed function (delegate, function ptr, etc) passed to the next function.

Also notice that I declared the function 2x – at the end of the 1st declaration I put a ; and at the end of the 2nd I put a “.” ( a dot). This is a convenient way to get switch statements out of our code nesting. We use the guard statements to see which of the functions get bound to at runtime. We could have also used if or case statements – but for this example’s sake – we are not

OK – that is all the time I have for today… more next weekend

-Chris

So – I am deep into Erlang (still learning for > 1.5 years), Getting into Clojure (new) and getting into Haskell (a few months now). I have decided to spurn the OCaml/ F# branch of the world at least for a little while…

Anyhow – Erlang is the functional language I have the most experience with, yet next to Haskell – the LANGUAGE feels weak. Erlang the language + OTP + the Erlang VM runtime is amazing but the Language itself is not as lovely as Haskell. When I am typing C# code, I am finding myself more and more functionally minded. For a few years now I have felt the move to functional programming on me. C# has had great improvements – but it still feels way too brittle and verbose next to Haskell.

I recently read these articles:

Haskell for C# Programmers Part 2: Understanding IO.

Haskell for C# Programmers Part 3: Visualizing Monads

I found them helpful understanding monads as someone with a large C# experience. I also found this one:

Monads as containers very helpful. I will admit to having read dozens of articles on monads and just now really getting to a point where I think I can actually perceive the need for writing one of my own.

Functional programming style has made great differences in my personal style in several languages. I like programming languages and I am literate in many. Seek the monad, but 1st seek to understand the primitive bits of functional programming and it will aide your programming in any language.

Due to the nature of my current project, and more-or-less my career developing trading related software – I have been thinking hard about concurrency lately. This is not a new thing, but I feel for the 1st time in some years that we are really ready to have some substantial changes in the ways that we write highly concurrent software.

Pub/sub messaging

I have had some great experience doing pub/sub messaging work in the past, specifically @ Merrill Lynch, and to a lesser degree at Morgan Stanley years back now. As it goes, many applications in finance tend to be based on pub/sub messaging in one form or another. These types of systems create a few dynamics that are interesting. Incoing messages are coming from elsewhere on the network so they are inherently immutable. As it goes immutability winds up being very important – essentially having as little shared state as possible is key. Messaging systems tend to make this true to some degree at least.

Immutability

We are constantly trying to make more and more immutabe in my current project. I have been looking a whole bunch at BCL Extras (more on that below) for immutable collections. (When we are doing ‘edits’ we are basically copy-on-write since we are still working on mutable systems to one degree or another). In any case, the whole immutability thing simply enables simpler concurrent access; things can’t change underneath you if they can’t change at all…

(Better Concurrent Collections)

This is an aside, but I am also working hard at creating/ finding/ using better concurent collections that help us to use them correctly. There are some good articles on low/ no lock techniques and Jared Par has a good post here.

BCL Extras

BCL Extras is an intestesting library written by Jared Par. The most interesting bits in here are good implementations of immutable collections, including a good immutable map. We are currently trying to find a good way to use immutable collections alongside our better ‘concurrent mutable’ collections (low locking, still mutable, etc). I recommend just checking this library out – it is along many of my favorite trajectories; Concurrency, Immutablity, Functional Programming and LINQ.

Scala, Erlang

Erlang and more recently Scala use the messaging/ actor based concurrency model. People wrote much more intelligently than I on this here, here, here, etc.

Axum

Axum (formely called Maestro) is here. This is an incubation project at Microsoft for implementing natively in the .net framework the actor/ message concurrency model. Their team blog is here. Axum is pretty exciting – if it ever makes it out of incubation.

Retlang

Written mostly by Mike Rettig for use at DRW Trading. Retlang is most interesting to me because #1 it is useable today in the context of my current project (unlike scala/ erlang/ haskell, axum, etc) #2 it is being used today for real world applications. #3 a former colleuge Mike Roberts and I had a few messages on it and he gave it is ‘thumbs up’. Personal vouchers from trustworthy sources are very useful. his slides for his presentation on Jetlang/ Retlang are here. 

Retlang feels like a pub/ sub messaging system, the difference is that you use it in process. The model that has worked successfully for messaging and driven a certain way of working that removes your locking code etc. is just used in process to give you similar dynamics.

Retlang gives you channles and fibers as its basic bits and pieces. You pub at one end of a channel, you sub at the other end of the channel. It gives you execution queues that your components/ subscribers feed from and act as if they are all working on 1 thread. This means you can punt out your locking all over your business logic. 

Again, with this model, within a fiber, you think about everything as single threaded. Dependency injection is for getting the assembly out of the business logic. With retlang (the messaging/ actor model) we get the multi-threaded-ness out of the business logic and it is wired together. Everything in a fiber is just a component/ set of classes that are wired together and are in a single thread together. We now can punt all the locking, etc code OUT of our business logic code. 

Channels are pub/ sub conduits and fibers are the wiring of what is executing on the same thread. You can have 1 fiber that listens to N channels, and all the subscribers on that fiber will be executing in the same thread. This is SO much like dependency injection it astounds me. We get to NOW assume we are single threaded within this component.

This all sounds good to me. I am going to seriously consider Retlang currently. 

Conclusions

I really think we are at the edge of changing the way that we build concurrent applications. We need to. Building concurrent apps the way to do today is too hard. Really it is. I am looking forward to working hard to make it better so I can deliver better software, faster to my desks!

-Chris-

Erlang vs Ruby

Great post on 2 of my favorite languages.

Getting setup with Erlang on OSX was simple – ./configure, sudo make install, etc.

I tried the Eclipse Erlang plugin… not so hot – they are trying – but it is not really there yet.

I found a Textmate bundle for Erlang. make a text file, chmod so you can execute it, run it. You do not even have to restart Textmate and you have basic erlang code highlighting, etc. That and an open terminal do the trick.

With this – I have been able to have a basic working model. Code, compile, etc. It is amazing how much tooling I rely on day to day coding C# or Java. Visual Studio, Resharper, Intelli J – these tools REALLY do a TON. On one hand it is refreshing to get to just a text editor and a compiler again, but on the other hand – I am just not sure how productive you can possibly be compared to an environemt with TONS of tooling. Simple refactorings becomde ‘grep’ efforts in stead of a code construct aware proper refactoring…

More to come;
Chris

I just had to mention this one- Rabbit MQ:

Built on the Erlang OTP
Implementation of AMQP
Java and .Net implementations
Open Source (similar to qpid)

I have been reading the prag progammers Programming Erlang – Software for a Concurrent World. I have been looking casually @ Erlang for a few months. The thing that initially interested me about Erlang was the “concurrency thing”. Erlang is beyond concurrency friendly – it is built into the language core. Erlang is all about concurrent programming on 1-N CPUs/ machines.

I have been a casual to semi-serious ruby lover for some time. I have not been able to use it 9-5 (except for the odd task I need to script – like aggregating unit test statistics across an enterprise, etc.). I have more recently gotten to giving python (yeah – the iron variety) a go. These languages are great – they are also inherently “object oriented”.

Erlang is not object oriented. This both repulsed and attracts me at once to it. It repulses me because I have baked in the OO-isms to my core.

Erlang is a general-purpose concurrent programming language and runtime system. The sequential subset of Erlang is a functional language, with strict evaluation, single assignment, and dynamic typing.

(wikipedia)

The fact that you have no classes, variables are declared in upper case and you deal primarily in functions is amazing and radically different than OO. Variables are “single assignment” (once you set X – you cannot change it). Again – I say – there are no classes!! I am really interested in working from a totally different perspective. The functional sequential portion of erlang is very interesting. I will have to post another time about pattern matching and how method lookup works – it is interesting.

Erlang is all about concurrency.

Erlang’s main strength is support for concurrency. It has a small but powerful set of primitives to create processes and communicate between them. Processes are the primary means to structure an Erlang application. Erlang processes are neither OS processes nor OS threads, but lightweight threads somewhat similar to Java’s “green threads“. Their estimated minimal overhead for each is 300 bytes, so many of them can be created without degrading performance.  (a benchmark with 20 million processes was tried)

(wikipedia – again)

I wll have to make another post about how you send/ receive/ spawn in erlang to communicate between processes – but it is very interesting.

In short – erlang is really a cool language that is pushing my brain to consider real – different alternatives to “the way I have been working for years more or less”.

Consider this post - Ralph Johnson thinks Erlang could even be “the next java” (something that apparently everyone is looking for – many in ruby, some in erlang, haskel, etc).

Keep watch – there is interesting stuff out there!

-Chris