Chris Donnan : Programming - Brooklyn Style

How I have fit IronPython, Ruby, JRuby and IronRuby into my existing .net projects0
chrisdrop posted in ruby, java, c#, Interop, programming, .net, dynamic languages, python on May 10th, 2008

Ruby and Python are great languages. I have spent several years now with ruby and I am just getting into python in the past year or so. I will still admit that I prefer ruby in every way, but I have come to like and appreciate python as well.

I have been working on building some very interesting global derivative trading applications - so how do python and ruby factor in?? Well - I could have ‘just used them’ in several places. That of course is not valuable, but I have found a few ways to use these tools to really add value to my projects.

Scripting for your existing .net application

I have been making an effort to make our plugin application suite scriptable. I have been able to get IronPython 2 running along with a syntax hi-lighting code editor. It has proved extremely useful. I have also tried to do the same for IronRuby, but with less useful success. Why you may ask - predominantly for 2 reasons:

IronPython vs IronRuby for scripting your existing applications

Basically, IronRuby does not let you do some things that are a must! You CANNOT unsub from events yet . The other big issue is calling existing .net APIs that need generic args. You can see Jon Lam’s response to my queries here. Jon also said:

Our .NET interop is fairly weak today. Because of our current push to get the language running in time for RailsConf, all .NET interop work is being pushed back into June on our schedule.

I understand, but this does make it virtually unusable for using as an existing scripting engine against my current .net applications.

IronPython 2 does a MUCH better job of .net interop. It basically just works as expected in 99% of the cases I have needed it for. This is just a joy I must say. Especially trying to use the early cuts of IronRuby on the DLR, it was painful to get anything interop based between existing .net and IronRuby really working.

Another thing I have done with IronPython is to support ‘calculated formula columns’ in a UI Grid control. We basically implemented this and it is currently ‘not supported’ in production, but in a future iteration we have planned (and requested from users), excel like formula summaries/ aggregations. We will continue this work with IronPython.

So - as you can see - IronPython made itself able to add value.

So how have I fit in Ruby in a meaningful way?

Essentially, we have a plugin framework so that several teams can concurrently develop ‘independent’ applications that can integrate. This means that there are several global teams developing plugins (think eclipse) that can integrate, use and extend one another. This is great, but it means there are all kinds of build, assembly versioning and other application packaging, deployment issues.

In any big bank, or other corporation I expect, there will be different ways of doing things. The continuous builds for differing teams may be different. Some are nant based, some are simple, some are complex, some are msbuild, some do a better job of CruiseControl setup than others, etc, etc. Now if you want to take the work from several teams like this, and build, package, test, deploy in a sane way, there is some orchestration needed. This is where I have used ruby to add real value.

Ruby, Rake/ Python, BuildBot

Rake (and plain old ruby, not j, iron, or another variety) have enabled me to have a single homogenous view of the otherwise complex world. It is basically my tool for integrating multiple teams that are doing things as is appropriate in their own area, but not necessarily what is the ‘best’ for use by everyone else in the world. Ruby has REALLY shined here. In stead of having dozens of XML, Nant, MSBuild scripts, MSBuild extensions, Nant extensions, cruisecontrol configs, bat files, etc - I have ‘owned’ them all with some rake build scripts. I have managed to do this with some ~200 lines of terse, easy to read ruby.

Interestingly, a smart colleague of mine has done something very similar with plain-old-python (and BuildBot). He has managed to take some 50 projects that can be put together in varying ways, built, deployed, etc. He has taken a very difficult problem (managing a large enterprise suite of .net of branched, versioned and VERY WIDELY UTILIZED software) and owned it with some 400-500 lines of python.

What did we get for our efforts with python and ruby?

Number one is just the joy of really getting into daily use of other languages. I have programmed several languages over the years from C/C++, to Java, to SQL, Javascript, Tradestation EL, to Python and Ruby. Working with new languages is a good experience - bottom line. I have had a lot more time with ruby than python. I always love ruby, and I have really enjoyed more real time with python in the past year. I think these languages have each made me a better programmer.

With regards to scripting applications, these languages, in particular IronPython have just made it too easy to add value fast. I have done the work with ANTLR, regex garbage languages, compiling C#, etc. These approaches ALL take much more effort to get similar rewards. I can see myself working in a mode of ‘core plumbing’ in C# and/ java and the higher level stuff in IronRuby JRuby.

That reminds me, a small note. Since we are java server/ .net client apps, I have used JRuby on several occasions to quickly do some send messages around the network, and otherwise mimic stuff coming from the server. I have had more effort on the client/ .net side of things, but I am sure if I was doing more server work, I would be using JRuby all the time.

Regarding build and integration - the big thing that we got IMO was a homogenous solution. Whereas before, there were the xml files, bat files, etc. - we each now have small bits of easy to manage code that does all we need. This is coupled to a small degree of ‘convention over configuration’, but essentially, we have gotten a lot of value for a little code and effort. We have also gotten easy to maintain solutions going forward.

Downsides??

The only real downside here is ‘adding more tools’. There is always resistance (appropriate more or less) to adding to the enterprise overhead of ‘more tech’. Developers also need to learn more to work in more ‘polygot programmer’ environment. So the ‘more tech is bad’ and ‘devs need to learn more’ arguments are valid arguments. That being said, I think the benefits are outweighing the downsides so far.

The future?

I see more coming in the same spaces. I will be working more on ‘higher level scripting’ of existing .net apps, and build/ integration etc. At some point, I can see doing ‘it all’ with these languages perhaps - but I think the current sweet spot in the stack is adding value at certain points of the abstraction. The right tools for the right job and all that.

RabbitMQ0
chrisdrop posted in java, programming, .net, messaging, erlang on March 2nd, 2008

I just had to mention this one- Rabbit MQ:

Built on the Erlang OTP
Implementation of AMQP
Java and .Net implementations
Open Source (similar to qpid)

Locking and signaling0
chrisdrop posted in java, c#, programming, .net on January 27th, 2007

I have had several chats with software developers lately that were working hard to use locking to perform some task. In each of these conversations, the simple solution was to use signaling, not just locking semantics.

I have found over time that for some reason, software developers all know about lock semantics. This has many varieties, lock keyword in C# (really a Monitor usage), Monitor.Enter, java synchronized, java ReentrantLock, etc, Mutexes, etc. The point of locking semantics is to mark a bit of code as a ‘critical section’ or a section that requires exclusive access to it - 1 thread or process at a time. There are varieties like Read/ Write locks, etc - but it is all essentially the same for lock semantics.

Lock/ synchronized/ critical section behavior is certainly mandatory, however as they say “when all you have is a hammer, everything is a nail”. Locks are not THE ONLY tool you can use to accomplish more complex multi-threaded goals. Signaling is another KEY set of abstractions that exists in java, .net, pthreads, Win32, etc, etc.

In java, all objects have signal and wait, in .net there are several choices, Monitor.Wait/ Monitor.Pulse, AutoResetEvents, ManualResetEvents, etc. Signals are used to communicate between concurrently executing threads.

I often ask developers in interviews to implement a blocking queue. Basically - what I am looking for is to see if they understand signaling. A simple blocking queue allows 1 thread to enqueue freely, and another thread to dequeue as long as there is at least 1 object to dequeue. If there is NO object to deque - the caller should be blocked. Signalling is the correct solution. People try to do Thread.Sleep, or say something like while ( _queue.Count > o) - both of which generally are not the best solution. There are plenty of more elaborate solutions - but here is a simple example.

class BlockingQueue< T >
{
Queue< T > _queue = new Queue< T >();
object _queueSync = new object();

void Enqueue(T value)
{
lock(_queueSync)
{
_queue.Enqueue(value);
Monitor.Pulse(_queueSync);
}
}

T Dequeue()
{
lock(_queueSync)
{
if( _queue.Count < 1)
Monitor.Wiat(_queueSync);

return _queue.Dequeue()
}
}
}

(PS Blog software killed some >’s and <'s in the code -sorry I am too lazy to fiddle)

This is a very simple signaling example. WaitHandles, Auto/Manual Reset Events are related - and allow you to accomplish all the more complex inter-thread communication. There are still other ‘higher level’ abstractions like Condition Variables, Latches and the like. I encourage EVERY developer to absolutely get more intimate with these constructs. It is SO common to have poor threading code because developers do not know the tools provided, and when to apply them. It is MUCH easier to use a countdown latch construct than to jury rig some interweaving of thread joins or locks etc. The 1.5 JDK now incorporates some of the very best constructs - read into them -use them if you are writing java. .Net also has lots of excellent constructs. Get beyond locks and your code will get simpler and it will actually work !

-Chris

Cross Language Validation - Ruby?? Python??0
chrisdrop posted in ruby, java, c#, .net, python on October 8th, 2006

For a current project - we have .net clients and java servers - seems to be the wall street standard - at least for my past several clients/ employers. That said - another common scenario is to standardize message formats via XSDs. If we standardize XSDs we now have a contract for message formats. This is the way it is usually - nothing new here. That said - XSDs are not so great at extended validation. You can do some amount of simple validation at the element level (regex, must exist, enumerations, etc) but to do more complex validations between elements of more complex types - it gets increasingly expressive and unwieldy to do in XSD.

Another complicated scenario is partial validation. UIs that have pages of forms to fill out are one example of where partial and/ or contextual validation must be considered. You cannot simply consider the whole thing valid or invalid - you must consider where in the scheme of things you are - and given that context -are you OK.

I am considering as an alternative using a dynamic shareable language like Ruby or Python. I would prefer Ruby - mostly because I think it is a nicer language. That said - JRuby and Ruby CLR could certainly do the trick on either side of the fence. If we were to have object schemas defined as XSDs and defer the complete validation to another set of constructs - we might be in business.

Say for example we have a ‘person’ defined in our message XSDs. We could then have java and .net objects that are ‘person’ class instances. We could then have a ruby routine that validates a person. This Ruby code could run in java and .net and thus - to a large extend be shared. The only tricky bit that comes in is the ‘includes’ or ‘imports’ into the Ruby in it’s respective runtime. In other words - if we have JRuby as the runtime - we would need to let JRuby load in the package for the correct java class ‘person’ in this example. Similarly - the .net runtime would need to reference the correct assembly for the .net object equivlent of ‘person’. Aside from that - it seems that Ruby should work to lay out the validation with arbitrary complexity.

I will soon see if I can get some buy on to try out this idea and I will let the blog world know how goes it.

-Chris

ESP/ CEP query languages1
chrisdrop posted in java, .net, ESP/ CEP, messaging on October 7th, 2006

Recently - my current employer has been looking for a solution for what it has been calling ‘continuous queries’. The CEP groups @ Yahoo have been debating about the most appropriate language for processing real time events. It winds up being quite an interesting topic. Depending on what your ‘query-able’ target is - you may want different semantics.

If you pre-suppose that you have an ‘in memory database’ with real time events when an insert, update or delete happens, the SQL-like languages seem to make sense. Since classic sql is bad at the temporal -  these SQL like languages basically add temporal functionality.

What if you want to mostly query XML messages passing in the wind?? Then some sort of XPath predicate query semantics seem most desirable.

What if you want to query state changes of in memory java or .net objects?? Do you use an OGNL like query language? JXPath?? Some db4o like Soda query, query by example or native query?? What makes most sense for registering interest in real time distributed events for object based updates? How does a delta update work in this world? Do I REALLY have to turn my nice objects into a relational database-like structure to use real time eventing???

These are just a few of the questions running through my heads and the heads of several others in my area. I can never say that I do not get to play with interesting problems

-Chris

Test Driven Development and Mock Objects Developer Session Slides0
chrisdrop posted in java, Agile Development, User Interfaces, programming, .net on August 6th, 2006

Here are some items from the Test Driven Development and Mocks developer session I did a few weeks ago in NYC. I excerpeted lots of people -see the refs at the end of the slide deck. The main things I wanted to hilight were #1 - how important unit tests are. How much better they work when they are done with other core development practices (IoC, etc), How useful mocks are - and a core mock pattern to use.
BigFatGuyTests.cs
TickTests.java
TDD Dev Session.ppt

Enjoy;
Chris

Apache ActiveMQ0
chrisdrop posted in java, c#, C++ on August 2nd, 2006

Apache ActiveMQ is a fast open source JMS 1.1 Message Fabric which supports clustering, peer networks, discovery, TCP, SSL, multicast, persistence, XA and integrates seamlessly into J2EE 1.4 containers, light weight containers and any Java Virtual Machine together with having a host of Cross Language Clients. ActiveMQ is released under the Apache 2.0 License 

I have been playing on and off with Apache’s ActiveMQ open source message queue project. It was really easy to get the server configured and running. I am impressed by the cross language access - as a ‘real’ message queue  product should/ would support. So far - the java connections were easy as pie. I have been able to download and build the C#/ .net client implementation of their ‘open wire’ format. My 1st crack at getting the unit tests running were unsuccesful so far, but I believe this is because I am running the server instance on a linux machine and the C# stuff is running on a windows box. I have been unsuccesful in reconfiguring their test suite it would seem… so far We shall see how this progresses -  another hour or so and I bet it will work nicely. Hopefully I will be able to get a server instance up and running, send messages to and from using C++, C# and Java. Then I will have met my goal with this project.

You can see examples of connecting with C++ here

C# here

Java here

Theoretically in Ruby via Stomp , Python via stomp

I say that is not too bad - providing it works nicely

-Chris

Test Drive Development Developer Session - Unit Testing and Mock Objects in .Net and Java2
chrisdrop posted in java, Agile Development, .net, unit testing on July 13th, 2006

It is time for the next ‘Developer Session’ in my series … The last one that we did - Spring Framework Developer Session (code, power point) was a hit - with maybe ~40 people who came out to hear about Spring. I am happy to report the next dev session is:

Test Drive Development Developer Session
NUnit, JUnit; Rhino Mocks in .net and Easy Mock in Java

Tuesday July 25th 6 - 8 PM

228 East 45th Street
6th Floor
New York, NY 10017

These sessions ARE OPEN TO THE PUBLIC - There is a snack before we get into it - then a presentation, demos and questions.

The sessions I have planned remaining are :

The Peer Frameworks Series - .Net and Java

3) Db4o Developer Session - Open Source Object Database in .net and Java
4) ORM Developer Session - Hibernate, NHibernate / IBatis

I will update the agenda as we finish planning it in the next week or so. I hope to have dial in access for people that want to dial in - and ideally a ‘webex’ type access as well. Feel free to contact me with any comments or questions. Also please message me to let me know if you want to attend - here test-driven-dev-session@chrisdonnan.com. This will help us plan food, chairs - etc.

I will be focusing on:

NUnit
Rhino Mocks
Junit
Easy Mock

My Best;
Chris