Concurrency Oriented Programming with Erlang

I'm not going to describe Erlang here, but try to explain the impact it had on this Micro Services Architecture concept. I got interested in Erlang a few years ago, when I was a hardcore Lambda the Ultimate lurker. I played a little with it, but I was even more interested in the discussions that took place on the mailing list (I even downloaded the complete mailing list archives and read them in vim, that's as hardcore as lurking can get!).

Erlang proponent have a slogan they often use to (gently) shock newcomers:

Objects are out. Concurrency is in.

To oversimplify things, where you would use objects in OOP, you use parallel processes in Erlang (they call it Concurrency Oriented Programming). Erlang use almost no mutable data structures, it is a shared-nothing message-passing architecture. Dabbling into Erlang made me think a lot about granularity in application architecture.

Erlang got its 15 minutes of fame a few months after Herb Sutter's "The Free Lunch Is Over" article, for some time it seemed everyone was working on an Erlang project, then it kind of faded away (the "Erlang everywhere" hype, not Erlang itself). Building applications in Erlang is not easy. Of course you can use some of the concepts (and  a lot of the spirit) in other languages (for Python you can try Concurrence), but the pure Erlang approach will teach you a lot of things.

With Erlang, processes have extremely fine granularity, you design applications with hundreds or even thousands of processes. But all these processes are inside the same execution environment: the Erlang Virtual Machine. With the Micro Services Architecture, my idea is to emulate these principles at a coarser granularity level and to expose the components of the application on the network, replacing encapsulation with an "exoskeleton" at the application level. More on that later.

Zero configuration networking

Stuart Cheshire's Google TechTalk is a really good introduction to Zeroconf. This is a really interesting technology and one of the most elegant ways to implement a solution I've heard of. When I first saw it, I decided all my applications should be designed this way, which was a little extreme, but it is really inspiring. It has the Ward Cunningham-esque simplicity that I try to achieve in my projects. 

But it's not only interesting as a lesson in system design, I think it should have a far larger use than it currently has. What I find fascinating is that it makes devices appear as network services; once they're plugged into the network they declare what service they provide and where to find them. Of course, Zeroconf-enabled applications behave the same way and it's a really good abstraction level and metaphor for program design. And I think it should be used farther than music library sharing or printer web interface access.

Why not use Zeroconf to interface components of an application? For instance, imagine you create a new wiki on your company's internal network. The wiki application could provide you with a choice of available storage engines that you could chose according to some features it provides: versioning, file storage, WebDAV access... You would get maximum decoupling between the application and the document repository.

You could also expose library functionality on the network. How many applications use a graphic library to transform images (e.g. to create thumbnails)? It wouldn't be hard to create an application that provides image transformation services and declare them with Zeroconf. It would reduce coupling, library dependancy and promote reuse.

The next step would be to design applications (or component) that are able to choose between different but similar available services. But that is a topic for another article.

tl;dr Zeroconf is great, let's use it everywhere!

Replacing D-Bus with HTTP

I like the idea of every application communicating through HTTP (HTTP everywhere!), but I think Zeroconf is better as an application registry. Of course these applications are network services, even when they are all on a local machine.

The messaging bus aspect is another problem, though. Can Zeroconf be used as a messaging bus? Maybe Multicast DNS can be used to broadcast messages? That would be cool, not sure it would scale, but then why not a separate messaging bus? Who said a system like this has to be monolithic?

Little Apps Instead of Little Frameworks

http://blog.ianbicking.org/little-apps-instead-of-little-frameworks.html

This post by Ian Bicking got me thinking about how we spend a lot of time discussing function size, object responsibilities, module boundaries, and package granularity, and then we ignore most of these considerations when designing applications.

Why are our applications so big, so monolithic? Even when our source code is modular our applications tend to be less so. What happened to components outside of software development? What happened to the Unix philosophy of small tools doing one thing and doing it right, and that users could combine to execute complex operations? I know, it still lives on the CLI, but why couldn't we extend it to the Web?

Douglas McIlroy (inventor of Unix pipes):

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.