Monday, September 14, 2009

Zookeeper as a Cluster Management Framework

As applications scale the line between operations and application development fades. Your application probably depends on multiple layers all tied together in a unique fashion. Index servers, database servers, BLOB storage servers, cache servers, etc. Who keeps everything connected properly? What happens when something doesn't work as expected?

Nati Shalom posted an article about applications taking responsibility for themselves. It takes ideas from the long standing meme of "self-healing applications" and combines it with move to dynamic cloud applications that interact with their environment through API's. Nati is the founder of GigaSpaces and in a later article talks about their implementation of this concept in their product.

An open source project that I have been following for a while is called ZooKeeper. It was written at Yahoo as a application coordination layer comparable to the Google Chubby Lock Manager. Unlike Chubby, ZooKeeper implements some basic primitives that support a really wide array of uses.

Although ZooKeeper isn't a Cluster Management API like Nati envisions (that is the application's responsibility) ZooKeeper is a powerful coordination framework that can give your application a sense of identity as a cluster with which to manage itself. Servers and services and report their status and needs, monitor each other, and store and find dynamic configuration information. Your application can also receive notifications when anything changes.

Check ZooKeeper out and let me know if it solves any problems you've been facing.

Sunday, September 13, 2009

A git based wiki

I decided that I wanted to document all the ideas in my head in a Wiki instead of Google Docs. I use git and vi a lot and like to have access to the text files backing anything. So I was excited to find WiGit.

WiGit is a php-based wiki that stores everything in Textile-formatted flat files in a git repository. 10 minutes and I had a wiki and a git clone of it on my laptop. I can edit things locally and push them back to the web server and the wiki automatically reflects the changes. Of course, the pages can be edited through the web interface too.

The power of Git as a web-based wiki is really awsome.