Monday, September 14, 2009

Zookeeper as a Cluster Management Framework

As applications scale the line between operations and application development fades. Your application probably depends on multiple layers all tied together in a unique fashion. Index servers, database servers, BLOB storage servers, cache servers, etc. Who keeps everything connected properly? What happens when something doesn't work as expected?

Nati Shalom posted an article about applications taking responsibility for themselves. It takes ideas from the long standing meme of "self-healing applications" and combines it with move to dynamic cloud applications that interact with their environment through API's. Nati is the founder of GigaSpaces and in a later article talks about their implementation of this concept in their product.

An open source project that I have been following for a while is called ZooKeeper. It was written at Yahoo as a application coordination layer comparable to the Google Chubby Lock Manager. Unlike Chubby, ZooKeeper implements some basic primitives that support a really wide array of uses.

Although ZooKeeper isn't a Cluster Management API like Nati envisions (that is the application's responsibility) ZooKeeper is a powerful coordination framework that can give your application a sense of identity as a cluster with which to manage itself. Servers and services and report their status and needs, monitor each other, and store and find dynamic configuration information. Your application can also receive notifications when anything changes.

Check ZooKeeper out and let me know if it solves any problems you've been facing.

No comments: