Thursday, December 20, 2007

Message queues in Ruby

Multiple times now I have looked for a message queue system for ruby. I love Amazon SQS, but sometimes, it doesn't fit the bill. Every time I search I only come up with the ruby queue (rq) tool. Recently I found Reliable Messaging for Ruby (reliable-msg), a ruby messaging library, that is both simple and feature rich.

I found that it exercises a bug in ruby 1.8.5 in rdoc. After applying the fix listed in the bug report, I was able create my directory for persisting messages just fine (I didn't want to take the time to set it up in mysql). It worked fine for me following the example on the web page, but I had to use ReliableMsg::Queue instead of just Queue.

Come to find out, Drb isnt' much of a framework for doing distributed processing. It's more of a protocol with some really nice helpers. Of course you can build your own distributed processing on top of it, but here are two pages about cool distributed processing systems for ruby.

Personal Finance

In college I took a personal finance course. This course didn't just teach you how to manage your checkbook and use a budget, but how to plan for the best possible financial situation from now through, insurance, your kids' college, and your retirement. The content of the course is now posted in really great detail online at This is the best resource I have seen to really learn about personal financial management.

Monday, December 03, 2007

ExtJS customizations

I have been working with the Ext javascript framework lately at on a client project and realized that I need to post this stuff because I have made quite a few nice components for it. It is at Right now it's not much more than a simple extraction from my project and there are no examples yet as I obviously can't post the client project. Some of the things I have built are:

  • CrudGridPanel: A grid panel that I reuse for each Rails model for all the CRUD interaction. Works together with the CrudEditor. One nice feature is that the grid saves it selection everytime it changes and reselects the rows based on id when the data store reloads.
  • CrudTreePanel: Same as the CrudGridPanel except that is displays it's data as a tree. It uses the DataStoreBacking plugin to load the nodes for the tree from an ext data store.
  • CrudEditor: Encapsulates the edit form and the server CRUD actions. Everything uses REST.
  • Some extensions to allow plugins for data stores, data stores that mirror each other's data (used to keep different sets of persistent filters), and better LoadMask handling.
  • LoadAttempts: A plugin for data stores that retrys failed loads.
  • ReloadingStore: A plugin that periodically reloads the data store automatically.
  • PersistentFilters: A data store plugin that creates filters that don't disappear with reloads. Uses the same model as event listeners (addFilter, removeFilter).
  • DataStoreBacking: A TreePanel plugin that loads the tree from a data store and keeps it up to date as the store's data changes.
  • DDBulkMultiSelect: A panel/form field that has two trees and works as a multi select. You drag items between them to perform the selection. In the future it will allow you to drag nodes with children and all of the child nodes will be properlly included in the selected column too.
  • TreeComboBox: Uses the DataStoreBacking to create a combo box whose's contents are an up to date tree rather than a DataView.
As I have time I will clean things up and post examples. This is my first open source code contribution and the first time my code has been exposed to the "wider world". So please give me feedback on how I can make it better code, more usefully, more generally applicable.

Friday, November 30, 2007

New job and polyphasic sleep again.

About 2 months ago I got a new job doing Rails web development. I really love it and also was able to go back to polyphasic sleeping. I am glad to be back and hope my schedule will let me stay with it for a very long time.

Alternative databases

When you think about the root purpose of a database, it is simply to store and access data in an orderly fashion. Relational databases like MySQL have long been the only real option (discounting flat files). Ever since I read about BigTable and S3 I have wanted a distributed data storage mechanism to go with EC2 (or for that matter, any other large web cluster that I don't have). The other day I came across CouchDB, a HTTP accessible document-oriented database server. I got quite excited, and then this article brought me back to my senses. CouchDb is cool, but it is not some holy grail. Every data storage mechanism, and there are lots, falls somewhere in the trade-off of the CAP theorem. Each is suited to a different situation. With that, I found three more data storage systems/styles for scalable web systems.

1) I didn't find out about sharding for the first time here, High Scalability has lots of good content about this , but I was reminded in my reading that RDBMS' have be around a long time and have gained many very nice features. Relational databases can also be made very scalable via sharding. In reality sharding is the basis for most any distributed system, but with most relational systems, you have to implement it yourself. Because you build the sharding mechnism, this option is extremely flexible in meeting the particular needs of your app, but it takes a lot if time.

2) Document-oriented databases like CouchDB are very nice when you just want to be able to throw data at it and deal with the variability later. My personal style is "always a prototype" and that is why I got so excited about this one. I had never read about document oriented databases other than XML databases, and the ones I found had high learning curves or lacked features that were important to me. Mind you I am only commenting on the open source space. I will most likely use this or a ruby equivalent called RDDB (Rails is my currently preferred framework) because they make it SO easy to get up and running. Perfect for prototyping. CouchDB doesn't directly address the distributed aspect yet, but will provide facilities to make sharding a piece of cake.

3) Thrudb is what I was always thinking about when I wanted a database for EC2 and S3. The creator compares it to BigTable and talks about document-oriented databases and their advantages. I highly recommend the article. This is by far my favorite, not because I can judge it's techincal merit, it just seems really cool and easy to use. On the flip side, like most good service clouds, it is a set of services that take some time and effort to deploy.

4) At the far end of the spectrum, is a style isn't much different from a file system, except that it uses S3 to distribute the data. This style, mixed with messaging and processing (like EC2 and SQS) to update views and indicies allows for an awsome amount of data processing without the manual sharding requirements of the other options. This just sounds cool because it seems like it would be linearly scalable, just throw more EC2 instances at it to update the indices faster.

I don't have any web systems bigger than 2MB and my desktop, but just thinking about the cool alternatives for data storage that are coming out makes me dreamy.

Thursday, September 06, 2007

A friendlier front

One of my clients asked me to send them something that they could forward on to a friend that was looking for IT services. I tried to write a good pitch and quoted my recently raised rates, which was all fine and dandy, but then I messed it up. I had been soured a bit by purely hourly work and the disjointed lack of focus that it would create for me so I said that I only do work on retainer, like I do with my current client.

What I failed to recognize, is that although I was being recommended, I still needed to earn my place in this prospective client's mind. Doing some small hourly projects is a great way to do that. Needless to say, that referral never called me back. Fortunately with a second recent referral, the friend giving it to me made some suggestions and that opened my eyes.

I still want my clients to pay a retainer in the long term because if I am managing their equipment I have to be constantly available, or have provisions made, and that is a large non-billable demand on my schedule. I of course wrap the retainer in with the price for phone support, minor incidents, and regular maintenance; but it is still a retainer.

As a side note, I also realized that I need to be more aggressive and/or professional in my sales and call referrals like that to follow up resolve any concerns. Maybe someday I'll learn this whole sales bit.

Replacing carpet

When the tenants moved out of one of our rental units recently, they left a big mess. The carpet was covered in stains that couldn't be removed and was full of cat urine. So we decided to replace the main areas with laminate and use remnants in the bedrooms. This is a basement unit and when we pulled up the carpet we found that the cement wasn't at all level. Neither of us had noticed when we bought the place.

The new tenants want to move in ASAP and we already have the laminate installation scheduled, so now we are scrambling get the issue resolved. This has all been a really big frustration for us, but we have learned a lot too.

Our biggest lesson is that playing the discount market is no fun. Charge higher rates, get better tenants, and provide better rental units. We have applied this in raising our deposit on the unit to $950. It has made a tremendous difference in the quality of applicants. In a situation where you can't discriminate, this has done a wonderful job of simply resolving the issue.

Another important lesson we have learned is to just hire the more expensive professionals and to stop letting the cheap guys waste our time. Get it done right the first time. It saves you so much money in the long run. The higher end market is so much nicer a place to be if you have the standards to match.

Stories...and a windy storm

In reading the book "You, Inc" I came across an item that talks about the importance on stories. Not inventions of the mind, but really describing to someone what happened, how you reacted and how you felt. That kind of story. So I am going to start telling more stories.

Yesterday I found classic, but terribly frustrating failure scenario. We had a power outage across the valley and that included one of my clients. The two floors of their building are connected by a decent 24-port linksys switch. Come Wednsday morning half of the computers and phones didn't work. We happen to use Asterisk and VoIP phones. I also happened to have rolled out bad configs the night before. So I regenerate the configs and reboot the phones. After that, in a particular department all the phones still didn't work. It took sometime to find out that the computers that were plugged in next to those same phones also didn't work.

So while half the floor didn't work at all, the other half worked fine. While I was racking my brains to figure it out and running around looking for the problem, people were mentioning minor issues like: the printer was slow, email was slow, the phone quality is poor. At first I just figured that the small issues could wait, I started to wonder. After thinking for a bit that something on the network was overwhelming it, it dawned on me. The switch wasn't on a battery backup. The power storm might have "given it a concussion". After getting the head honchos off the phone so I could power cycle the switch, remember it was only one department on the floor that had the problem, I rebooted the switch and like magic all the problems went away and all the computers and phone worked again.

I made sure to put that switch on a battery backup after that and plan on adding monitoring alarms for out of sync changes to the phone configs.

Wednesday, July 25, 2007

Unprofessional professionals

Well I just learned the hard way how difficult it is to find good help. We needed to tile the bathroom of one of our apartment units because it had was no solid flooring to keep water from running through. We thought we had found someone who would do a good job, but he had a backlog of work. When he called us and was ready, we didn't have time to doubt check all the fine points of what we expected and in general failed to do our due diligence regarding the bid. Now the job is done and we have to get him to fix all the things we think he should have done the first time. He did do a poor job by any standards, but I should have made those standards very explicit from the beginning. I have been reading about outsourcing your less productive tasks, but after this I don't know how I can trust someone to do a good job. Right now I'm thinking I'll have to tile the kitchen myself. It certainly wouldn't be any worse than the job done by a "professional."

Tuesday, July 24, 2007

Baby steps: Quicken

Well, I would like to get to the point of saying insightful things that people are interested in. So lets take baby steps. First I will say things. Then I will build up to insightful things. Lastly, I will try to understand what people are interested in. So here it goes.

Last year I took my taxes to the Accountant and found that if I had stored my finances in Quicken that he could just take my Quicken file and get most of my information form there without me having to explain, on paper, all my finances. So I did a major project over several months to move my finances into Quicken from GNUCash. It was a lot of work now that Quicken doesn't support native QIF imports. I found some importers, but they just didn't work out for me.

So by the end of 2006 I had a full year of finances ready to take to the accountant. It was so nice. I explained some of the stranger items, handed him my tax documents and my Quicken file and that was it. No worrying about whether he saw this or that, it is all right there. Now mind you I trust my accountant to account for everything in the file or ask me about it, but then again, I wouldn't use an accountant I couldn't trust.

Sunday, July 22, 2007

No More Polyphasic

Well, I got a job doing support for managed services at a local datacenter. That forced me to end my polyphasic sleep experiment. It'll be a good opportunity remember what it is like to sleep in a big chunk, although I do think I'll go back when I eventually move to full time consulting.

I won't have that to write about, but I do plan on writing at least weekly and improve my blogging skills. Lately I have been thinking about business processes especially in IT and the importance of documentation, auditing, and automation of those processes. So I may write about that. We'll just have to wait and see.

I personally have a great need to actually try things I think of to generate a proper opinion, but perhaps we'll forgo that because I don't have the time to fully explore many of my ideas.