Monitoring processes – hurrah for supervisord (maybe)

November 26, 2011 at 3:21 pm Leave a comment

Monitoring processes, recording their output, and restarting them when they are exit is a common task and quite a fiddlesome and time-consuming one at that. Therefore it makes sense to use a shared code for this task, rather than rolling your own code repeatedly. There are a few solutions for this: upstart and supervisord being two candidates.

In using such shared code you are entering into the world of frameworks (code that you write configuration for, which then calls into your code, rather than APIs that you call into yourself). Frameworks have some risks associated with them:

* Hitting a brick wall where they can’t be made to do what you want to do (extending them is then difficult)
* Having a domain specific configuration language that makes performing tasks harder than it would be to just write all the code yourself in a language that you already know (either due to fixed or variable costs)
* Being difficult to debug (because they solve general problems)

=== Experiences with upstart ===

My experiences with upstart have not been too rosy. This is mainly because because failure mode is largely “your process exitted”. You are made responsible for logging your own output. This is all well and good, until you find that the command line itself is erroring out and you have no output. The obvious work around for this is to make your upstart scripts as simple as possible and push everything into separate jobs that are guaranteed to log output – but this loses some of the features of upstart. The other disturbing things is that these other jobs have to run as root!

It’s also a bit of pain having to do everything on a system level. I prefer not to be root too much of my time, and I like to be able to run multiple copies of things.

The reason for these problems is not so much that upstart is badly designed (though features like capturing all your output or telling you what is wrong with your configuration file wouldn’t go amiss), but that upstart is designed to handle a different task – being robust, never erroring out, handling events and handling a lot of processes. The debug output for complicated chains of events seems to be quite good for example

=== Supervisord to the rescue ===

Supervisord’s purpose is a lot more restrictive and more closely aligned with what I am often trying to do – just monitor some processes. It seems to do this quite well having a few nice features:

* Automatically capturing output!
* Not requiring to be run as root
* Minimal configurations

There are a few losses however compared to upstart. It doesn’t really support the whole starting daemons in response to events thing that upstart has. It does have a notion of events, but these are fed out to specially written listeners rather than being fed back into the configuration files.

Further, I don’t really know how stable supervisord is, upstart will have had many obscure bugs fixed since it is run all the time on millions of machines… this isn’t really the case with supervisord.


Entry filed under: Uncategorized.

Working around networkmanager wireless problems Turning your computer into a wireless router – easy-to-run code.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

November 2011
« Oct   Mar »

%d bloggers like this: