Monitoring processes – hurrah for supervisord (maybe)

Monitoring processes, recording their output, and restarting them when they are exit is a common task and quite a fiddlesome and time-consuming one at that. Therefore it makes sense to use a shared code for this task, rather than rolling your own code repeatedly. There are a few solutions for this: upstart and supervisord being two candidates.

In using such shared code you are entering into the world of frameworks (code that you write configuration for, which then calls into your code, rather than APIs that you call into yourself). Frameworks have some risks associated with them:

* Hitting a brick wall where they can’t be made to do what you want to do (extending them is then difficult)
* Having a domain specific configuration language that makes performing tasks harder than it would be to just write all the code yourself in a language that you already know (either due to fixed or variable costs)
* Being difficult to debug (because they solve general problems)

=== Experiences with upstart ===

My experiences with upstart have not been too rosy. This is mainly because because failure mode is largely “your process exitted”. You are made responsible for logging your own output. This is all well and good, until you find that the command line itself is erroring out and you have no output. The obvious work around for this is to make your upstart scripts as simple as possible and push everything into separate jobs that are guaranteed to log output – but this loses some of the features of upstart. The other disturbing things is that these other jobs have to run as root!

It’s also a bit of pain having to do everything on a system level. I prefer not to be root too much of my time, and I like to be able to run multiple copies of things.

The reason for these problems is not so much that upstart is badly designed (though features like capturing all your output or telling you what is wrong with your configuration file wouldn’t go amiss), but that upstart is designed to handle a different task – being robust, never erroring out, handling events and handling a lot of processes. The debug output for complicated chains of events seems to be quite good for example

=== Supervisord to the rescue ===

Supervisord’s purpose is a lot more restrictive and more closely aligned with what I am often trying to do – just monitor some processes. It seems to do this quite well having a few nice features:

* Automatically capturing output!
* Not requiring to be run as root
* Minimal configurations

There are a few losses however compared to upstart. It doesn’t really support the whole starting daemons in response to events thing that upstart has. It does have a notion of events, but these are fed out to specially written listeners rather than being fed back into the configuration files.

Further, I don’t really know how stable supervisord is, upstart will have had many obscure bugs fixed since it is run all the time on millions of machines… this isn’t really the case with supervisord.


November 26, 2011 at 3:21 pm Leave a comment

Working around networkmanager wireless problems

After upgrading to ubuntu 11.10 I had a few a problems with wireless internet on network manager

* Networkmanager failed to turn wireless back on if it was switched off until the machine was restarted.
* Gnome shells network manager applet is (deliberately) broken and doesn’t allow you to disconnect from a wireless network.

All in all this is quite irritating, as a horrible work around you can use nmcli to manage your connections.

nmcli conn list — shows you all connections
nmcli conn down uuid UUID — takes down the connection

November 17, 2011 at 11:30 pm Leave a comment

R plot a number of graphs on the same plot with different colours

plot(xs, y1s, col=”red”)
points(xs, y2s, col=”blue”)

November 15, 2011 at 4:48 pm Leave a comment

Doing horrible things to fix problems in apt

If a dpkg package is refusing to install on an ubuntu machine it is possible to tweak the installation process until it does work.

There script files in /var/lib/dpkg/info are run to perform various dpkg operations. By tweaking these fills you can fix a package which is refusing to install. Needless to say this is evil and dangerous.

October 22, 2011 at 1:25 am Leave a comment

Making ipython less ugly

Ipython is a better shell than the python shell, this much should be clear, it supports:

* Completion
* Better plotting of graphs using matplotlib
* Magic return values for every expression you write _1 etc
* pretty printing by default

However despite this I didn’t use it for a long time, and this was for only one reason: It’s ugly.

To be more specific:
* The initial banner takes up half a screen
* It prints out too many newlines.

Since I appear to be spending this evening yak-shaving, I’ve decided to make ipython less ugly, and therefore just about useable.

=== Making ipython useable ===

==== Removing hideous banner ====

Knowing the version of python you are using is pretty useful in server environment, so we still want this in then banner. We just want want less noise. This is acheived by putting the following executable on your path and using it instead of ipython. (I named it ipy).

It seems like you can modify the banner via options alone.

#!/usr/bin/env python

import sys

import IPython.Shell
IPython.Shell.start().mainloop(banner='IPython %s\n' % sys.version)

===== Making the prompts less ugly ====

You can do this via your ipythonrc file. Just take the newline out of prompt_out

prompt_out ‘Out[\#]: ‘

And there we go a useable shell…

October 5, 2011 at 10:38 pm Leave a comment

Automatic security updates on Centos and Redhat

October 2, 2011 at 10:12 pm Leave a comment

Problems when trying to use gmail for mail forwarding

You can set up postfix to forward mail via gmail ( This is nice because it lets you use command line tools to send mail without you have to go the whole hog and implement a mail server.

When I tried to do this I got the following error, despite my dns plainly failing

Oct 2 21:03:20 moment postfix/smtp[14469]: 431DAE045C: to=, relay=none, delay=984, delays=984/0.04/0/0, dsn=4.4.3, status=deferred (
Host or domain name not found. Name service error for type=A
: Host not found, try again)

This mail explained what was going on ( Apparently postfix does its own dns – which is a little crazy, and my resolv.conf file was missing. I think this is ever so slightly broken….

P.S Gmail will mangle your from address but it’s okay for small personal tools.

October 2, 2011 at 8:42 pm Leave a comment

Older Posts Newer Posts

July 2018
« Nov