https://github.com/argandgahandapandpa/computer-router

You need to run sudo bash start.sh and sudo bash stop.sh to start and stop the script, and create a local settings file to store your configuration.

In using such shared code you are entering into the world of frameworks (code that you write configuration for, which then calls into your code, rather than APIs that you call into yourself). Frameworks have some risks associated with them:

* Hitting a brick wall where they can’t be made to do what you want to do (extending them is then difficult)
* Having a domain specific configuration language that makes performing tasks harder than it would be to just write all the code yourself in a language that you already know (either due to fixed or variable costs)
* Being difficult to debug (because they solve general problems)

=== Experiences with upstart ===

My experiences with upstart have not been too rosy. This is mainly because because failure mode is largely “your process exitted”. You are made responsible for logging your own output. This is all well and good, until you find that the command line itself is erroring out and you have no output. The obvious work around for this is to make your upstart scripts as simple as possible and push everything into separate jobs that are guaranteed to log output – but this loses some of the features of upstart. The other disturbing things is that these other jobs have to run as root!

It’s also a bit of pain having to do everything on a system level. I prefer not to be root too much of my time, and I like to be able to run multiple copies of things.

The reason for these problems is not so much that upstart is badly designed (though features like capturing all your output or telling you what is wrong with your configuration file wouldn’t go amiss), but that upstart is designed to handle a different task – being robust, never erroring out, handling events and handling a lot of processes. The debug output for complicated chains of events seems to be quite good for example

=== Supervisord to the rescue ===

Supervisord’s purpose is a lot more restrictive and more closely aligned with what I am often trying to do – just monitor some processes. It seems to do this quite well having a few nice features:

* Automatically capturing output!
* Not requiring to be run as root
* Minimal configurations

There are a few losses however compared to upstart. It doesn’t really support the whole starting daemons in response to events thing that upstart has. It does have a notion of events, but these are fed out to specially written listeners rather than being fed back into the configuration files.

Further, I don’t really know how stable supervisord is, upstart will have had many obscure bugs fixed since it is run all the time on millions of machines… this isn’t really the case with supervisord.

* Networkmanager failed to turn wireless back on if it was switched off until the machine was restarted.
* Gnome shells network manager applet is (deliberately) broken and doesn’t allow you to disconnect from a wireless network.

All in all this is quite irritating, as a horrible work around you can use nmcli to manage your connections.

nmcli conn list — shows you all connections
nmcli conn down uuid UUID — takes down the connection

There script files in /var/lib/dpkg/info are run to perform various dpkg operations. By tweaking these fills you can fix a package which is refusing to install. Needless to say this is evil and dangerous.

* Completion
* Better plotting of graphs using matplotlib
* Magic return values for every expression you write _1 etc
* pretty printing by default

However despite this I didn’t use it for a long time, and this was for only one reason: It’s ugly.

To be more specific:
* The initial banner takes up half a screen
* It prints out too many newlines.

Since I appear to be spending this evening yak-shaving, I’ve decided to make ipython less ugly, and therefore just about useable.

=== Making ipython useable ===

==== Removing hideous banner ====

Knowing the version of python you are using is pretty useful in server environment, so we still want this in then banner. We just want want less noise. This is acheived by putting the following executable on your path and using it instead of ipython. (I named it ipy).

It seems like you can modify the banner via options alone.

#!/usr/bin/env python

import sys

import IPython.Shell
IPython.Shell.start().mainloop(banner='IPython %s\n' % sys.version)

===== Making the prompts less ugly ====

You can do this via your ipythonrc file. Just take the newline out of prompt_out

prompt_out ‘Out[\#]: ‘

And there we go a useable shell…

When I tried to do this I got the following error, despite my dns plainly failing

Oct 2 21:03:20 moment postfix/smtp[14469]: 431DAE045C: to=, relay=none, delay=984, delays=984/0.04/0/0, dsn=4.4.3, status=deferred (
Host or domain name not found. Name service error for name=smtp.gmail.com type=A
: Host not found, try again)

This mail explained what was going on (http://www.sslug.dk/~chlor/postfix/http://www.sslug.dk/~chlor/postfix/). Apparently postfix does its own dns – which is a little crazy, and my resolv.conf file was missing. I think this is ever so slightly broken….

P.S Gmail will mangle your from address but it’s okay for small personal tools.

#!/usr/bin/python
import osgb.convert
import sys
import re

def convert(x):
    long, lat = osgb.convert.osgb_to_lonlat(x.group(0))
    return '%s ({{coord|%.3f|%.3f|source:enwiki-osgb36(%s)}})' % (x.group(0), lat, long, x.group(0))

data = sys.stdin.read()
print re.sub('[A-Z][A-Z][0-9]{6}', convert, data)

—

https://github.com/argandgahandapandpa/Headless-selenium-screen-scraping-example

This is an example of generating a screen scraping using the selenium IDE.

The idea is to carry out a prototypical navigation to all of the pages that you are interested in scraping. You can then export this to a python unit test.

This unittest can then be customized by hand to also perform the scraping that you want to do and add parameterization. Unfortunately, this customization process will almost always involve modification of this generated source code by hand since one wishes to intersperse scraping within navigation. This is a bit of a shame since it means that you can’t regenerate your code.

One then adds code to create a headless selenium using Xvfb (see http://www.alittlemadness.com/2008/03/05/running-selenium-headless) which can be used to perform the scraping.

Caveats to this approach:
* Although the machine doing the scraping doesn’t need be running X is requires firefox, X libraries and full gtk library. This is quite a lot of disk space, quite a lot of memory and quite a lot of installation. This is fine for running scripts on your server, but if you say want to distribute your scrapers over a number of EC2 servers this becomes less fun.

* This is slow. The start up is slow. More pages are fetched. If you hand roll a scraper you can often cut out a lot of the data (all the images, all the js files, a bunch of intermediate pages, any flash components). You can probably get a similar effect by using firefox profiles and switching off images and flash.

* This still doesn’t work too well with with flash (though there is a flash extension)

* I’m still concerned about nasty bugs where the browser slips out of sync with the remote control, even though I haven’t seen this happen.

Advantages of this approach:

* No boring re-writing of http queries which you then get slightly wrong, no having to match headers. Of course it would probably

* No having to pick apart javascript to work out what they are doing.

I feel that this approach works fairly well for hacking together scrapers quickly for batch processes (scrapers rather than robots) where they just need to run rather than run well.

——
Library for this.

I’ve created a small library for spawning headless seleniums cleanly. It deals with things like:

* Avoiding port collisions
* Avoiding display collisions
* Ensuring that resources are clean up
* Logging
* Waiting for processes to initialize.

This is available here:

https://github.com/argandgahandapandpa/selscrape