Why might a set API be useful

Why use a set api rather than a list API if one doesn’t care about performance

In most languages in built set classes have better performance than lists. But suppose one doesn’t care about performance – why would you use a set rather than a list.

Broadly speaking a set is distinguished from a list by the lack of

a) Ordering
b) Duplication of elements.

How can these properties be useful. In a sense it isn’t true that sets don’t have an ordering, since in most languages you can iterate over the members of a set – rather they just don’t have a very good ordering. So this is not really of any use to us.

So the only thing that could be useful about a set api is that it allows one to avoid duplication of items without writing any code to do this. This will be useful for any problems in which you explicitly want to avoid duplication, examples of this include:

* Only wanting to email a number of people once
* Only wanting to rerun tests once
* Only wanting to list each result once (but having an algorithm that generates duplicates).

This concept of avoiding duplication can be generalised

A dictionary allows one to avoid duplication of parts of an object. That is, if we consider two objects to be near enough identical only parts of them match then we can use these parts as keys in a dictionary – and thereby avoid duplication.

In fact we can encapsulate this behaviour in an equivalence set – a collection with ensures uniqueness up to some equivalence function (if our elements are immutable we could use the following).

class EquivalenceSet(object):
def __init__(self, getRepr):
self.getRepr = getRepr # get representation of equivalence class
self.mapping = {}

def add(self, el):
self.mapping[self.getRepr(el)] = el

def __iter__(self, el):
return self.mapping.itervalues()

This assumes that out “uniqueness” can somehow be represented by a function such that f(x) = f(y) implies x and y are equivalent . However, sometimes we have more interesting notions of not “unique” for example if we are representing objects in a plane – two objects might only be considered “unique” if they do not intersect. If we are in this situation it may not be clear what we should do when elements are not unique.

Javascript: keyCode versus charCode

Javascript key press events have both a keyCode and a charCode property. These are mutually exclusive – whenever one is non-zero the other is zero.

Which one is set depends on the type of event: keydown and keyup events give one events with keyCode set (they produce corresponding character) whilst keypress events give one events with charCode set.

Automatically loading a truecrypt share at startup

I’m toying with the idea of moving all of my data into the cloud, or rather, keeping all of my data dropbox so that I can access if from 2 different machines and have those changes synced without thinking, and so that I’m less dependent on any single piece of hardwire.

However, although I want some data to be present on all the machines I use, I don’t want all data to be present on every machine I use, and I don’t think I can easily have several dropbox shares.

One solution is to keep some of the data in a truecrypt volume which is automatically mounted on some machines but not on others.

For this purpose I adapted the following bash init script from here. This takes a truecrypt file, reads a password from disk and mounts the file in my home directory. I’m suspicious that there might be problems with conflicts when dropbox updates the truecrypt file whilst it is already mounted… but we’ll see. (dropbox has version control so I should be moderately safe).

Note that this approach may place your volume password into the list of processes – so you might prefer not to use this on shared machines. Also, you probably would want to change the umask and the owner of the share.

#!/bin/bash
#
#   /etc/rc.d/init.d/truecrypt
#
# Mounts the /home partition with truecrypt.
#
# chkconfig: 2345 90 10
# description: Truecrypt

# processname: truecrypt

[ -x /usr/bin/truecrypt ] || (echo "truecrypt can't be found" ; exit 1)

RETVAL=0
prog="truecrypt"
desc="Truecrypt" 

start() {
   echo -n "Mounting encrypted volume..."
   uid=$(cat /etc/passwd | grep moment | cut -d ':' -f 3)
   truecrypt -t --fs-options='umask=000,user' --non-interactive VOLUMNE_FILE VOLUME_MOUNT_POINT -p "$(cat PASSWORD_FILE)"
   RETVAL=$?
   [ "$RETVAL" == "0" ] || (echo "FAIL" ; exit 1)
   echo "OK"
}

stop() {
   echo  -n  "Unmounting encrypted volume..."
   truecrypt -t -d /home/moment/cryptshare
   RETVAL=$?
   if [ "$RETVAL" == "0" ]; then
        echo "OK";
   else
      echo "FAIL";
   fi;
}

case "$1" in
  start)
   start
   ;;
  stop)
   stop
   ;;
  restart)
   stop
   start
   RETVAL=$?
   ;;
  condrestart)
        [ -e /var/lock/subsys/$prog ] && restart
   RETVAL=$?
   ;;
  *)
   echo $"Usage: $0 {start|stop|restart|condrestart}"
   RETVAL=1
esac

exit $RETVAL

Not sure whether this is a good use of time…

De-nesting decorators in python

Definitions of decorators are quite nested

def decorate(f):
    def patched(*args, **kwargs):
        # do stuff involving f
    return patched

About 25% I also forget to write the final “return patched”.

One way to work around this is to write a ‘denest decorator decorator’ so you can write the following as an alternative to the above:

@denest
def decorate(f, *args, *kwargs):
    # do stuff involving f

This has the disadvantage that you can’t perform operations at the time of decoration – but it simplifies code slightly for the standard case (at the cost of making your code less idiomatic).

The denest decorator is defined like this:

def denest(func):
     def decorate(f):
           def patched(*args, **kwargs):
               return func(f, *args, **kwargs)
            return patched
     return decorate

Example usage:

A decorator that converts converts a function which could raise an Excpetion into a function which returns None on error.

@denest
def makeIntoTry(f, *args, **kwargs):
    try:
        return f(*args, **kwargs)
    except Exception:
        return None

Python generator to list decorator

Python generators can be evil because of the implicit state. How many time have you written done this?

items = get_items()
print items
f(items, x)

It might prevent confusion if generators are only used when you need lazy iteration.

However, creating a list, appending to it several times and then returning it is an effort – which is probably why I want to use generators when I don’t need to.

One workaround is to create a python generator to list decorator like so

def listify(gen):
    "Convert a generator into a function which returns a list"
    def patched(*args, **kwargs):
        return list(gen(*args, **kwargs))
    return patched

@listify
def f(x):
     for i in range(x):
        yield "item" + str(i)

assert f(5) == "item0 item1 item2 item3 item4".split()

One can also define a dictate decorator as follows

from functools import wraps

def dictate(func):
    @wraps(func)
    def patched(*args, **kwargs):
        return dict(*func(*args, **kwargs))
    return patched

Automatically populating a gcal event with reminders

I find it irritating to have to set reminders for google events by hand, so in the theme of efficient-over-effective here is a the urls.py of a django project which will create a new event in google calendar with several reminders and redirect you to a page for editing its details. If you have a virtual server you could host this project there. You can then bookmark this url on the machines you use.

A little (if any) adaptation would make this work on google app engine (and might even allow you to login without hard coding a password).

Alternatives would be to write a greasemonkey script for google reader or write a ubiquity command that does this and redirects you to the page – but both these things seem more difficult to debug.

from django.conf.urls.defaults import patterns
from django.http import HttpResponseRedirect

import atom
import datetime
from gdata.calendar import CalendarEventEntry, When, Reminder
from gdata.calendar.service import CalendarService

def create_reminder(req):
    S = CalendarService()
    S.email = "EMAIL@ADDRESS"
    S.password = "password1"
    S.ProgrammaticLogin()

    event = CalendarEventEntry()

starttime is manditory - tomorrow seems as good a day as any


    def make_google_dt(dt):
      return dt.strftime('%Y-%m-%dT%H:%M:%S.000Z')

    now = datetime.datetime.now()

    start_datetime = now + datetime.timedelta(days=1)
    start_time = make_google_dt(start_datetime)
    end_time = make_google_dt(start_datetime + datetime.timedelta(minutes=30))

    occurence = When(start_time=start_time, end_time=end_time)
    occurence.reminder = [
        Reminder(hours=3, extension_attributes={'method':'sms'}),
        Reminder(days=2, extension_attributes={'method':'email'}), 

to feel happily surprised but not panic


        Reminder(days=7, extension_attributes={'method':'email'}),
        Reminder(days=14, extension_attributes={'method':'email'}),
        ]
    event.when.append(occurence)

    new_event = S.InsertEvent(event, '/calendar/feeds/default/private/full')
    edit_link = new_event.GetHtmlLink().href
    return HttpResponseRedirect(edit_link)

urlpatterns = patterns('',
    (r'', create_reminder)
)

Note that this is a slighlty evil – insofar as a GET request is having side effects… but POST commands are slightly troublesome to book mark, and adding a layer of indirection slightly defeats the purpose.

Technical notes

• Underneath the python library this is constructing and xml command and sending this to a google webservice.
• The python library does really support email reminders. It does however support adding arbitrary attributes to the xml element representing the reminder:

  Reminder(days=7, extension_attributes={'method':'email'})
  

  this add method=”email” to the reminder tag. (One can also set method=sms).

  This is not documented in the google python documentation. It is (I think) documented in the gdata plain xml documentation.

• For a working reference implementation in javascript using plain xml see the gtd tickler greasemonkey gmail plugin.

Useful references
Google calendar interface from python </a

Getting python argspecs

For the benefit of google:

def f(x, *args, **kwargs):
    return x
import inspect
print inspect.getargspec(f)

Human string comparison in python

NOTE: An alternative way to do this

I’ve found a possible alternative to this. If you only want to sort strings in your current locale you can use locale.strcoll, which is a cmp function in your current locale.

import locale
["one", "two", "three"].sort(locale.strcoll)
l

On my machine this still isn’t a very human friendly sort but depending on your LC_COLLATE environment variable it may be. This is very much less of a reinvent-the-wheel-because-buying-one-is-too-hard approach – though I suspect it might take one 4 times as long to get working…

Of course at times what you want your commadline to do isn’t what you want all other programs to do…

---

The string comparison in python does not order lists as one would expect:

>>> l.sort(weird_strcmp)
>>> l
['hello', '?', '}']
>>> l = ["1", "Hello", "abc", "?", "}"]
>>> l.sort()
>>> l
['1', '?', 'Hello', 'abc', '}']
>>>

python sorts strings as sequences of bytes.

I couldn’t find any easy way to sort strings in a more friendly fashion (Though surely one must exist?!) So here is an quick implementation of one (use this code as you wish):

from itertools import chain

letters = list(chain(*zip(range(0x41, 0x41 + 26),
               range(0x61, 0x61 + 26)))) # interspersed upper and lower

numbers = range(0x30, 0x30 + 10)

symbols = sorted(list(
        set(range(0x80)) - set(letters) - set(numbers)))

weird_order = list(chain(letters, numbers, symbols))

assert set(weird_order) == set(range(0x80)), "Didn't get every character"

def weird_strcmp(a, b):
    """Compare strings for lists for English humans who hate
         unicode and aren't using python 3k any time soon."""

    a = map(ord, a)
    b = map(ord, b)

    for x in chain(a, b):
        assert x < 0x80, "%s is not in the ascii character range" % x

    a = map(weird_order.index, a)
    b = map(weird_order.index, b)
    return cmp(a, b)

>>> l.sort(weird_strcmp)
>>> l
['abc', 'Hello', '1', '?', '}']

Be aware that this isn’t really tested and may not order symbols how you think they should be. It, however, has the advantage of not taking you 15-30 minutes to write and debug. Say if you can think of any foibles and I’ll fix them.

Debugging Wifi on Ubuntu Linux

To actually get ubuntu to give you some logging when connecting to wifi (you know so that you can act according to some information rather than at random):

1. Become superuser
2. Run killall NetworkManager
3. Run NetworkManager –no-daemon

This will then give you output for every stage of the connection process.

Network manager is (I believe) the tool that handles the overall process of opening a wifi connection using different processes (iwlist, iwconfig, wpa_supplicant etc) to do the actual connection.

Notes

Messages of the form “supplication connection state change number -> number” are (i think) from wpa_supplicant. Someone who isn’t me should change NetworkManager so that these are written in english. For now note that the states mean the following:

0 – WPA is disconnected
1 – WPA is inactive (no enabled connections and wpa isn’t trying to connect
2 – WPA is scanning
3 – WPA is associating (a loose-form of connection)
4 – WPA is associated
5 – WPA 4-way hand shake
6 – WPA group handshake
7 – WPA completed

I was seeing a lot of 3 -> 0 state changes when I was debugging. [This was taken from the wpa_supplicant source code in src/common/defs.h]

Updating to intrepid broke my left arrow key!

Seems that the keycodes on my keyboard (thinkpad T43p) changed after installing the new version of intrepid. I note that the kernel version changed. This meant that my xmodmap was mapping this key to alt…