Redis for light-weight rpc in python

March 27, 2011 at 4:18 pm Leave a comment

I’ve decided redis might be very nice as a solution to a particular type of problem.

The pickle module in python solves the problem of: “I just want a hacky way to persist this data quickly.”

Redis similarly gives you a solution for the following problems:

I just want to quickly share some data between some running python processes

I just want to quickly persist some data used by multiple python processes

This is better than solutions like sqlite because the interface to your data is simpler (no sql or orm) and because you can share the data between machines.

This is a hackish solution and there are of course some caveats:

i. Redis has namespaces indexed by numbers (yuck). You can work around this by using a single database for multiple purposes (if your problem is simple enough that you can get away with hackish code), by using globally unique prefixes or by running several redises. At the moment, I feel like globally unique prefixes are the way to go

ii. Redis only supports a limited number of types. It’s probably an idea to avoid storing pickled data in redis if you can avoid it.

iii. Redis is not a relational database Relational databases are wonderful things. If you have a lot of structured complicated data that you are going to run arbitrary queries about and which you want to ensure is correct then a relation database is a way to go. Storing your data in adhoc data structures when you want to do arbitrary things with it will cause you produce reams of code re-implementing features provide by sql until you decide to move your data into sql.

Example:

Scripts to farm out downloads to a number of different machines. (Hasn’t been run)

# Slave - port forward redis port to server first
from redis import Redis
r = Redis()
while True:
    url = r.blpop('urlgetter::tofetch')
    result = urllib.urlopen(url)
    r.rpush('urlgetter:pages')
r = Redis()
urls_to_fetch = sys.argv[1:]
for url in urls_to_fetch:
    r.rpush('urlgetter::tofetch', url)
Advertisements

Entry filed under: Uncategorized.

Perl one-liners Matplotlib: Graph with vertical lines ascending to points

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


March 2011
M T W T F S S
« Feb   Apr »
 123456
78910111213
14151617181920
21222324252627
28293031  

%d bloggers like this: