Global challenge: Cambodia #

13:29, Wednesday March 8th, 2006 • feeling enthusiastic • no comments

Project O revealed!

In November, Louise and I are going to ride 475km through Cambodia and Vietnam to raise money for Oxfam!

We've put up a website with all the info, you can even sponsor us online. We need to raise £5,200 between us, so we're grateful for any help you can give. We'll also be blogging (and moblogging) about our training and the event itself soon.

Aggregate operations in shell #

17:25, Monday February 27th, 2006 • feeling relaxed • no comments

You might be familiar with aggregate clauses in SQL, you know, count(*) and friends. I was interested in performing aggregate functions on files of tab-separated data. I'd do an aggregate query in SQL, get a set of result rows, then I needed to total one column.

I output the data to TSV, using a mysql -e command and redirecting the output to a file. The data looks something like:

id      viewcount       comments
127067  341     44
127076  66      2
127077  158     6
127111  379     25
127112  83      3
127119  105     10
127131  47      0
127135  51      1
127137  133     17

I found that awk has the required mix of unix command line tool sensibilities and numeric functions. Here awk accumulates the 3rd value of each row and displays the total.

cat data.tsv | awk '{x+=$3} END {print x}'

awk is basically C-like and has a small core set of numeric functions built-in. To calculate mean, divide the accumulator result by the built-in variable NR, which is incremented each time a record (line) is processed and so contains the total line count by the time we reach the END block.

Doctests vs unit tests #

3:11, Friday February 24th, 2006 • feeling relaxed • no comments

OK, I might have to eat my words here. Doctests are quite cool as small dabs here and there, but the module doesn't have the flexibility of unit tests. It's much harder to chunk the tests and refer to them individually. This is hugely exacerbated by the whole Django model magic madness. Hopefully most of the gymnastics I'm doing in testing at the moment will become redundant when magic-removal hits the shelves. In the mean time I'm considering moving all my tests back to units, which will be a pain because of the whole from django.models... boilerplate that's required everywhere.

Also, having the tests in with code is quite cool in some ways, but it does make getting around the files tricky when the blocks get large. So much for literate programming.

Testing Django apps #

4:15, Friday February 17th, 2006 • feeling jubilent • no comments

I've started work on a new, quite large Django project. It will be six weeks until I've implemented everything in the specification, so I wanted to get in early with some tests.

Building on the code and ideas of Hugo, Ian Maurer and Sune Kirkeby, I created a simple Python module that provides:

A testenv_setup method that modifies the database settings in the current DJANGO_SETTINGS_MODULE to point to a in-memory sqlite database, installs a specified set of models into that virgin database and changes the MEDIA_ROOT to point to a folder in /tmp/.
A testenv_teardown method which cleans up the temporary MEDIA_ROOT.
A unit test base class (derived from unittest.TestCase) that uses testenv_setup and testenv_teardown.
A new handler, TestHandler, derived from WSGIHandler which allows the user to construct requests to the Django processing framework very easily (e.g. get_response("/my/url/")) and returns the original objects created for the response and allows exceptions to bubble rather than just pushing back the HTML only. This facilitates easy test writing. Of course the HTML is there too, and, like Sune Kirkeby, I push it into a BeautifulSoup object for handy parsing.

The result is a simple module that can be used to create either doctests and unit tests that test both models and views. I prefer doctests, because they just seem more pythonic to me. Unit tests are great, but there's a weighty Javaness to them, they're more clunky to write, which is critical. Doctests are also a better fit for Django's model magic. Because of the DB settings monkey-patching, unit tests must import the models after setUp, within each test function. This just adds tonnes of lines of cruft. With doctests, I just have a block of imports at the top of each test docstring.