I’ve been slowly reading On Lisp still and probably not picking much of it up because I haven’t really been able to think of anything to use as a testbed for code techniques.

I realised this was silly and decided to stay within the bounds of what I know and implement a simple web site engine of some kind, probably towards the CMS end of the spectrum, although we’ll see. For now I’ve created a teeny weeny script that uses two functions to facilitate the generation of valid CGI output. Check it out (feedback welcome).

I found just writing this taxing enough (it took about three hours to get to that point). I had to learn basic string manipulation and that involved learning how to learn about particular functionality rather than style. On Lisp is aimed more at people who can survive down in the trenches with Lisp already. I found the HyperSpec too far in the other direction and there’s little in between. Although the Common Lisp Cookbook is pretty useful.

Kill Bill was as I expected.

It was quite different from other Tarantino movies, having practically no dialogue. The violence was good, I liked it, but then I really like all the old kung fu films and anime. It had all the camera angles etc, but it was empty. It was a bit dull.

I found the dramatic pauses over-long. Instead of sitting on the edge my seat, I looked around at the audience. As Louise said, Japan and katanas aren’t “just cool” anymore. They’ve been used in to many hip hop videos and Kill Bill relied on us just being in awe of everything it showed us. It looked like the Matrix without the sci fi and got away with less because you kind of expect more from Tarantino than the Wachowskis.

The soundtrack was pretty good.

When a user submits a form, sometimes it can take a long time for the result to come through. In this situation the user often clicks the button repeatedly to try and get results. This results in many requests being sent to the system, all of which are valid and are then processed. Hence the double posts on Hype and multiple purchases on ecom sites.

Paypal have a nice hack. Disable the button with Javascript once it’s been clicked.

The only worry with this is what happens if the page load genuinely fails? You could set a timeout of say 60 seconds to re-enable the button. Or maybe just repond to successive clicks with an input box, checking that the user isn’t just being impatient.

I made a middle C with SDL. It’s very basic. I allocate a buffer which is 172 samples long — which happens to be the wavelength of middle C when playing at 44khz. Then I dump than buffer on to the sound card repeatedly.

I would have expected the result to be a nice crisp clear tone, but no, it’s more like FM.

I don’t understand how synthesisers are made.

I finally got around to going to Dorkbot, after only a year of meaning to. It was cool.

It was held at State 51, a new media company occupying a disused factory just at the top of Brick Lane, on the fringe of the Shoreditch New Media Mafia Zone. The factory was ace, practically derelict except for signs of workshop-like activities. I was assured that they have proper offices upstairs. If it were me I would be worried that the place was about to collapse.

Apparently they let several artists use the space, hence the workshoppiness. The best bit is that just inside the front door they had a seven-foot black security droid with a single cylon-like red eye. He was only wooden, but I still wanted to escape his gaze pretty sharpish.

I was late so I missed some of the talking, however I saw Ben Woodeson talking about some of his installation pieces, I saw Alex (of Slab and Slub) perform some really nice IDM with nowt but a bunch of console windows with Perl scripts in them and then I saw James Larsson frying things with old monitors and generally giving advice on handy stuff to do. Christ monitors are dangerous.

Afterwards I got talking to Alex about his sound software. I learned that he has a C synth module running underneath sending stuff straight to the DSP (intense) and a Perl server doing timing. He was also playing samples as well, so I guess his C module is pretty handy. It was written by the other guy in Slub, who I didn’t meet. The music he made was excellent and he’d obviously spent a while putting it all together. It started sounding like a bunch of random FM synth tones, but it was obviously much more carefully orchestrated than that. I really enjoyed it. I think it’s a really cool thing to be doing with your computer and it’s yet another thing that I’m jealous of. I’m such a grass-is-always-greener person.

I saw Peter, a guy from the Dorkbot list who I had an interview with a few weeks ago. After the interview I decided I didn’t want the job, but neglected to be polite enough to actually write to him and tell him. Then there he was at the bar, serving beer, so I had no chance of avoiding him. Turns out he didn’t really mind. I think he twigged in the interview that I didn’t really want to give up freelance that much.

Peter introduced me to Paul Makepeace, another coder (I only know that having been to his site though). We chatted for ages about open money, which he’s involved in. Basically open money and similar systems seek to create small community currency systems where the emphasis is much more on using value to encourage co-operation rather than selection in a competitive environment. It’s a fascinating subject and I agreed to do some hacking for the community site at some point. I kind of told him that I didn’t have much free time, which I feel a bit bad about know, because having read his CV I’m quite interested in having a bit more of a chat with him. I may also try and get Charlie involved in some way.

So all in all it was a fascinating and stimulating evening and will definitely, definitely be going again. Next time I will try and take some other people from the gang, particularly Miles.

I’ve been having a few problems with an IT department lately. Being freelance, you wouldn’t think this happens much, but sometimes my clients consist of a marketing/managerial person, who wants to do something interesting online, and the IT department, that wants to go home early.

I pitched for a job building a small site that had quite bit of content saying that I would deploy a simple CMS, written in PHP or something and with a little DB backend, to store the content. It will be really simple, I said. You’ll be able to access any page and edit it, move it around or create new pages all through a nice simple web interface (I’m envisaging something kind of like Zope’s interface). My client doesn’t know HTML so it was cool. We agreed a price and a date for the work to start.

I got a bit nervous about IT, as I’ve had problems before. I said to my client that since we had a month before the project was to begin we should contact IT and get all the technology and access set up so that on the first day of the project I could FTP or telnet in and start running code. Oh boy.

IT don’t like CMSs and they tell us so. They tell us about this great strategy they have of making all the web editors from the different departments train in an HTML edit and learn to FTP. Fine I say, but you could be saving a lot of time and keeping your pages more consistent and well structured if you had a CMS to help you.

This conversation has gone back and forth and has become religious in proportions. I think I said in one of my earliest emails to my client that I didn’t want this to turn into a religious war with her stuck in the middle and this is exactly was has happened. I feel bad, she’s getting close to just forgetting the whole thing.

Today it reached new levels, with the IT’s lead agitator issuing the following statement:

[…] I can guarantee that a CMS will make things harder and more expensive for you in the long run. It’s easier to just run a static website on [our] central webserver and learn how to make changes to it.

Which is just malicious IMHO. First of all the whole point of the CMS is to make the process of updating quicker and easier, thus cheaper. If it doesn’t do that then I wouldn’t be recommending that my client pay me to develop one. Contrary to any opinion you may have formed, I recommend what is good for my clients, not for myself. Sometimes I have bias, but I’m an expert and there are reasons for the things I tell them.

The second sentence just illustrates that this guy isn’t thinking this through from the client’s point of view. It’s easier for him if she gives in and edits the pages by hand. It’s not easier for her. How could it be, she’s going to have to learn HTML. Sysadmins think HTML is easy, because to make a simple page is easy. Making a working site that has good design, good implentation (i.e. bug-free, no broken links), good usability and good information architecture is hard for anyone, even an expert. Somebody who is a) making their first site and b) supposed to be doing something else is going to find it an uphill struggle. In this situation they are most likely to just not do it, or to do the minimum. It’s to alleviate this problem that they hire experts in the first place.

I understand where IT guy is coming from to a certain extent. When faced with the option of doing things themselves, IT people, very much including myself, will often opt to learn. That’s a big big part of what makes computing interesting. Other people don’t think like this, they just want it done without the hassle.

This issue still hasn’t been resolved and with each passing email I worry that the argument grinds my client down more and more and that soon she will just throw her arms up the air and that will be the last of it.

IT people need to try and put themselves in the shoes of their customers more. First of all they need to treat them more like customers and not like a flock of sheep. Next they need to understand the business reasons behind why something has been requested. The people on the other side often don’t make the original requirements known to IT, which is why IT people don’t trust them in return. I recently had a project which would have gone a lot better if I’d gone down into the trenches to talk to the users, but instead somebody who knew less than me about UI design created a design and handed it over to me to implement. When I went to train the users, I got lots of “Why does it work like that?”, “It would be better if it did this…”

So IT people, I know it’s hard when you just want to write shell scripts/reboot servers/play god games, but please try and understand your customers needs more and, shock, put them before your own rather than passing the buck. Be brave enough to tell them the real costs and to suggest implementation techniques that differ from their own, but keep to the user’s needs and people will hear you out.

For my part, this is the core of my job. When people contact me they’ve often been thinking about their new site for a while, but they will have got some things wrong. If I don’t go back to first principles, I end up implementing their mistakes and then getting the blame when a system that does the wrong things is shipped. I can’t work like that, I need to make sure people get what they need and to do that I need to be able to be honest with them and eager to help them.

Well it was a weekend of little sleep (I’ve caught up now :-), but much good stuff and some bad. Mostly to do with a certain large public transport provider here in the capital of the UK.

On Friday I went to see Mouse on Mars, Four Tet and others at The End with Miles and Fran (Lad and Squid of Hype fame). There was a derailment at Baron’s Court on the Piccadilly line and it took me an extra hour to get into the city (not bad really, considering). It was OK though as the night didn’t start until 10, we sat and drank in a little pub around the corner from Holborn tube, near both the End and the college where Louise is having lots of fun doing her photography course.

The End is smaller than I expected and it was packed. It wasn’t death star loud either. Obviously it was wicked loud, but my ears didn’t instantly start hurting (Miles’s did though). When we got in the door somebody I should know was playing. I knew the track from a Tigerbeat compilation, which is odd, because it was a Domino Records birthday bash. Hmm. Still, they were cool, they finished with a Tigerbeat-esque rendition of So Long, Farewell from The Sound of Music.

After a “quick” trip around the club to find ear plugs and to the bar to get booze (they take plastic! they do cashback!) Mouse on Mars started. They were ace, playing a really cool dubby bass-heavy set, which dipped in and out of mashing electronics. Some really energetic set pieces and loads of sequencers with flashing lights. At first I thought they didn’t even have a computer. But then I realised that they did, but they weren’t really using it. It was really good to see them. I recognised nothing that they played and they gave off a feeling of being really creative.

Next up was Four Tet. He was really good too. He played stuff from Rounds, the latest album, and some older tracks as well. He played a version of one of the tracks from Rounds (I forget which one), with a jungle break on it (Amen for the technically-minded) and it worked pretty well. About half way through the set he played Everything Is All Right and I just lost it, I was already dancing really hard, but I think I may have broken my wrists air-drumming along to that, one of my favourite tunes.

Four Tet’s set was really good and people were really excited by it. The booth where he was playing was in the centre of the room and he was surrounded by a sea of happy faces. It was quite something to watch. He finished at about three and I swear half the people left there and then.

Afterwards we hung around to see Manitoba DJing for a bit, he played a crazy combination of tunes, a bit of Roni Size, some old funk, I saw Zep III at one point, but he obviously decided against playing any of it. We decided to go home about 4:30. That was when my real transport fun began.

We walked to Tottenham Court Road and split up, Fran and Miles having to get a bus from nearer Covent Garden and me from Oxford Street. I got a bus down Oxford Street to the right end. Then I spent half an hour finding a bus stop, then half an hour waiting for a bus, then asked a policeman when the tube opened, only to be told that the gates were already open. Then I found out that an open tube station is not necessarily one with trains. The first train wasn’t for half an hour. I was trapped in the station. After walking around for a bit I finally settled on walking up the down escalators as the most interesting thing to do. It’s hard work. Especially when you’re so tired you’re practically hallucinating. I finally got home just before seven.

On Saturday we went to Miles’s housewarming party in Wimbledon, taking 24 beers (£7 for 12 bottles of Pilsner Urquell in Budgens — bargain) and bottles of Margherita, White Russian and Port between me and Charlie. Louise made some biscuits from the copy of How to be a Domestic Goddess that I bought her for her birthday. They were incredible, really really good. I think she’s destined for goddesshood.

It was a good party, big enough that I didn’t talk to everybody. Ben, Miles’s housemate seemed pretty nice and the other people were cool. Kate was there and we chatted for a bit about family members and stuff, always funny. I also had a long discussion with Miles’s friend Duncan who is a computer scientist as well. Much more successful than me, having ported GDB for Cray and worked for Sophos as well as beginning a Phd at Oxford. We chatted about languages mostly, a subject that he was quite knowledgeable about. He tried to convince me to learn Haskell. I may do so, but I think I’ll wait until after I’ve learned Lisp.

Then yesterday I had to redesign one of the sites that I’ve been working on with Arena. It took me ages because a dynamic version of the site has already been built and any major changes to the HTML would just have been impossible to accommodate. Strangely enough, the customer was the same organisation who had such a hard time taking me out and bringing me home on Friday.

Ah Emusic. Unfortunately they’ve realised that their prices are ridiculously low. I mean, $10 a month for as much music as you can download. It’s changing to 40 tracks a month at the end of this month. So I’m downloading fast.

But the client’s being a bastard! I think they must be really heavily loaded by people panic downloading like me. I’m getting loads of timeouts and stuff. Worse than that, of 372 tracks downloaded in the past few days, 144 are zero bytes. Even worse, as I listen to tunes I’m finding some are just truncated. A nine-minute track that’s 700k? I think not. Pain. In. The. Arse.

Perhaps it’s the RIAA at work…

I totally forgot to write about Louise’s birthday — don’t take that to heart Louise! We had a really nice day. I bought her a pretty big bouquet of roses and thistles. They were very pretty and the whole thing was lovely. It was the first time I’d bought flowers and also the first time Louise had been bought flowers.

I went to a nice looking shop on Munster Road and confessed my ignorance and bought the flowers. When I went back to collect them I got to watch the guy making the bouquet, which was pretty good.

The reason I mention this is that they’re just starting to drop. A week is a pretty good innings. The thistles will keep going for a while yet.

The rest of Louise’s birthday was cool too, we went for a meal at the Chop House. It was nice because lots of people came. I got us a bit lost though, assuming that Bermondsey tube would be nearer to the restaurant than London Bridge and being very wrong. Louise wasn’t overly impressed.

After the meal we went to see David Blaine. We had nothing to throw and the whole thing was pretty dull really. We left quickly and sat in the flat while Miles drank port and Adam had rum. I tried to motivate people towards poker, but it was late.

Today Mat found London Bloggers Tube Map. A site where you can catalogue your blog according to your nearest tube station and view others in your area. There are 644 people registered already, which is pretty cool.

Given my burning desire to make new friends in London, I’ve decided to try and arrange a meeting for Fulham and nearby bloggers. This basically means sending an email to everybody listed under Fulham Broadway, Parson’s Green, West Brompton, Putney Bridge, Gloucester Road, Earl’s Court, Hammersmith, Baron’s Court, West Kensington and perhaps even South Kensington. Each of these tube stops is within walking distance of here. South Ken is probably a good twenty or thirty minute walk, but that’t the furthest.

There are quite a lot of people in the area. I hope some of them will come out for a meetup. I may have to lure them with beer, we shall see.

I tried to make Tempura tonight. It sounded like a good idea. I made some batter and chopped some vegetables. I was talking to Louise and I told her what I was making and she said that I wouldn’t be able to deep fry the vegetables in olive oil, which is the only kind of oil we stock. Nuts I thought. I resorted to shallow frying and this didn’t really work out. The batter didn’t crisp properly and clumped instead of thinly coating the vegetables.

The recipe I followed included a sauce for dipping in. That was pretty good. Soy, some honey and some vinegar and it was really tasty. Lucky really.

I made the Tempura because I fancied a change, but it wasn’t that great and I probably won’t bother again as it’s quite a bit of work for just some vegetables really.

After reading LShift’s NMK case study I started reading one of their references, Modeling Web Interactions. This was an interesting research paper that looked at two problems in developing web software. The first was the problem of getting the right input from forms, the second was the problem of users opening secondary windows and following two paths through a system, resulting in one becoming out of date, because the server only stores one set of data.

The second is a tricky problem without a great solution. They recommended time stamping the forms. At least this way if the user does submit an out of date form this problem can be caught and the user warned. However, they suggest no technique for recovering from the situation.

They also don’t really take into account security problems. E.g. anything sent to the client becomes mutable and anything received from the client can’t really be trusted. This is an impossible problem really, but it directly contradicts some of the things required to combat the above issue. If infomation is stored on the client then there’s no risk of that page becoming outdated as the server has no state and simply relies on the client to do that. However, if that state is untrustable…

The other problem outlined in Modeling is more interesting and their techniques more applicable. They discuss the problems of forms not matching the programs required to process the data and the problem of multistage conversations between user and system. Whilst they provide little in the way of solutions for the second part, they find a radical solution to the first. They create a programming language which is type checked with respect to web forms. A form definition is created and as a form-consuming program is installed on the server it can be checked against that form definition for correctness. The authors advocate creating type-checking facilities for all the languages used in web programming, which is obviously slightly useless. However, I think that their idea can still be applied.

The simple way to do it is with a code generator for your favourite language. The generator takes a form definition and produces objects/functions to first of all create an instance of that form and secondly handle the form when it is submitted. This generation ensures consistency and also allows for validation etc to be built in easily, preventing mistakes and retyping the same fucking validation routines. This is an easy and safe way to statically check forms. However, two problems arise.

Firstly the HTML output by the object can’t be decorated very easily. Solutions here include generating XML that is then transformed by a configurable snippet of XSLT or by exposing the output form HTML tree to editing somehow (probably via a DOM like mechanism). XSLT is best as it’s much easier to add a large amount of extra markup, as is required with a commercial web page.

A more serious problem is that of forms which vary dynamically. E.g. a shopping cart form will have a quantity and an ID field for every item in that cart. A statically created form can’t replicate sections and the processor object can’t retrieve and check the data. The solution here is to be able to define possibly repeating sections in the original form definition and to create a form generator that can dynamically repeat the right sections. The processor will be similarly specialised. The only tricky part is communicating between the generator and the processor the number of repetitions that have been created. The generator, or simply the tally, can be stored in the state, but care must be taken to ensure that the form coming back in conforms to the pattern sent out.

Was reading more of On Lisp last night, starting to get into the thick of the language now. I’m slightly worried that I’m not applying as I’m learning. I just can’t think of any applications that I want to develop on the console at the moment. In addition to that, all the made up ones are so simple that they don’t really provide fertile ground for experimenting with programming techniques.

As I was reading the author mentioned Scheme, which I don’t know much about other than that it is Lisp-like in style. I decided to go and find out more and ended up reading an interesting article (PDF) written by some guys at LShift about their use of Scheme inside Java on the NMK website.

The site uses an array of off-the-shelf technologies in order to simplify development of the middle tiers. Cocoon provides the controller. Castor creates XML from JavaBeans which is then transformed by XSLT to the output HTML. The JavaBeans are created by business logic routines written in Scheme and interfacing with the outside world using Java-to-Scheme interface in SISC, a Scheme interpreter written in Java.

The article discusses the relevance of this product-bulky approach and the benefits of using Scheme for the business logic. In a word, continuations. Continuations are, as far as I can tell, a Scheme feature whereby an interpreter can be halted and it’s state stored in a closure. Then it can be brought back to life whenever and executed from the same point onwards. This has unique applications in the world of HTTP. What LShift managed to achieve was an abstraction that allows them to invert the control flow of HTTP.

Off the web, programs go about the process of processing and when they need input from the user, they ask for it. With HTTP, this is reversed. The user asks for something and perhaps sends some data in at the same time and the program processes it there and then and returns output as best it can. This can make things confusing when data needs to be gathered over a number of pages or the user starts using their back button etc. LShift hide the fact that the program has to be halted and the user waited on, and indeed that they may never return the required input, by halting the program secretly and then reviving it transparently when required. This allows the flow of a piece of higher level business logic to proceed as if it were a normal console or GUI app.

Which is nice.

Louise bought me a big tub of Yeo Valley organic Raspberry yoghurt as part of this week’s shopping. It is the fittest yoghurt ever. It’s so good. The whole tub has gone within less than 24 hours of opening. It was as much as I could to not just eat the whole lot last night.

Then, when I threw the pack away it had details of recycling on the inside, but also gave me the URL of a recycling point locator. We already recycle, but we don’t know anywhere to recycle plastics at the moment, which is a big part part of our rubbish.

Sun don’t provide any mock exams for the Certified Java Programmer exam that I’m going to sit. About.com do though. Thanks About! I got 17 out of 30, which would scrape a pass mark (it’s a 50% exam) just about, and demonstrates that I need to do a bit of revision on some of the tricky technicalities.

I always have mixed feelings about About. They have one of the best collections of content on the net and have done for years and years. Their idea is cool and works well. Often there are questions where the experts at About are the only people publishing the right answers.

On the other hand they are terrible advert whores. They popup with every single page load. With the quiz I was doing, the answer would be popped up in a window which would contain a body of explanation — very handy. Then the window would redirect to another About page full of sponsored links and stuff, within about ten seconds. I had to run the test by quickly copying the explanation and pasting it into an email window. Stupidness. I think they maybe risk their userbase by annoying them with so many adverts. It can’t be good.

I forgot to do this yesterday, my one year in London post. We are now half way through our two year contract on the flat.

I still like it.

I just feel like I haven’t really sampled enough of it yet. I guess you never can, but I feel like there’s a lot of stuff that I could have been doing and instead I’ve been sat inside trying to work but prevaricating instead. As I am now in fact.

I’m also a bit sad that I haven’t made any new friends here. There’s Louise and Tom, JC, Adam and Charlie. Al is around. I don’t really know him, but I reckon we’ll get together for beer at some point. Each of those people has friends and work colleagues here, so they’re OK. I’ve met quite a few of these friends and colleagues and many of them were cool, but none of them are the kind of people I would be really good friends with. Because I work from home I haven’t got a line to work people. At Soup I really got along with all the people I worked with and I’d like to have that again. Albeit without the 9-5.

So, solutions.

First of all I need to go out more, tag along with other people on what they’re doing, rather than setting up stuff myself. There’s only ever a very small crowd at stuff I arrange. I also need to go to more internet-organised stuff, such as Dorkbot. The sort-of-party that Furtherfield held that I went to was pretty good and I did meet some pretty cool people there. I’d like to work with Furtherfield at some point, but I think we’ve both just got too much other stuff to be doing before we can collaborate.

I’ve also been thinking about finding a shared office to join in on or something. New Media Knowledge run one up in Harrow, but that’s a bit far to be going each day. Plus they don’t return my emails asking for pricing. I’ll have to keep looking on that front, but I think that shared offices are a bit of a neat idea that haven’t really caught on yet.

Everything’s OK. I have reasonable documentation!

With the help of about ten different pages, I created a simple handler module and installed it on to a virtual host by throwing stuff into my .htaccess file. I even have @INC pointing to a custom libs location so that I don’t have to put it all into the root folder.

Check out mp.ben2.com. Try sticking stuff on the end of the URL.

I’ve got a job coming up where I’ll be writing a bunch of Perl. I had a look at the response of the web server I’ll be running on and found that it’s running mod_perl. Ooh I thought, I’ll go and have a look at that and see what it looks like.

What I found is a tutorial that tells you at great length that you will fall on your arse if you dare to use closures, or destructive subroutines in any way.

Duh!

Closures are interesting, I never really use them. Destructive functions however: that way leads pain. Welcome to the functional age, coders! By functional, I don’t just refer to Lisp et al, I mean any code where a function is a black box.

Anyway, I just want to know how to write mod_perl programs that make pages. A great long page that tells me about the traps isn’t actually very much use unless I’m in one. This is often a problem with educators. They know the gotchas are more important, but they assume that their students have a large capacity to remember a block of knowledge that doesn’t relate to anything else yet.

Once I’ve written and run my first mod_perl script, I’ll have a reference point. Then I will appreciate this advanced knowledge. But until it has something to link to, it’s just a discussion on the nature of Perl, and therefore not what I need at all.

And I signed up for an exam to become a Sun certified J2SE programmer.

Features I implemented today:

• Replies
• Viewing individual messages
• Trucking back through days of messages, a la v2
• My messages in the My Hype area, though this is just a raw dump of message data ATM and will suffer the same problems as the v1 version. My profile page in v1 has grown to several megabytes!