The first version of the technical paper is complete. The version at that URL is now fixed, and won't be modified. I want it to remain there to reflect my original thinking and early empirical results for zsync. I will continue to update a newer version of the paper with new results as I make changes and implement new optimisations.

I updated the paper first to reflect the improved compressed file support in zsync-0.1.2+, which had not been fuly reflected in the discussion of the implementation previously. I have also finished off the references to other projects and papers that I got ideas from.

Ah, finally figured out why rxvt-unicode was giving me so much trouble. Xlib doesn't seem to grok locale aliases, so it didn't understand LANG=en_GB.utf8 (which is what Debian's libc configuration suggests) but does understand LANG=en_GB.UTF-8. Cut and paste, and some other interactions with text and xlib, were quite broken before.

Before I launched zsync, I searched around to see if anything similar had already grabbed the name. I decided that I could easily knock the Zaurus file sync tool off of the rankings on Google, and couldn't see anything else too similar.

This paper (and this presentation of it) seems very close to zsync though, and they used the name zsync in fact. Oh well. The approach they describe has some overlap with zsync too: they do the rsync bit on the client side, but with a special server still; and they use compression on deltas (which is different to zsync, which works on already-compressed files).

The decomposable hashes and continuation hashes seem interesting too. I had considered decomposable hashes, but had not come gone into it enough to come up with a good way of using them yet. I don't quite get the continuation hashes yet, but I suspect that both ideas are much easier to do when you have a custom server, than they would be for zsync.

I just spotted that Linux Format included an article about PrBoom in their February issue (LXF63). No online version I'm afraid. They talk about it in the context of client-server network games, and it has a little column to itself in amount their "Network Everything" series of articles.

…We'll examine PrBoom, a modern version of the classic shooter. Linux and Windows versions of this are available…

Like many other network games, PrBoom is separated into client and server parts… the server is started first and the client's machines log in to the server to play.…

It's particularly nice for me, because the network code is the one bit of PrBoom I can claim is all my own — since I rewrote it years ago, and am responsible for the current client-server design. They liked the wget-retrieval of WAD files to clients, although it is years since I wrote that — I'm surprised that it still works.

This book, by Robert Cialdini, is the best thing I have read in ages. Fascinating insights into the irrational responses that people have that cause their judgement to be unreasonably swayed. The chapters on consistency and authority are very good, and the book is full of well-cited academic research, statistics and real-life examples.

The best part of the book, however, is the chapter on Social Proof, and this is outstanding. I have never read an account better, more compelling or as thoroughly explained as this into the way people react in a herd mentality, and the ways in which it causes silly responses in a whole range of situations.

The book also has some great newspaper cuttings, photographs and a great selection of newspaper and magazine cartoons, which keep it from becoming heavy reading.

I must put in a good word for rsvg, which I used to generate the PNG versions of the bootcharts that I put up previously. I have been looking for a decent SVG viewer for ages, but this tool seems to do an excellent job of rendering them to PNG which is then easily viewed — even doing quite a good job with alpha and antialiasing. Another win for command line tools, IMO.

I came across this interesting one today: Jevons paradox. Briefly stated, it is that more efficient use of a resource will often cause its consumption to increase.

Clearly it is just a restatement of a simpler principle from economics — cheaper goods sell more — but stated in this way it is a little new and clever, and certainly gives an interesting rebuttal to the idea of energy efficiency as a way to solve the world's energy crisis. Just as government intervention in Britain to reform the car industry caused the price of cars to fall, and hence caused car use to rise, in direct opposition to their election promises to cut car use, the law of unintended consequences and Jevons paradox may be the death of energy efficiency as a silver bullet.

I just finished reading Secrets and Lies. Probably, having worked in IT security for approaching 5 years, I should have read it sooner. I wasn't that impressed by the bulk of the content, though, as it is clearly aimed at a less technical audience (the Star Wars analogy is a rather feeble twig offered to the techies), and it offers mainly critisism and very little constructive advice. In fact, most of the book is about the inevitability of endless security problems that will never be prevented.

But at the end it was redeemed by Schneier admitting that he felt the same about it — he simply realised that there was no hope to offer. Essentially, the book it good at explaining why there are problems, bad at explaining any way of dealing with them, and therefore good at advocating the important and inevitable alternative:

• Software companies will need to accept liability — to some extent, yet to be determined — for faults in their products.
• Once there is liability, it will be dealt with as modern economies deal with all such risks: Schneier focuses on insurance, but industry codes of practice and external audit, even without the insurance driver, are powerful defences in court to say that due diligence was done to prevent faults.
• The importance of people watching people watching people — concious consideration of security at management levels (instead of treating it as one-off technical bugs), and expert, external auditing to gauge and minimise risks.

In this book Schneier is certainly in fear-uncertainty-and-doubt mode — and doing a better job as a self-publisist than a security advisor. But it's an excellently written book from the point of view of raising the problems with a non-technical audience, and I'm very glad to see such an influential book taking time for well written defences of important concepts like open source, full disclosure, and the value of anonymity. Essentially he is arguing that perfect security is hopeless: but we shouldn't worry about it, instead just balance the risks and ensure that no individual person or company bears too much risk on their own. Provided the credit card industry can cover the fraud out of set-asides from their transaction charges, who cares if credit card numbers are stolen occasionally?

Have I discovered a new way of attracting search engines? I posted a graph of user agents on this site two days ago, commenting on how busy google and msnbot were. Two days later, two new engines enter the lists, and are already gaining ground.

Search engines watching for mention of rivals and jumping in to compete? Well, okay, I have to admit I am paranoid. The search engines do make a big percentage of the hits, and an even bigger percentage of the transferred data: I guess only a computer can put up with reading all the way through my dull prose :-).

I also corrected the graph this time for various things which were incorrectly classed under Mozilla on the previous graph. I'm still surprised by how far ahead Firefox is (although a good chunk of them are myself, checking the site).

(Ego trip ahead — skip if you want!) A passtime of mine, in very idle moments (actually, mostly idle moments at work…) is to see what percentage of hits on google for my name actually correspond to me (as opposed to someone with the same name).

I used to have an excellent record here: when I was at university, and filing Debian bug reports every week, not to mention PrBoom releases, nearly every reference would be to me. It dropped to a low of around 50% two years ago. In particular, there's some Dr Colin Phipps who works for an oil company, who gets mentioned in a lot of press releases who is my main competitor — he holds the top spot on google for now. But I am back up to 83% at the moment (looking at the first 3 pages (30 links) on Google, which is fairly good I think. But then my surname is hardly common; anyone with the surname Smith has my sympathy.

Sadly, all of this is based on memory, so from now on I am going to keep a record of the percentage of references to me in the first 30 hits, and track it.

I see, from my referrer logs, that zsync has made it onto Wikipedia. The entry is fairly well written too.

And it is being added to FreeBSD, which would explain the FreeBSD query-pr.cgi referrers in the log.

Direct google searches remain the main referrer, though. I am a little surprised at the distribution of user agents, though — since when were MSN so voracious? Google still has the lead, but not by a long way.

I had intended to make some progress on zsync over the holiday period, but, perhaps wisely, I decided to give it a break instead :-). So there has been little progress since december. However, I have been working on a few bugs that have cropped up.

zsync 0.1.6 is now available from the download page. Most importantly, it fixes a corner case in the compressed file download which caused compressed file transfers with large differences to miss blocks and potentially fail, or at least transfer data less efficiently. I have also fixed a problem with servers that return fewer ranges than requested, and zsync not parsing common http_proxy settings.

And, yes, zsync is now in Debian. I am still interested in positive feedback by the way, particularly if you have any good uses for zsync.