This is one of the funnier mails I have seen at work. Getting to a 100 day uptime isn't easy — so many possible causes of failure to look out for :-). It's the third item here.

I've been trying to fix gzip-transfer-encoding on moria.org.uk. mod_negotiation was refusing to offer a .gz as an alternative to an uncompressed file. After trying a lot of different arrangements, I spotted that even when it served a .gz, it did not return the underlying content's MIME type; it seemed to be ignoring the AddEncoding for .gz. Finally, I stopped messing around and read the source code. It seems that apache simply looks up each component of a filename against its list of extensions, and that it checks MIME types before it checks encodings. Remove .gz from /etc/mime.types and everything starts working. Always read the source!

Here is 0.3.3. The major new feature is the optimised gzip compressor, which I described previously. This, implemented by the new -z option, is the recommended way of distributing files that are not already compressed.

I have also added the -k option, for keeping the zsync file on your local computer. This is useful if you ujst want to run zsync from a cron job, and have it download only when there is something newer on the server. Also new is the -e option, which causes zsyncmake to abort it it would not give the client an exact copy of the target file; and -C to disable the recompression support (which restores the 0.3.1 and earlier behaviour — sorry about that unannounced usage change in the previous version.

Apart from that, there are some bug fixes, for crashes that could occur in unusual cases. As usual, you can get the latest version from the download page. I am increasinly happy with the way zsync is working, and assuming the latest changes all settle down well, I am thinking of declaring a beta (instead of alpha) quality release. So I am interested for feedback on how people are getting on with it.

Another interesting application for zsync: downloading updated source tarballs. For regular software updaters, this has to be a big win. I wrote a small wrapper script for the fetch operation in the FreeBSD ports system, and was able to cut the total download for updating XFCE by roughly a third.

% make FETCH_CMD="/home/cph/src/zsync-fetch -ARr"
===>  Vulnerability check disabled, database not found
===>  Found saved configuration for xfce4-wm-4.2.1
=> xfwm4-4.2.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/xfce4.
=> Attempting to fetch from http://www.us.xfce.org/archive/xfce-4.2.1/src/.
Downloading xfwm4-4.2.1.tar.gz; old versions ./xfce4/xfwm4-4.0.6.tar.gz found.
#################### 100.0% 21.2 kBps DONE    
Read /usr/ports/distfiles/./xfce4/xfwm4-4.0.6.tar.gz. Target 27.4% complete.
downloading from http://www.us.xfce.org/archive/xfce-4.2.1/src/xfwm4-4.2.1.tar.g
z:
#################### 100.0% 47.3 kBps DONE    
verifying download...checksum matches OK
used 1681408 local, fetched 857709
===>  Extracting for xfce4-wm-4.2.1
=> Checksum OK for xfce4/xfwm4-4.2.1.tar.gz.
…
=> libxfcegui4-4.2.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/xfce4.
=> Attempting to fetch from http://www.us.xfce.org/archive/xfce-4.2.1/src/.
Downloading libxfcegui4-4.2.1.tar.gz; old versions ./xfce4/libxfcegui4-4.0.6.tar
.gz found.
#################### 100.0% 15.2 kBps DONE
Read /usr/ports/distfiles/./xfce4/libxfcegui4-4.0.6.tar.gz. Target 33.8% complete. 
downloading from http://www.us.xfce.org/archive/xfce-4.2.1/src/libxfcegui4-4.2.1.tar.gz:
#################### 100.0% 48.8 kBps DONE
verifying download...checksum matches OK
used 1582080 local, fetched 548199
===>  Extracting for libxfce4gui-4.2.1
=> Checksum OK for xfce4/libxfcegui4-4.2.1.tar.gz.
…

In total, a 6.1MB download was reduced to 4.0MB. You can see the benefits of the recompression support added in 0.3.2 now — it reconstructs the original .gz file, allowing it to pass the checksum applied by the ports framework. This was on a fairly large update, from 4.0.x to 4.2.x, so a smaller update would probably see a larger benefit; this has to be a big application of zsync, I think.

For the moment I am running a CGI on moria.org.uk which will automatically generate .zsync files for FreeBSD package sources. And I am providing the zsync-fetch wrapper script which you see in use above. Feel free to try it out (note that you should upgrade to zsync-0.3.3 at least before using it, though, as 0.3.2 had some glitches in the recompression support) — it obviously only kicks in if you have an older version of a file to update from, and it only works for ports with gzip-compressed source tarballs (it just falls back to fetch for .tbz, .tar.bz, and files that moria.org.uk rejects or is unable to download). This is highly experimental, but I hope gives a good idea of the potential that zsync has.

Mmm, Tchaiovsky's Symphony #4 on the radio just now. I'm spending the whole Easter weekend as I always do, listening to the ClassicFM hall of fame. That piece is one of the ones that I used to have saved on the computer (as a .ogg — as a Debian user I dropped mp3 years ago) until I accidentally deleted the whole of ~/Audio a couple of months ago. I only normally rip CDs when the CD itself is failing, on the reasoning that normally I would listen to stuff straight from the CD; but in practice I can't be bothered to put CDs in, so I usually end up listening to the few that I have ripped much more than anything else. There has to be something wrong with an arrangement where I listen to stuff less when I have it on CD…

Committed revision 399.
cph@athlon zsync/c% svn log -r399 makegz.c
-----------------------------------------------------------
r399 | cph | 2005-03-25 09:36:36 +0000 (Fri, 25 Mar 2005)

Built in gzip compressor which optimises for zsync.

I have been thinking about the problem of compression with zsync. Strangely, the code for looking inside gzipped files — written as a workaround, to help kickstart zsync when there is little rsync-able content already available — is currently the most efficient way of transferring most files. I was sure that there had to be something more efficient than compressing stuff with gzip --best and then doing elaborate hacks in zlib to enable us to decopresss mid-file.

I had been intending to look at Transfer-Encoding: gzip, using mod_deflate/mod_gzip, to see if this could get us compresion without the nasty hacks. But I was sceptical that this could take off, because it puts some load back on the server, and these modules are far from ubiquitous. What we want is the individual blocks to be stored compressed on the server, so it does not have to do any compression when clients connect.

Now we have it. I have imported the deflate code from zlib into zsync, and written a small gzip program (actually built into zsyncmake) which optimises the comressed file for zsync. It starts a new deflate block in the output at the start of every zsync block (so every 1024, 2048 or whatever bytes). Initial tests are very promising — on my main test case, of updating a 12MB Debian Packages file with 1 week of changes, the total transfer drops from 140KB to 107KB, taking us amazingly close to rsync -z's best result of 82KB (in fact, the difference between zsync and rsync now is almost precisely the size of the Z-Map — the map of the .gz file — at 23KB).

I have updated the technical paper with the theory and the results.

I've been working with some people on the use of zsync for Gentoo mentioned previously, and we came across this oddity of tar. We were trying to work out why a given directory tree took 160MB as a .tar, but only 88MB when stored in a different format. It turns out that tar uses a block size of 512 bytes, so every file takes at least 512 bytes for the data, and 512 bytes for the metadata (filename etc). So half the tarball was empty space because... because the tar format was designed to go to 512-byte-block tapes. Given that most of tar's use is for distributing files online now, I don't want to know how much space is wasted just because the old format demanded it.

cpio seems to be more efficient - it doesn't use anything like as much padding. Somehow I don't think I will boost zsync's popularity if I advise people to use cpio :-). zip also seems to have very compact metadata.

Some discussion about possibly using zsync to speed up emerge downloads for Gentoo. The idea of building the data into an ISO so the download can be immediately mounted for use is certainly interesting. It's nice to see that zsync is holding up fairly well against rsync; but more importantly, being build on HTTP it has simpler requirements and can benefit from proxies, web caching etc.

I have added some better progress indication to downloads now, which should make it easier to see how fast zsync is going, how near it is to completion etc. I have also fixed a SEGV that you could run into using locally downloaded .zsync files. So I hope this version is a little more user-friendly.

The big feature in this version is recompression. zsync will now compress a file after you download it if the original was compressed; and it does its best to make the gzip file identical with the original (fixing the timestamp, filenames etc in the gzip header). I think this opens up a whole new area of applications, but I'll defer talking about that until I can put up some kind of demonstration.

I was reading through some of the blogs off of newsdot, and came across mewse talking about the lure of cheap computer books. I know this feeling too well — in the center of Bristol there are a couple of bookshops that are often selling computer books cheap, and I always end up browsing through them all. It's worth it sometimes, though: I got a copy of Python Essential Reference for £8 last year (RRP £25).

The best deal I ever got was when they were clearing a lot of Java books. I got Just Java 1.2 for £8. Genuine official Sun Java books for under a tenner!

Worst purchase: well, apart from all the ones I haven't even got around to reading, there was Web Application Development with PHP4. Admittedly it's not the author's fault that PHP is utter rubbish, but the book basically spends its whole time telling you how to avoid making all the mistakes that PHP makes it so easy to make.

This quick hack makes the consolehm utility for FreeBSD return the right CPU & motherboard temperature for motherboards with the it87 i2c chip. I just nicked the register offsets from the it87.c driver for Linux. (Needed it to verify the CPU power saving was working under FreeBSD.)

• % ssh moria.org.uk
• cph@moria% wget http://www.mirrorservice.org/sites/ftp.x.org/pub/X11R6.8.2/src/X11R6.8.2-src3.tar.gz
• cph@moria% ~/zsync-0.3.1/zsyncmake -b 2048 -u http://www.mirrorservice.org/sites/ftp.x.org/pub/X11R6.8.2/src/X11R6.8.2-src3.tar.gz ./X11R6.8.2-src3.tar.gz
• cph@moria% mv X11R6.8.2-src3.tar.zsync /var/www/zsync/s/
• cph@moria% logout
• % wget http://zsync.moria.org.uk/s/X11R6.8.2-src3.tar.zsync
• % zcat /usr/ports/distfiles/xorg/X11R6.8.1-src3.tar.gz | zsync -i /dev/stdin X11R6.8.2-src3.tar.zsync
• reading seed file /dev/stdin: ****************************************************
• downloading from http://www.mirrorservice.org/sites/ftp.x.org/pub/X11R6.8.2/src/X11R6.8.2-src3.tar.gz:..
• downloading from http://www.mirrorservice.org/sites/ftp.x.org/pub/X11R6.8.2/src/X11R6.8.2-src3.tar.gz:
• hashhit 898738, weakhit 29601, checksummed 42053, stronghit 21757
• verifying download...checksum matches OK
• used 44537856 local, fetched 2389527

10MB download reduced to 0.29MB .zsync file and 2.39MB transfer to update from the previous version. But there are far too many steps here; there's no way this is convenient to use yet. Top of the todo list:

• Automatically recognise compressed local file data.
• rsh/ssh client support

Just minor cleanups this time, fixing things that broke in the last release. Most importantly, some uncompressed streams would fail to download fully with 0.3.0 — this is fixed now.

Apparently I am now being syndicated on alkali.org by the old #doomroom crew. So I immediately flooded their front page with my Debian rant :-).

Anyway, I cleaned up the feed a bit now, so it should be a bit less cruel to news readers. I hope.

Hey, I got quoted on ZDNet:

Colin Phipps, a developer at Netcraft, said that usage of Red Hat Enterprise Linux has been dropping off since Red Hat changed its licensing model. ... "We saw a boost in Debian around the time that Red Hat made a change in its licensing model," said Phipps. "Fedora was a really new name so a lot of new sites that were setting up chose to go with Debian or SuSE instead. Fedora has now overcome this obstacle."

They even managed not to mangle what I said too badly — unlike some of my encounters with the press in the past.

I read with interest the extensive discussion surrounding Debian's future release plans. As I have been a user of Debian, both as a developer and as part of my work, I should have an opinion. So here it is.

It might be worth dwelling a moment on just how amazing Debian's package management is. The idea of supporting — what is it, 11 architectures? — across the whole range of many thousands of packages, is no small feat. Having buildds that automatically compile packages, providing build logs so developers can see the cause of build failures on obscure architectures, and activaly blocking packages until they are able to compile on enough platforms — this has put portability and compilability of the packages as a major project goal.

If Debian is to get used on servers, it has to make stable, supported releases. There are nearly 800,000 websites running Debian now; we are within shooting distance of being on the same scale as RedHat. A year ago Debian was the fastest growing; now the moment has passed, and with Fedora making regular releases, they have got the credibility now.

It is hard to advocate Debian for use in a live application when there is no viable stable release. Nobody wants to build a server today with a 3 year old OS (age by itself is not necessarily a problem, but age and the nearness of replacement and so obsolecence is). Woody should be unsupported in 6 months; conversely, sarge may not be supported for another 12 months, for all we know right now. So which one do people like me, who use Debian on servers at work, use? Right now, while I use Debian for development systems, I don't feel that I can advocate either release for new servers.

It is not easy, looking on, to see precisely what has gone wrong. I can see that basic software, like zsync, normally makes its way into testing without much fuss. It is unclear whether the problem is purely numbers (with so many platforms, at least one is sure to have problems?) or the small developer base of some of the platforms (few developers to clear problems, manage buildds?). But it clearly has gone very badly wrong, with not only no releases for years but apparently, until very recently, no clear sign that there was any definite timeframe for getting one.

Assuming Debian targets any live, production use, it has to make stable releases. Making these releases, for i386 as a minimum, is essential for new users starting with Debian, and necessary for Debian to continue to see use in stable environments. Frankly, it's not essential for these stable releases to cover every platform, as developers can always use testing on those other platforms (and so those other platforms will not be suitable for servers). Debian should drop as many architectures from the stable release target as is necessary to make annual stable releases. Having some stable architectures is better than none.

Finally it is in. I have now upgraded the .zsync streams for the sarge and unstable package lists to 0.2+ format; you cannot use zsync-0.1.x with them anymore.

The 0.2+ format is much more compact, reducing the size of the .zsync to download by around 60%.

For a while there I was worried that zsync-0.2.3 wouldn't make it into Debian testing at all. But it seems that it is about to go in ("zsync is going in today", after 19 days). Good — I will finally be able to upgrade the version of zsync generating the example streams on this site.

I have just released zsync v0.3.0. It is available on the download page. This version does not add any features, instead I have cleaned up the code a lot. The zsync process is now entirely contained in a library, and I expect to make this into a shared library which other programs could use in the near future.

The client program is quite simple now - it just contains the URL and HTTP code, and the housekeeping code for handling the downloaded files. It should be very easy to add alternative protocols now; I may have a go at writing a rsh/ssh file fetcher, to allow zsync to be used more like rsync (a popular feature request).

As there are some fairly deep code changes in this version, I would not be surprised if there were a few problems at first. Let me know if I have introduced any bugs.

zsync 0.2.x is only held out of Debian testing now by this build failure on powerpc. It looks like the buildd just failed. I am waiting for it to make it into testing before I switch the zsync feeds on this site over to the more efficient 0.2.x format.

I didn't realise failed buildd logs were available to the whole world until now - full credit to Debian for being exceptionally open.

As I am having to use FreeBSD a lot at work now, I thought I would switch my home machine to it as well. I have used FreeBSD before, but I find it rather easier to use now I have found a bit more about it. The portinstall framework is less complete and nothing like as slick compared to Debian's package management, but it does reliably build stuff from source. And I rather like the idea of pulling the sources from the main distribution site, instead of getting the sanitised versions that Debian provide.

There were a few minor glitches. I have an SIS 7018 on-board sound chip, which did not work out of the box. But adding snd_t4dwave_load="YES" to loader.conf got it working (it's a cheap copy of the trident chip). And I am rusty at configuring XFree (well, x.org in this case) the old fashioned way — Debian's debconf interface to the X config is much nicer.

Of course, the best part of FreeBSD is the use of CVSup for updating. That is one fine program: like rsync it gives the impression of a very efficient and optimised transfer protocol.

After switching my home machine to FreeBSD 5.3, I have found that pyblosxom just does not like FreeBSD. Or the version of pythons is broken, or something else that causes the process to leak memory like crazy. It did the same thing on my hosting account. So I have switched to blosxom, which after a little work does the same job. There might be a few rough corners for a few days, but mostly everything should be about the same as before.