Not that much has been happening to zsync in the last two weeks, as I took a bit of a break from working on it. This release is the result of some general tweaking to the support for compressed files. The code is somewhat cleaner now, and compressed downloads now make full use of HTTP/1.1 pipelining — doing all the downloading in a single connection — as I had already implemented for uncompressed downloads. zsyncmake also creates a more detailed map of compressed data files and tries to align block boundaries with the boundaries that zsync uses; this has caused a further improvement in the efficiency of compressed downloads. Compressed downloads now appear to beat uncompressed in most circumstances now — I have not yet updated the technical paper with new figures to reflect these improvements, although I will do so soon.

Due to the changes, I have been able to reduce the amount of data per block for the compressed file map, so, although more blocks are stored the .zsync file, it is not significantly larger in my test cases. This does mean that the .zsync format for compressed downloads has changes incompatibly. Sorry folks, but while I am at 0.x I reserve the right to break compatibility at any time if there are significant benefits from doing so. Uncompressed downloads remain compatible though, so the streams on my streams page should still work with either version. I will switch them over to the new format in a week or so, at which point zsync-0.1.0 will no longer work with them. Also, I notice that as of 0.1.0 it looks like compiling on Solaris is broken; I should get around to fixing that by the next release.

It looks like I only narrowly beat another project with the same principle into existence: cnysr. The site is short on detail, with no releases so far, so it looks like Goswin has not got as far as I have. Anyway, I am talking to him now, and as we seem to be going in a similar direction it is possible that the projects will merge in some way.

This is the completion of a major cleanup. The compressed file code is now somewhat better organised. As a result, I have been able to fully implement the gzip mapping code, including starting deflating in the middle of the block (zsync-0.0.6 had to deflate from the start of a block). I have updated chapter 5 of the technical paper with my latest test results. This is also the first update to the paper since fixing the extra-block-download bug a couple of releases ago — zsync is very close to rsync now, and beats it by a good margin for gzipped data.

I have dropped the OpenSSL dependency. This was an awkward compile time requirement, as many Linux distributions do not install its headers by default. Well, awkward given I only used a couple of checksum algorithms, anyway. I have switched to some public domain MD4 and SHA1 code from OpenBSD's libc. This also solves a problem that zsync-0.0.6 could have been awkward to distribute, as the OpenSSL license conflicts with the GPL.

Finally, I have switched to the Artistic License (version 2) as of 0.1.0. It's a much less restrictive license — essentially you are free to use it and modify it how you will, provided my copyright is respected, the acknowledgements stay in place, and you respect that the open-source version of zsync remains the official one. But see the COPYING file for the details.

Licensing open source programs is too complicated. I have started off with the GPL for zsync, as I had expected to use more code from rsync. However, I only took one file, and that turned out to be buggy so I have dumped it.

But now I'm using OpenSSL, but strictly speaking the OpenSSL license is not GPL compatible. I could switch to a BSD license, but I would still have to abide by OpenSSL's old-BSD-style advertising clause — fair enough, they have the right to put it in there, but it would make zsync out-of-bounds for linking to GPL projects.

So I dropped the only fragment of rsync code and it is time to drop the OpenSSL dependency too. It was a rather heavy compile-time dependency anyway, considering I only needed the checksumming code (at run time it only required libcrypto, which was less of a problem). I looked around and found the code from md5sum in OpenBSD, which looks just the ticket. And, as a gesture in favour of more liberal licenses like the zlib license (which thankfully has no problematic restrictions), I will probably go to something like the Artistic license for zsync-0.1.0.

Well I have put up a stream for Fedora Core 3, and it is on the streams page. I am disappointed with the results though: only a 20% saving on the total download (starting from Fedora Core 3 test3). Perhaps I need to tune the blocksize, or do something else to optimise for iso9660 filesystems.

Anyway, some people might be interested in another trick, which is to combine zsync with bittorrent. First, use zsync -i FC3-test3-i386-disc1.iso http://zsync.moria.org.uk/s/FC3/FC3-i386-disc1.iso.zsync to start a zsync download of FC3 using data from the test3 ISO. As soon as it reaches the downloading from stage, ctrl-c it. Then start a bittorrent download, and exit after bittorrent has begun to download. Then copy the get.part file over the FC3-i386-disc1.iso.zsync that bittorrent has started. And restart bittorrent. Bittorrent should think for a bit, and then show the file as 20% completed or so. It will then fetch the rest of the file via bittorrent — and will even upload chunks of the file from the zsync'd results. Uploading from a file you've never downloaded, now that's a bit scary :-).

Nothing much to say about this release. I am in the process of cleaning up the huge function in zfetch.c that provides the compressed file support. This version breaks out the gzip navigation code into its own library, so this is a bit more reusable and less is embedded into the client implementation. I'm not putting this into libzsync though, as I can see plenty of uses for that where no zlib is needed.

The good news is that the changes from 0.0.5 seem stable, and I hope within a couple of releases to have improved the download efficiency again. The figures in the technical paper are starting to look old, so I will look at updating those soon.

The latest version is on the download page. Thanks to those people who have taken the time to let me know what they think, and to report problems or suggest ideas for further development. I see there are some hits on the streams page — at this stage I am interested in hearing from anyone who is finding zsync useful, and at this early stage I am very open to suggestions for how the project should be developed further.

I came across a strange bug in the code to do MD4 checksums, which I had nicked from rsync. Never use other people's code without reading it thoroughly :-(. The bug didn't actually hurt much, because while it got all the block checksums wrong, it did so consistently, so zsync was happy. But it's a bug, and 0.0.5 fixes it.

Unfortunately that means the checksums have changed incompatibly. So zsync-0.0.5 is not backwards compatible with older streams, and zsync-0.0.4 will not work with newer streams. It's 0.0.x, if you were relying on the format then tough. I have changed all the URLs on the streams page so there should be as little confusion as possible.

Also this version doesn't miss matches at the end of the stream, and it should compile at least on Solaris. And you really need OpenSSL now (whereas before it was used only for the optional SHA-1 checksums). Sorry.

Get it here.

.zsync file available here, and listed on the streams page.

If people have other suggestions of things I should put up for people trying zsync out, let me know.

Okay, zsync 0.0.4 is out. I found the stupid bug in the HTTP fetcher that was causing some redundant information to be transferred. The download efficiency improves by between 5 to 25 percent with this fix.

I have also put in a very simple first heuristic for using both compressed and uncompressed source URLs. By using uncompressed for small blocks (where available), zsync saves downloading the inflate table at the start of the block and the intervening data ahead of the bit you want to download. While for large blocks it continues to use the compressed data. This cuts another 100k off the inter-day Debian Packages update - in fact I have increased the block size on the two streams being offered now, as the .zsync was now bigger than the required transfer of real data.

I must update the technical paper with some new figures now these changes are in...

On the practical front, zsync-0.0.4 now allows relative URLs in the .zsync file, and follows HTTP redirects when downloading the control file. Get it here.

Finally a decent and simple blogging program. I refuse to use anything which requires a CGI on the server. Maybe I am paranoid, but given the endless stream of vulnerabilities in everything PHP based, I now distruct all cheap-and-cheerful dynamic content. Everyone wants a blog, so there are a thousand out bits of blogging software there, which means a thousand crap programs. So I will stick with pyblosxom --static, thanks very much.