I was pointed today at the EVE Online Dev Blog, where they are apparently using zsync as a backup system for repairing incomplete downloads or corrupted files. It sounds like that are not using zsync itself but have probably taken the idea and/or parts of the code and built it into their own repair tool.

It seems like an odd application to me. Incomplete downloads are a rather easy case and do not require the rsync algorithm (there are plenty of HTTP and of course FTP clients that will resume partial downloads). And corrupt files - well I'm surprised if having corrupted parts of files are so common that they have to optimise for it. But I don't know anything more about their problem than is in their blog post. zsync does have the advantage that it combines downloading with verification of what you have, and it does it with minimal work for the central servers (as I guess EVE have the common problem of a high ratio of users to servers).

A new release of Kye! No major changes in this release ‒ this is mostly an update to fix issues running Kye with newer versions of GTK and librsvg. In particular, my standard tileset no longer rendered correctly with the latest librsvg. There are also minor interface improvements.

You can get the latest version and installation help from the download page (or direct links: tgz, rpm ).

While I'm here, I was playing some levels to test the release and so I will throw out some level recommendations. Crowds is good fun. And my own level HOLD UP (from Problem.kye) remains the best of my own levels. And, um, yes I can't link to the other level set I liked because the site is down. I'll put up a few more recommendations with the next release, anyway.

I'll do a stable 1.0 release soon, once Robert's Kye page is back up (it's temporarily down while at the moment), and assuming there are no issues reported with this release.

Someone drew my attention to Ubuntu using zsync to distribute ISOs (thanks Jon). Cool — that's one of the use cases that I was testing with many years ago.

The Royal Crescent, where I am often to be found on a sunny Sunday morning.

[[!img Error: Image::Magick is not installed]]

more

Bath is looking quite picturesque at the moment under an inch of snow.

Photos taken yesterday:

The Circus: [[!img Error: Image::Magick is not installed]]

more

I am required to run skype (yuk) for work, and, since I am rather paranoid, I had up to now been running it under a separate user account (so it's long history of security problems would not endanger me too much). This was a bit inconvenient, though; it meant that I had to start it manually instead of the desktop session manager starting it for me, for instance.

I was intrigued by the AppArmor support in the latest version of Ubuntu, however (Ubuntu AppArmor), and decided to try that instead. I took a sample apparmor file suggested by SuSE and updated it to work on x86_64 and Ubuntu 9.10.

AppArmor profile for skype 2.1.x on Ubuntu 9.10 is here. Note that this is a very conservative profile ‒ it prevents reading or writing to your own homedir (hopefully preventing malware or skype exploits reading your ssh keys, private files, etc) other than ~/.skype and config files that it needs from other areas. Your abstractions/audio needs to be correct for audio to work. It specifically prevents opening URLs by clicking on them (protection against spammed malware links) ‒ you have to copy and paste to a browser. If you think the inconvenience outweighs the safety gain, this patch is not for you :-).

No warranty. Importantly, this can't protect you against the poor security of the X window system itself; skype exploits could still do keylogging, for example (unfortunately skype refuses to work with x.org's security extension enabled for it, as do so many programs; indeed, skype 2.1.x it itself now using the evil are-you-typing monitoring that is now a staple feature for malware, so you probably can't distinguish from its normal behaviour).

There seems to be a lot of confusion about zsync's custom embedded version of zlib. I had not documented the exact reason for the patches very well; so I have now committed an explanation of the changes to my own repo. As I may not make another release for a little while yet, I am posting the explanation here also.

I have started a discussion with the zlib maintainer about what sort of API changes could be made such that I could use the standard zlib, but so far no-one other than me understands the requirements and I'm not actually bothered about it (it's the Fedora people who are worked up about it). So, absent anyone else stepping in to do the work, it may take me a while to get to it.

Local changes to zlib used by zsync

There are two different modes of operation that zsync supports that these patches are designed to support:

Changes to the deflate code: Compressing a file in a way that is optimised for zsync's block-based rsync algorithm ‒ starting a new zlib block for each 1024 byte (for example) block in the source file. cf http://zsync.moria.org.uk/paper/ch03s04.html . This is used by makegz.c in the zsync source.

Changes to the inflate code: Working with files compressed with the standard gzip(1). To enable people to get started with zsync, I want it to work with existing compressed content. To achieve optimal results with standard gzip files, I made zsync capable of starting decompression in the middle of a block. In these cases it has to download the block header, then skips forward to the part of the block that gives it the data that it wants. cf http://zsync.moria.org.uk/paper/ch03s02.html

Contrary to some internet discussion, the changes are not related to rsync's changes nor to rsync compatibility (zsync isn't compatible with rsync - whatever that would mean ‒ nor do these changes relate in any way to the rsyncable gzip patch).

Changes to the deflate code

Essentially, I hijacked Z_PARTIAL_FLUSH to mean something new ‒ I want to start a new zlib block, but unlike Z_PARTIAL_FLUSH I don't need to emit a whole byte between blocks, so I took that out. Correctly this ought to be implemented by adding a Z_NEWBLOCKONLY_FLUSH or something like that instead of repurposing an existing state.

(If this were the only issue preventing the use of a standard zlib, distros could change it to use Z_PARTIAL_FLUSH with only a slight loss of compression efficiency.)

Changes to the inflate code

zsync uses the rsync algorithm to construct a desired file from an (e.g.) older local version of the file and then downloading any new/needed blocks from a server; the aim being to minimise the amount of data downloaded to construct the target file. It supports downloading those blocks from a gzipped version of the file on the server. If I want e.g. bytes 4096-8192 of the file from inside the gzipped file, I could download the whole zlib block (using a map of the compressed file that I construct beforehand and is downloaded first) containing the range 4096-8192 (zsync 0.1.0 used this method); but it can do better (fewer bytes downloaded) than that, by downloading just the block header and then downloading the bytes within that compressed block that correspond to bytes 4096-8192 of the contained data.

To do that, I need: a) to be able to start inflating at the start of "any length/literal/eob code in any dynamic or fixed block, or at any stored byte in a stored block.". That is what additional function inflate_advance() and the export of updatewindow() allow me to do.

b) make a map of the gzip file that lets me know what points I should start downloading at in order to inflate particular byte ranges of the contained content. To do this, I can decompress each byte range into a buffer of that size and then quiz zlib for the position in the stream; but I need to know that the position in the stream does correspond to the start of a code or the middle of a stored block (not, e.g., that we have just read a backref and the backref expands to span the boundary; in that case, I would need to know that position where the backref started and the lib doesn't give me a way to find that out).

This is given by inflateSafePoint(), by the modification to cause the inflator to return to the caller at each code in a dynamic block (the LENDO change), and the implicit guarantee provided by using my own copy of the library that I know how the library behaves around internal states and stream position (I need a guarantee that the library won't read ahead more than it needs to, and I need to access certain member variables directly to get the bit position in the stream).

I also removed inflate_fast as I did not want to spend the time working out if it was compatible with these changes.

For the record, my result from devnull 2009: joint 2nd in the grand prize (up from 4th last year). I played fairly well, but a few mistakes cost me the grand slam, and Adeon was in a league of his own this year.

Character	Conducts	Max HP	Turns	Time	Score	Outcome
cph-Arc-Gno-Mal-Neu	4	212	30367	15:17:17	1752828	ascended
cph-Rog-Orc-Mal-Cha	1	208	39290	14:57:15	2153678	ascended
cph-Val-Dwa-Fem-Law	1	265	29190	13:50:43	1834433	ascended
cph-Bar-Hum-Mal-Neu	6	106	6775	1:49:59	17912	petrified by a chickatrice
cph-Mon-Hum-Mal-Neu	5	216	31253	14:16:43	1742840	ascended
cph-Sam-Hum-Mal-Law	2	224	33586	13:21:07	2267255	ascended
cph-Pri-Hum-Mal-Neu	1	164	30841	11:17:22	2256523	ascended
cph-Cav-Dwa-Mal-Law	2	230	32592	11:53:27	2017528	ascended
cph-Wiz-Elf-Fem-Cha	5	234	34173	13:32:20	1873665	ascended
cph-Bar-Orc-Mal-Cha	2	199	28349	9:18:50	2093334	ascended
cph-Hea-Gno-Fem-Neu	5	34	2138	0:25:37	1121	killed by a sewer rat, while helpless
cph-Hea-Hum-Fem-Neu	2	265	33685	14:19:03	2053517	ascended
cph-Kni-Hum-Mal-Law	2	227	27442	12:53:02	1593765	ascended
cph-Ran-Hum-Mal-Neu	1	185	39448	15:57:07	2181653	ascended
cph-Kni-Hum-Mal-Law	12	16	1	0:00:04	0	quit
cph-Tou-Hum-Mal-Neu	6	50	8424	2:05:36	6312	killed by a Green-elf
cph-Tou-Hum-Mal-Neu	3	96	20850	5:07:33	171246	petrified by Medusa

I have added packages of Kye for python 2.5 and 2.6 to the download page.

zsync 0.6.1 is now available from the download page. This fixes a few bugs that have been spotted in the previous version, plus a few minor feature changes:

• recompression support for gzip files made with zlib:gzio.c or gzip -n
• fix compilation on MacOS X
• allow HTTP redirects on the target file; not sure whether this is a good idea or not...
• fix unecessary transfer of whole file where file is smaller than the context size (1x or 2x blocksize)
• use sequential_matches=1 when there is only one block; otherwise we're forced to transfer the whole file for files below 2kiB
• fix librcksum handling of zsync streams with sequential_matches == 1; it was giving false negatives when applying the rsync algorithm, resulting in poor use of local source data when sequential_matches == 1 (which didn't actually occur in any recent version of zsync)