flyzip: ZIP files on-the-fly

This is a work-in-progress to create ZIP files dynamically, such as within a web application, where a user’s selection determines which files it must contain.  This might be desirable where a user would find it inconvenient to download the files individually.

I would potentially use this in a photo gallery, if a user wants to select specific photos to download, or all photos matching a certain tag.  For compressed photo files, only a ‘store-only’ ZIP file is needed (without trying to apply further compression of the contents), so that is all I have implemented so far.  Although, if the contents really were further compressable, an HTTP server might apply it’s own gzip or deflate compression anyway as it is served.

This is also useful if you just want to offer ZIP downloads of many collections of files, but you don’t want to pre-generate ZIP files of all collections in advance, because they might never be downloaded by anyone, or because storing these pre-generated ZIP files would take up more disk space.  Or, the contents of each collection may change frequently and it would be a burden to keep the ZIP files up-to-date.  Generating the ZIP files on-demand, with optional caching of the output, is probably more efficient and scalable in such a case.

Source code

Required third-party libraries

Current progress

So far it only supports storing a single file, uncompressed, within a ZIP file that will unzip correctly.  When this runs within the Lua interpreter, it requires a total ~4560 KiB RAM;  comparable to a compiled C implementation of Info-ZIP performing the same task.  But the CRC-32 calculation, implemented in pure Lua, is very slow, achieving here 278 KiB/sec overall on a single 2.6 GHz amd64 core:

3.68user 0.01system 0:03.69elapsed 100%CPU (0avgtext+0avgdata 4560maxresident)k 0inputs+0outputs (0major+333minor)pagefaults 0swaps

It makes use of co-routines, an interesting feature of the Lua language, which helps it make efficient use of a single thread of execution (demonstrated as using a full 100% CPU time here).


Copyright © 2011, Steven Chamberlain <steven@pyro.eu.org>