Optimizing the optimization - Performant Incremental Updates for Packages files

admin Wednesday January 6, 2016

In 2006, Michael Vogt implemented support for PDiff (differential Packages) files in APT to optimize the process of updating Packages. At the time, the Packages file, which was already several MB-s compressed, needed to be downloaded entirely to update package indices. Joerg Jaspert and probably other members of the archive maintenance team implemented support for generating PDiff files on the archive side.

Unfortunately, APT's performance when applying (several) PDiff files was quite poor, sometimes worst than the performance for a non-incremental update, as reported (at least) in ticket 372712 and ticket 376158, which was particularly problematic for testing users - until APT 1.1.7, a nice X-mas present from Julian Andres Klode, who identified the bottleneck and optimized the process.

I haven't had the time to test testing since Jessie's release, but I'm starting to miss it smile I wish to thank Michael, Anthony Towns, Andreas Barth, Joerg and others who contributed to the initial implementation, as well as jak, who's finalizing this work a decade later with an optimization job even more thankless than the initial implementation. The next step? Differential updating of packages, with a lowercase "p"... which promises to be even harder to get right.

Finally, I'm using this opportunity to thank APT contributors - particularly its current maintainers mvo and jak - for all of their work. Progress has been slow over the last decade, but the direction is right, and each step is appreciated.

Permalink: https://www.philippecloutier.com/blogpost34-Optimizing-the-optimization-Performant-Incremental-Updates-for-Packages-files