8f17ae2ce0
- New upstream version of sparsify.c. - Fix the license. - Modernize the spec file. - Remove sparsify patch (equivalent patch has gone upstream).
142 lines
5.8 KiB
HTML
142 lines
5.8 KiB
HTML
<html>
|
|
<head>
|
|
<title>Keeping filesystem images sparse</title>
|
|
</head>
|
|
<body>
|
|
<h2>Keeping filesystem images sparse</h2>
|
|
<p>
|
|
Filesystem images in local files can be used by many PC emulators and
|
|
virtual machines
|
|
(<a href="http://user-mode-linux.sourceforge.net/">user-mode Linux</a>,
|
|
<a href="http://fabrice.bellard.free.fr/qemu/">QEMU</a> and
|
|
<a href="http://wiki.xensource.com/xenwiki/">Xen</a>, to name but three).
|
|
Typically these filesystems are created as sparse files using
|
|
commands like:
|
|
<p>
|
|
<pre>
|
|
dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0
|
|
/sbin/mke2fs fs.image
|
|
</pre>
|
|
where the enormous <code>seek</code> value causes <code>dd</code> to move
|
|
forward by 2GB before writing nothing at all. This results in the
|
|
creation of a sparse file which takes disk space only for blocks which are
|
|
actually used:
|
|
<p>
|
|
<pre>
|
|
$ ls -l fs.image
|
|
-rw-rw-r-- 1 rmy rmy 2048001024 Apr 18 19:10 fs.image
|
|
$ du -s fs.image
|
|
31692 fs.image
|
|
</pre>
|
|
As the filesystem is used, more and more of the non-existent blocks are
|
|
filled with data and the size of the file on disk grows. Sometimes it would
|
|
be nice to be able to reclaim unused blocks from a filesystem image. However,
|
|
deleting files from the image doesn't return the space to the
|
|
underlying filesystem: even free blocks in the image still consume space.
|
|
Reclaiming the space can be achieved in two stages:
|
|
<ul>
|
|
<li>Fill unused blocks with zeroes
|
|
<li>Make the file sparse again
|
|
</ul>
|
|
<p>
|
|
One traditional way to zero unused blocks is to create a file that fills
|
|
all the free space:
|
|
<p>
|
|
<pre>
|
|
dd if=/dev/zero of=junk
|
|
sync
|
|
rm junk
|
|
</pre>
|
|
<p>
|
|
The disadvantage of <code>dd</code> in this context is that it destroys
|
|
any sparseness that exists: free blocks that were originally represented
|
|
as holes in the image file are replaced with actual blocks containing
|
|
zeroes.
|
|
<p>
|
|
As an alternative approach, and as practice in mucking about with ext2
|
|
filesystems, I've written a utility which scans the free blocks in an
|
|
ext2 filesystem and fills any non-zero blocks with zeroes.
|
|
The source, <a href="zerofree-1.0.3.tgz">zerofree-1.0.3.tgz</a>, is
|
|
available for download. It's also available in a git repository:
|
|
<code>git clone http://intgat.tigress.co.uk/rmy/git/zerofree.git</code>
|
|
<p>
|
|
<ul>
|
|
<li>A cautious user would run fsck on the filesytem both before and after
|
|
running zerofree.
|
|
<li>The filesystem to be processed should be unmounted or mounted read-only.
|
|
<li>The utility also works on ext3 or ext4 filesystems.
|
|
<li>Binary packages are available in the standard repositories for Debian and Fedora.
|
|
<li>The <a href="http://partedmagic.com">Parted Magic</a> live distribution includes zerofree.
|
|
</ul>
|
|
<p>
|
|
Better than either of these would be to have the guest kernel keep the free
|
|
blocks empty. My original inspiration was the
|
|
<a href="http://www.uwsg.iu.edu/hypermail/linux/kernel/0401.3/1058.html">
|
|
ext2fs privacy (i.e. secure deletion) patch</a> described in a Linux
|
|
kernel mailing list thread. I've also made use of a later patch for ext3
|
|
entitled
|
|
<a href="http://marc.theaimsgroup.com/?l=linux-fsdevel&m=113986429313502&w=2">Secure Deletion Functionality in ext3</a>
|
|
from the linux-fsdevel mailing list. (See also the authors' paper on
|
|
<a href="http://www.filesystems.org/project-sdfs.html">Secure Deletion File Systems</a>.)
|
|
I've modified the patches to make them more suitable for the present purpose.
|
|
<ul>
|
|
<li><a href="linux-2.6.25-zerofree2.patch">linux-2.6.25-zerofree2.patch</a> (for ext2 filesystems)</li>
|
|
<li><a href="linux-2.6.25-zerofree3.patch">linux-2.6.25-zerofree3.patch</a> (for ext3 filesystems)</li>
|
|
</ul>
|
|
When a filesystem is mounted with the <code>zerofree</code> option (added
|
|
by these patches) all the blocks freed when a file is deleted are filled
|
|
with zeroes.
|
|
Remember, this extra work will hurt disk performance.
|
|
Note that the ext3 patch doesn't support data journalling
|
|
mode, so deleted metadata isn't zeroed. It also hasn't been tested
|
|
as thoroughly as the patch for ext2. And neither has been maintained for a
|
|
very long time.
|
|
<p>
|
|
However, the above techniques are only half the story: the empty free
|
|
blocks still consume space in the underlying filesystem, so something
|
|
must to be done to reclaim that space. For filesystems and kernels that
|
|
support it the fallocate call with <code>FALLOC_FL_PUNCH_HOLE</code> can
|
|
be used to deallocate space in a file.
|
|
<p>
|
|
An existing alternative is to use the sparse file handling capabilities
|
|
of the GNU <code>cp</code> command to take a copy of the filesystem image with
|
|
<code>cp --sparse=always</code> (though this does require the original
|
|
and sparse files to exist at the same time, which may be inconvenient).
|
|
<p>
|
|
As an alternative alternative I've written a utility which can make
|
|
any specified files on an ext2 filesystem
|
|
sparse, <a href="sparsify.c">sparsify.c</a>. This doesn't require any
|
|
additional disk space to work its magic, but it does require that the
|
|
filesystem containing the filesystem image is unmounted, which is just a
|
|
different sort of inconvenience.
|
|
<p>
|
|
<ul>
|
|
<li>The usual disclaimers apply: this worked for me when I tested it but it
|
|
might destroy your data.</li>
|
|
<li>Versions of libext2fs prior to 1.41 do not support ext4;
|
|
later versions should work with all filesystem types.</li>
|
|
<li>In 1.41 the progress monitor may exceed 100% as the iterator visits some
|
|
blocks more than once.</li>
|
|
<li>The code doesn't support huge (>2TB) files. It will issue a warning and
|
|
refuse to process them.</li>
|
|
</ul>
|
|
<p>
|
|
As an example, suppose we have an unmounted filesystem
|
|
image, <code>fs.image</code>, in the directory <code>/data</code>, which is the
|
|
root of the <code>/dev/hda2</code> filesystem. We can reclaim deleted
|
|
blocks and make it sparse like so:
|
|
<p>
|
|
<pre>
|
|
zerofree /data/fs.image
|
|
umount /data
|
|
sparsify /dev/hda2 /fs.image
|
|
mount /data
|
|
</pre>
|
|
<p>
|
|
<hr>
|
|
<address>
|
|
<a href="mailto:rmy@tigress.co.uk">Ron Yorston</a><br>
|
|
9th August 2012
|
|
</address>
|
|
</body>
|
|
</html>
|