Compare commits
4 commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
86c260f563 | ||
|
87794a3afa | ||
| d5b09f8853 | |||
| 4067bb4c15 |
7 changed files with 402 additions and 21 deletions
1
.gitignore
vendored
Normal file
1
.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
|||
zerofree-1.0.1.tgz
|
||||
21
Makefile
21
Makefile
|
|
@ -1,21 +0,0 @@
|
|||
# Makefile for source rpm: zerofree
|
||||
# $Id$
|
||||
NAME := zerofree
|
||||
SPECFILE = $(firstword $(wildcard *.spec))
|
||||
|
||||
define find-makefile-common
|
||||
for d in common ../common ../../common ; do if [ -f $$d/Makefile.common ] ; then if [ -f $$d/CVS/Root -a -w $$/Makefile.common ] ; then cd $$d ; cvs -Q update ; fi ; echo "$$d/Makefile.common" ; break ; fi ; done
|
||||
endef
|
||||
|
||||
MAKEFILE_COMMON := $(shell $(find-makefile-common))
|
||||
|
||||
ifeq ($(MAKEFILE_COMMON),)
|
||||
# attept a checkout
|
||||
define checkout-makefile-common
|
||||
test -f CVS/Root && { cvs -Q -d $$(cat CVS/Root) checkout common && echo "common/Makefile.common" ; } || { echo "ERROR: I can't figure out how to checkout the 'common' module." ; exit -1 ; } >&2
|
||||
endef
|
||||
|
||||
MAKEFILE_COMMON := $(shell $(checkout-makefile-common))
|
||||
endif
|
||||
|
||||
include $(MAKEFILE_COMMON)
|
||||
129
index.html
Normal file
129
index.html
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>Keeping filesystem images sparse</title>
|
||||
</head>
|
||||
<body>
|
||||
<h2>Keeping filesystem images sparse</h2>
|
||||
<p>
|
||||
Filesystem images in local files can be used by many PC emulators and
|
||||
virtual machines
|
||||
(<a href="http://user-mode-linux.sourceforge.net/">user-mode Linux</a>,
|
||||
<a href="http://fabrice.bellard.free.fr/qemu/">QEMU</a> and
|
||||
<a href="http://wiki.xensource.com/xenwiki/">Xen</a>, to name but three).
|
||||
Typically these filesystems are created as sparse files using
|
||||
commands like:
|
||||
<p>
|
||||
<pre>
|
||||
dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0
|
||||
/sbin/mke2fs fs.image
|
||||
</pre>
|
||||
where the enormous <code>seek</code> value causes <code>dd</code> to move
|
||||
forward by 2GB before writing nothing at all. This results in the
|
||||
creation of a sparse file which takes disk space only for blocks which are
|
||||
actually used:
|
||||
<p>
|
||||
<pre>
|
||||
$ ls -l fs.image
|
||||
-rw-rw-r-- 1 rmy rmy 2048001024 Apr 18 19:10 fs.image
|
||||
$ du -s fs.image
|
||||
31692 fs.image
|
||||
</pre>
|
||||
As the filesystem is used, more and more of the non-existent blocks are
|
||||
filled with data and the size of the file on disk grows. Sometimes it would
|
||||
be nice to be able to reclaim unused blocks from a filesystem image. However,
|
||||
deleting files from the image doesn't return the space to the
|
||||
underlying filesystem: even free blocks in the image still consume space.
|
||||
Reclaiming the space can be achieved in two stages:
|
||||
<ul>
|
||||
<li>Fill unused blocks with zeroes
|
||||
<li>Make the file sparse again
|
||||
</ul>
|
||||
<p>
|
||||
One traditional way to zero unused blocks is to create a file that fills
|
||||
all the free space:
|
||||
<p>
|
||||
<pre>
|
||||
dd if=/dev/zero of=junk
|
||||
sync
|
||||
rm junk
|
||||
</pre>
|
||||
<p>
|
||||
The disadvantage of <code>dd</code> in this context is that it destroys
|
||||
any sparseness that exists: free blocks that were originally represented
|
||||
as holes in the image file are replaced with actual blocks containing
|
||||
zeroes.
|
||||
<p>
|
||||
As an alternative approach, and as practice in mucking about with ext2
|
||||
filesystems, I've written a utility which scans the free blocks in an
|
||||
ext2 filesystem and fills any non-zero blocks with zeroes.
|
||||
(The source, <a href="zerofree-1.0.1.tgz">zerofree-1.0.1.tgz</a>, is
|
||||
available for download.) The <code>zerofree</code> utility is faster
|
||||
than <code>dd</code>, especially when the filesystem is already partly sparse.
|
||||
The filesystem to be processed should be unmounted or mounted read-only.
|
||||
<p>
|
||||
Better than either of these would be to have the guest kernel keep the free
|
||||
blocks empty. My original inspiration was the
|
||||
<a href="http://www.uwsg.iu.edu/hypermail/linux/kernel/0401.3/1058.html">
|
||||
ext2fs privacy (i.e. secure deletion) patch</a> described in a Linux
|
||||
kernel mailing list thread. I've also made use of a later patch for ext3
|
||||
entitled
|
||||
<a href="http://marc.theaimsgroup.com/?l=linux-fsdevel&m=113986429313502&w=2">Secure Deletion Functionality in ext3</a>
|
||||
from the linux-fsdevel mailing list. (See also the authors' paper on
|
||||
<a href="http://www.filesystems.org/project-sdfs.html">Secure Deletion File Systems</a>.)
|
||||
I've modified the patches to make them more suitable for the present purpose.
|
||||
<ul>
|
||||
<li><a href="linux-2.6.25-zerofree2.patch">linux-2.6.25-zerofree2.patch</a> (for ext2 filesystems)</li>
|
||||
<li><a href="linux-2.6.25-zerofree3.patch">linux-2.6.25-zerofree3.patch</a> (for ext3 filesystems)</li>
|
||||
</ul>
|
||||
When a filesystem is mounted with the <code>zerofree</code> option (added
|
||||
by these patches) all the blocks freed when a file is deleted are filled
|
||||
with zeroes.
|
||||
Remember, this extra work will hurt disk performance.
|
||||
Note that the ext3 patch doesn't support data journalling
|
||||
mode, so deleted metadata isn't zeroed. It also hasn't been tested
|
||||
as thoroughly as the patch for ext2.
|
||||
<p>
|
||||
However, the above techniques are only half the story: the empty free
|
||||
blocks still consume space in the underlying filesystem, so something
|
||||
must to be done to reclaim that space. One approach would be to
|
||||
implement a system call, like the legendary
|
||||
<a href="http://www.uwsg.iu.edu/hypermail/linux/kernel/0106.3/1180.html">
|
||||
sys_punch</a>, which could be used to write a utility to make any
|
||||
suitable file sparse.
|
||||
<p>
|
||||
An existing alternative is to use the sparse file handling capabilities
|
||||
of the GNU <code>cp</code> command to take a copy of the filesystem image with
|
||||
<code>cp --sparse=always</code> (though this does require the original
|
||||
and sparse files to exist at the same time, which may be inconvenient).
|
||||
<p>
|
||||
As an alternative alternative I've written a utility which can make
|
||||
any specified files on an ext2 filesystem
|
||||
sparse, <a href="sparsify.c">sparsify.c</a>. This doesn't require any
|
||||
additional disk space to work its magic, but it does require that the
|
||||
filesystem containing the filesystem image is unmounted, which is just a
|
||||
different sort of inconvenience.
|
||||
<p>
|
||||
(The usual disclaimers apply: this worked for me
|
||||
when I tested it but it might destroy your data. Only use it on
|
||||
disposable filesystems, or have a full backup available. <code>e2fsck</code>
|
||||
is your friend.)
|
||||
<p>
|
||||
As an example, suppose we have an unmounted filesystem
|
||||
image, <code>fs.image</code>, in the directory <code>/data</code>, which is the
|
||||
root of the <code>/dev/hda2</code> filesystem. We can reclaim deleted
|
||||
blocks and make it sparse like so:
|
||||
<p>
|
||||
<pre>
|
||||
zerofree /data/fs.image
|
||||
umount /data
|
||||
sparsify /dev/hda2 /fs.image
|
||||
mount /data
|
||||
</pre>
|
||||
<p>
|
||||
<hr>
|
||||
<address>
|
||||
<a href="mailto:rmy@tigress.co.uk">Ron Yorston</a><br>
|
||||
29th July 2008
|
||||
</address>
|
||||
</body>
|
||||
</html>
|
||||
1
sources
1
sources
|
|
@ -0,0 +1 @@
|
|||
a8c772fdd134448f25ab4e7e12004595 zerofree-1.0.1.tgz
|
||||
193
sparsify.c
Normal file
193
sparsify.c
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
/*
|
||||
* sparsify - a tool to make files on an ext2 filesystem sparse
|
||||
*
|
||||
* Copyright (C) 2004 R M Yorston
|
||||
*
|
||||
* This file may be redistributed under the terms of the GNU General Public
|
||||
* License.
|
||||
*/
|
||||
#include <ext2fs/ext2fs.h>
|
||||
#include <stdio.h>
|
||||
#include <unistd.h>
|
||||
#include <string.h>
|
||||
|
||||
#define USAGE "usage: %s [-n] [-v] filesystem filename ...\n"
|
||||
|
||||
struct process_data {
|
||||
struct ext2_inode *inode ;
|
||||
unsigned char *buf;
|
||||
int dryrun ;
|
||||
int changed ;
|
||||
} ;
|
||||
|
||||
static int process(ext2_filsys fs, blk_t *blocknr, e2_blkcnt_t blockcnt,
|
||||
blk_t ref_block, int ref_offset, void *priv)
|
||||
{
|
||||
struct process_data *p ;
|
||||
errcode_t ret ;
|
||||
int i, group ;
|
||||
|
||||
p = (struct process_data *)priv ;
|
||||
|
||||
if ( blockcnt >= 0 ) {
|
||||
ret = io_channel_read_blk(fs->io, *blocknr, 1, p->buf);
|
||||
if ( ret ) {
|
||||
return BLOCK_ABORT ;
|
||||
}
|
||||
|
||||
for ( i=0; i < fs->blocksize; ++i ) {
|
||||
if ( p->buf[i] ) {
|
||||
break ;
|
||||
}
|
||||
}
|
||||
|
||||
if ( i == fs->blocksize && !p->dryrun ) {
|
||||
ext2fs_unmark_block_bitmap(fs->block_map, *blocknr) ;
|
||||
group = ext2fs_group_of_blk(fs, *blocknr);
|
||||
fs->group_desc[group].bg_free_blocks_count++;
|
||||
fs->super->s_free_blocks_count++ ;
|
||||
/* the inode counts blocks of 512 bytes */
|
||||
p->inode->i_blocks -= fs->blocksize / 512 ;
|
||||
*blocknr = 0 ;
|
||||
/* direct blocks need to be zeroed in the inode */
|
||||
if ( blockcnt < EXT2_NDIR_BLOCKS ) {
|
||||
p->inode->i_block[blockcnt] = 0 ;
|
||||
}
|
||||
p->changed = 1 ;
|
||||
return BLOCK_CHANGED ;
|
||||
}
|
||||
}
|
||||
|
||||
return 0 ;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv)
|
||||
{
|
||||
int verbose = 0 ;
|
||||
int dryrun = 0 ;
|
||||
errcode_t ret ;
|
||||
int flags ;
|
||||
int superblock = 0 ;
|
||||
int open_flags = EXT2_FLAG_RW ;
|
||||
int blocksize = 0 ;
|
||||
ext2_filsys current_fs = NULL;
|
||||
struct ext2_inode inode ;
|
||||
ext2_ino_t root, cwd, inum ;
|
||||
int i, c ;
|
||||
struct process_data pdata ;
|
||||
|
||||
while ( (c=getopt(argc, argv, "nv")) != -1 ) {
|
||||
switch (c) {
|
||||
case 'n' :
|
||||
dryrun = 1 ;
|
||||
break ;
|
||||
case 'v' :
|
||||
verbose = 1 ;
|
||||
break ;
|
||||
default :
|
||||
fprintf(stderr, USAGE, argv[0]) ;
|
||||
return 1 ;
|
||||
}
|
||||
}
|
||||
|
||||
if ( argc < optind+2 ) {
|
||||
fprintf(stderr, USAGE, argv[0]) ;
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
ret = ext2fs_check_if_mounted(argv[optind], &flags) ;
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: failed to determine filesystem mount state %s\n",
|
||||
argv[0], argv[optind]) ;
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
if ( flags & EXT2_MF_MOUNTED ) {
|
||||
fprintf(stderr, "%s: filesystem %s is mounted\n",
|
||||
argv[0], argv[optind]) ;
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
ret = ext2fs_open(argv[optind], open_flags, superblock, blocksize,
|
||||
unix_io_manager, ¤t_fs);
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: failed to open filesystem %s\n",
|
||||
argv[0], argv[optind]) ;
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
pdata.buf = (unsigned char *)malloc(current_fs->blocksize) ;
|
||||
if ( pdata.buf == NULL ) {
|
||||
fprintf(stderr, "%s: out of memory (surely not?)\n", argv[0]) ;
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
ret = ext2fs_read_inode_bitmap(current_fs);
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: error while reading inode bitmap\n", argv[0]);
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
ret = ext2fs_read_block_bitmap(current_fs);
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: error while reading block bitmap\n", argv[0]);
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
root = cwd = EXT2_ROOT_INO ;
|
||||
|
||||
for ( i=optind+1; i<argc; ++i ) {
|
||||
if ( verbose ) {
|
||||
printf("processing %s\n", argv[i]) ;
|
||||
}
|
||||
|
||||
ret = ext2fs_namei(current_fs, root, cwd, argv[i], &inum) ;
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: failed to find file %s\n", argv[0], argv[i]) ;
|
||||
continue ;
|
||||
}
|
||||
|
||||
ret = ext2fs_read_inode(current_fs, inum, &inode) ;
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: failed to open inode %d\n", argv[0], inum) ;
|
||||
continue ;
|
||||
}
|
||||
|
||||
if ( !ext2fs_inode_has_valid_blocks(&inode) ) {
|
||||
fprintf(stderr, "%s: file %s has no valid inodes\n", argv[0],
|
||||
argv[i]) ;
|
||||
continue ;
|
||||
}
|
||||
|
||||
pdata.inode = &inode ;
|
||||
pdata.dryrun = dryrun ;
|
||||
pdata.changed = 0 ;
|
||||
ret = ext2fs_block_iterate2(current_fs, inum, BLOCK_FLAG_DATA_ONLY,
|
||||
NULL, process, &pdata) ;
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: failed to process file %s\n", argv[0],
|
||||
argv[i]) ;
|
||||
continue ;
|
||||
}
|
||||
|
||||
if ( pdata.changed ) {
|
||||
ret = ext2fs_write_inode(current_fs, inum, &inode) ;
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: failed to write inode data %s\n", argv[0],
|
||||
argv[i]) ;
|
||||
continue ;
|
||||
}
|
||||
|
||||
ext2fs_mark_bb_dirty(current_fs) ;
|
||||
ext2fs_mark_super_dirty(current_fs) ;
|
||||
}
|
||||
}
|
||||
|
||||
ret = ext2fs_close(current_fs) ;
|
||||
if ( ret ) {
|
||||
fprintf(stderr, "%s: error while closing filesystem\n", argv[0]) ;
|
||||
return 1 ;
|
||||
}
|
||||
|
||||
return 0 ;
|
||||
}
|
||||
78
zerofree.spec
Normal file
78
zerofree.spec
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
Summary: Utility to force unused ext2 inodes and blocks to zero
|
||||
Name: zerofree
|
||||
Version: 1.0.1
|
||||
Release: 5%{?dist}
|
||||
License: GPL+
|
||||
Group: System Environment/Libraries
|
||||
|
||||
Source0: http://intgat.tigress.co.uk/rmy/uml/%{name}-%{version}.tgz
|
||||
Source1: http://intgat.tigress.co.uk/rmy/uml/sparsify.c
|
||||
Source2: http://intgat.tigress.co.uk/rmy/uml/index.html
|
||||
URL: http://intgat.tigress.co.uk/rmy/uml/
|
||||
|
||||
BuildRequires: e2fsprogs-devel
|
||||
|
||||
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
|
||||
|
||||
|
||||
%description
|
||||
zerofree is a utility to set unused filesystem inodes and blocks of an
|
||||
ext2 filesystem to zero. This can improve the compressibility and
|
||||
privacy of an ext2 filesystem.
|
||||
|
||||
This tool was inspired by the ext2fs privacy (i.e. secure deletion)
|
||||
patch described in a Linux kernel mailing list thread.
|
||||
|
||||
WARNING: The filesystem to be processed should be unmounted or mounted
|
||||
read-only. The tool tries to check this before running, but you
|
||||
should be careful.
|
||||
|
||||
|
||||
%prep
|
||||
%setup -q
|
||||
cp -p %{SOURCE1} .
|
||||
cp -p %{SOURCE2} .
|
||||
|
||||
|
||||
%build
|
||||
make CC="gcc $RPM_OPT_FLAGS"
|
||||
gcc $RPM_OPT_FLAGS sparsify.c -o sparsify -lext2fs
|
||||
|
||||
|
||||
%install
|
||||
rm -rf $RPM_BUILD_ROOT
|
||||
|
||||
install -D -p -m 755 zerofree $RPM_BUILD_ROOT%{_sbindir}/zerofree
|
||||
install -D -p -m 755 sparsify $RPM_BUILD_ROOT%{_sbindir}/sparsify
|
||||
|
||||
|
||||
%files
|
||||
%defattr(-,root,root,-)
|
||||
%doc COPYING index.html
|
||||
%{_sbindir}/zerofree
|
||||
%{_sbindir}/sparsify
|
||||
|
||||
|
||||
%clean
|
||||
rm -rf $RPM_BUILD_ROOT
|
||||
|
||||
|
||||
%changelog
|
||||
* Fri May 15 2009 Richard W.M. Jones <rjones@redhat.com> - 1.0.1-5
|
||||
- Include the index file as a source file.
|
||||
- Improve the description, remove spelling mistakes and other typos.
|
||||
- Use the upstream SRPM directly, unpacking source from it.
|
||||
- Fix use of dist macro.
|
||||
- Pass the RPM OPTFLAGS to C compiler (should also fix debuginfo pkg).
|
||||
- Use 'cp -p' to preserve timestamps when copying index.html file.
|
||||
- Fix the defattr line.
|
||||
- License is GPL+ (any version of the GPL including 1).
|
||||
- Use a simpler install command to install the binary.
|
||||
- Fix the upstream URL to point to the real original project.
|
||||
- Add the sparsify command.
|
||||
|
||||
* Thu May 14 2009 Richard W.M. Jones <rjones@redhat.com> - 1.0.1-1
|
||||
- Initial packaging for Fedora, based on R P Herrold's package.
|
||||
|
||||
* Wed May 13 2009 R P Herrold <info@owlriver.com> - 1.0.1-1
|
||||
- initial packaging
|
||||
Loading…
Add table
Add a link
Reference in a new issue