Moving large directories

I had a need to move a large directory tree on my Linux server. For this there are a number of options:

Using ‘mv’

Of course you can just issue ‘mv /sourcedir /destinationdir’ and be done with it. The downside is that if you interrupt the process, both source and target directories will be left in an inconsistent state. There is no easy way to resume the process.

Using rsync

Rsync is a Swiss army knife for a number of file-related operations and of course you can use it for local move operations as well. Rsync offers the big improvement of being able to interrupt and resume the move process in a smart and safe way. One of the limitations hower is that, even though it can delete the source files, it will leave you with a source directory full of empty subdirs. First of all, let’s move all files:

rsync -avr --remove-source-files /sourcedir/ /destinationdir/

Note the ‘–remove-source-files’, it does exactly what you think it does (after files have been successfully transfered). So what to do afterwards with the tree of empty subdirs? This is a nice trick I learned:

rsync -av --delete  `mktemp -d`/ /sourcedir/

This effectively syncs an empty directory to your sourcedir and from my (and other peoples experience) this is actually the quickest way to delete a large directory tree, even if there are files in it. It is supposed to be 15% quicker than ‘rm -rf’ due to ordering advantages but I’ll let you decide this for yourself.

Using tar and rm

While the rsync solution seems nice, it sometimes is a bit slow between two local disks. You can of course do ‘cp -a /sourcedir /targetdir’ beforehand and rsync afterwards but it seems to be even quicker to use tar for this purpose:

(cd /sourcedir ; tar cf - . ) | (cd /destinationdir ; tar xvpf -)

I read this trick on Stackoverflow and it seems to be a bit quicker indeed. I’ll let you decide this for yourself as well :)

Conclusion

For my moving task, I actually decided to combine both the ‘tar’ and ‘rsync’ tricks. This made for a quick copy, followed by rsync checking if everything was in sync and deleting the source files. Afterwards I used the ‘rsync to empty dir’ method to quickly delete all empty subdirs in the source directory.