Rsync

From charlesreid1

Notes

I initially tried to use rsync on Mac OS X (Leopard and Snow Leopard both), but ran into some problems trying to rsync a large directory structure with many files (~10 GB) with a Linux server running Ubuntu.

I ran into some of the errors listed in the #Errors section below, so hopefully the solutions there will help you if you run into any of those issues. If you are using rsync for the first time and you follow the #Installation instructions, you should be able to avoid major issues.

Installation

In order to install a working/healthy rsync, you shouldn't use the rsync that comes with Mac (Leopard or Snow Leopard, both are broken). You should also avoid using the Fink version of rsync, for the same reason.

In order to install Rsrync, follow the instructions below.

Mac OS X

The problem with both Mac OS X rsync executables is that they are all earlier than version 3. There were some major improvements made with rsync version 3, so the version of rsync you use should be AT LEAST' version 3.0.

The latest version of rsync is available here: http://rsync.samba.org/ftp/rsync/

Mac OS X 10.5 (Leopard)

First: download the latest version of rsync from http://rsync.samba.org/ftp/rsync/ (called rsync.X.Y.Z.tar.gz)

Second: download the patches available for rsync (same location, called rsync-patches.X.Y.Z.tar.gz)

To apply the patches to the source code and install, run these commands: 

#!/bin/sh

mkdir /tmp/rsync
cd /tmp/rsync

# Download the tarballs
wget http://rsync.samba.org/ftp/rsync/rsync-3.0.8.tar.gz
wget http://rsync.samba.org/ftp/rsync/rsync-patches-3.0.8.tar.gz

# Untar the tarballs
tar xvzf rsync-3.0.8.tar.gz
tar xvzf rsync-patches-3.0.8.tar.gz

# Move to the source code directory location
mv rsync-3.0.8.tar.gz ~/pkg/_sources/.
cd ~/pkg/_sources/.

# Apply patches
patch -p1 < patches/atimes.diff
patch -p1 < patches/crtimes-64bit.diff
patch -p1 < patches/crtimes-hfs.diff
patch -p1 < patches/crtimes.diff
patch -p1 < patches/fileflags.diff
patch -p1 < patches/osx-xattr-nodev.diff

# Run configure 
./configure \
 --prefix="${HOME}/pkg/rsync/3.0.7" \
 --with-rsh=rsh

Mac OS X 10.6 (Snow Leopard)

Follow instructions given above.

Ubuntu Linux

If you use Aptitude, you can run apt-get install rsync. This will install the following version of rsync: 

$ rsync --version
rsync  version 3.0.4  protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

As far as I know, this version works as expected. Your rsync should be above version 3.


Usage

Mac and Ubuntu

I have a Ubuntu backup server that I want to use to sync files between two Macs. I can do this with two rsync commands, both run periodically from the Macs: one to push files, the other to pull files.

Push Command

#!/bin/sh

echo "rsync: pushing" >> $LOG
echo "`date "+%Y-%m-%d"` " >> $LOG


~/pkg/rsync/std/bin/rsync \
    --progress \
    -e ssh \
    --bwlimit=300 \
    --log-file=/tmp/tmprsync.log \
    -a \
    -r \
    -f"+ */" \
    -f"+ *.pdf" \
    -f"+ *.bib" \
    -f"- *" \
    ~/local_dir/ \
    charles@charlesmartinreid.com:~/remote_dir

cat /tmp/tmprsync.log >> $LOG

Note that the first directory has a trailing slash and the second does not. This is important.

Pull Command

#!/bin/sh

echo "" >> $LOG
echo "rsync: pulling" >> $LOG
echo "`date "+%Y-%m-%d"` " >> $LOG

~/pkg/rsync/std/bin/rsync \
    --progress \
    -e ssh \
    --bwlimit=300 \
    --log-file=/tmp/tmprsync.log \
    -a \
    -r \
    -f"+ */" \
    -f"+ *.pdf" \
    -f"+ *.bib" \
    -f"- *" \
    charles@charlesmartinreid.com:~/remote_dir/ \
    ~/local_dir

cat /tmp/tmprsync.log >> $LOG


Errors

Bus Error/Segmentation Fault

Initially, when I tried to rsync large directory/tree structures, I ran into a "Bus Error" (equivalent to a "Bus Error/Segmentation Fault" error) before it transferred files, or after it had transferred only a small number of files (between 10 and 30 files).

This page [1] says this is caused by a "buffer overrun", meaning it tries to prep the list of files it's going to transfer, but there are too many (or they are too big), and it overfills the buffer.

The solution that is specified by the same page (http://www.lartmaker.nl/rsync/), is to install a patched version of rsync 2.6.3.

HOWEVER! This patched version has several problems. First, it is a very old version of rsync. Second, it fixes one problem, but causes another.

Corrupted MAC on input

Once I had tried the above solution, I got a new error, with error text that looks like this: 

Received disconnect from 155.97.192.16: 2: Corrupted MAC on input.
rsync: writefd_unbuffered failed to write 32768 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (8874 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(453) [sender=2.6.9]

This problem is likely due to lack of memory, based on information here: [2]

Here [3] someone says you should see it say that it is incrementally updating files.

But here [4] it says that that only happens with rsync version 3+: 

-r, --recursive
    This tells rsync to copy directories recursively. See also --dirs (-d). 
Beginning with rsync 3.0.0, the recursive algorithm used is now an
    incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.

This led me to the conclusion that the underlying issue was that I was using rsync version 2, NOT version 3. I installed version 3 by hand. I'm not entirely sure this has solved all my problems, though.

Follow-Up

Upgrading to rsync version 3.0.7 did not fix this problem. I still see this: 

Received disconnect from 155.97.192.16: 2: Corrupted MAC on input.

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (329454 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
wiki.charlesmartinreid.com: Operation timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]

I think this is actually a different problem with a different solution. Adding -e ssh fixed it.

More Errors

sent 448940044 bytes  received 818 bytes  305090.63 bytes/sec
total size is 11987303971  speedup is 26.70
wiki.charlesmartinreid.com: Operation timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]

More "Corrupted MAC on Input" Errors

Received disconnect from 155.97.192.16: 2: Corrupted MAC on input.

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (351 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]

Resources

Retrieved from "https://charlesreid1.com/w/index.php?title=Rsync&oldid=3066"