Stunnel/Rsync

This page was immensely helpful: http://www.netbits.us/docs/stunnel_rsync.html

NOTE: I was not able to get this working reliably on port 273, but was able to get stunnel+rsync working reliably over port 443. Mileage may vary.

Overview/Approach

Used the following approach:

• Reorganized all of the files in the repository so that each service had its own configuration and build sequence
• Makefile was entry point: make rsync or make ssh
• The Makefile would set the port, and feed that as an argument to each script.
• Build script, run script, open firewall script
• This enabled flexibility in port number, and enabled switching from 273 to 443 more easily
• Better organization of stunnel - since each service requires different configurations and ports should be easy to switch in and out.

Server

• Server:
  • Adding secure rsync protocol
  • Setting up rsync server config (which dir to rsync to)
  • Running rsync service
  • Setting up stunnel config
  • Running stunnel service

Rsync

Configure Rsync Server

Main repository: https://git.charlesreid1.com/docker/d-stunnel

To configure the rsync stunnel server:

• First, define a universal rsync stunnel configuration file.
• Everywhere you need to specify the numerical port, use the token PORT
• The Makefile will ultimately set the port number, and pass that into the build script. The build script will use sed to find/replace the token PORT with the numerical port.

Makefile that calls build and run scripts: https://git.charlesreid1.com/docker/d-stunnel/src/master/Makefile

Build rsync stunnel docker container script: https://git.charlesreid1.com/docker/d-stunnel/src/master/build_rsync.sh

The build docker container script actually uses the Dockerfile, which assembles the various files that are needed to run both stunnel and rsync. The build script updates the port number in both the Dockerfile and in the rsync stunnel configuration file using sed, then bakes these files into the final container image using docker build.

Dockerfile_rsync_stunnel: https://git.charlesreid1.com/docker/d-stunnel/src/master/Dockerfile_rsync_stunnel

Universal rsync stunnel configuration file: https://git.charlesreid1.com/docker/d-stunnel/src/master/stunnel.server.rsync.conf

Finally, the Makefile calls the run script and passes the port number to the run script, which hooks up the necessary ports between the host and the container, mounts the necessary volumes, and sets up the docker network interface (as well as running the container in the foreground or background).

Run rsync stunnel container: https://git.charlesreid1.com/docker/d-stunnel/src/master/run_rsync.sh

Running Rsync Server

Once you have set the rsync configuration file, the rsync daemon will run like any standard system service.

$ sudo service rsync start

Alternatively, you can get an rsync docker container up and running: Docker/Rsync

Adding Tcp Wrappers for Secure Rsync Protocol

The next thing we need to do is give the system a bit more information about the protocol we are using. Rsync is just tcp traffic, so we can define a new service and tell the system what port and protocol it uses. We do this using /etc/services.

Add the following line to the server's /etc/services (match it exactly):

ssyncd 273/tcp # secure rsync over stunnel

(Optional: may need to add this to /etc/hosts.allow but this messed things up for me:

ssyncd : A.B.C.D

where A.B.C.D is the server's IP address.)

See https://git.charlesreid1.com/docker/d-stunnel/src/master/stunnel.server.rsync_over_273.conf for details

Stunnel Server for Rsync

Configure Stunnel Server for Rsync

The rsync daemon will run on port 873. The stunnel connection we will set up will be over port 273. The stunnel configuration file will accept connections on 273 and connect them to port 873.

Note these require the rsync service to be defined following the above steps.

Server stunnel.conf file for rsync over port 273:

output	= /var/log/stunnel4/stunnel.log
cert	= /etc/stunnel/stunnel.fullchain.pem
key		= /etc/stunnel/stunnel.key.pem
pid		= /var/run/stunnel4/stunnel.pid
client	= no

debug = 7
foreground = yes

[rsync]
accept = 273
connect = 873

See https://git.charlesreid1.com/docker/d-stunnel/src/master/stunnel.server.rsync_over_273.conf

Running Stunnel Server for Rsync

Fire up stunnel with the stunnel command:

[remote] $ stunnel

Ideally, nothing will happen. Errors may be printed to the screen, or stunnel may fail silently. Check the log to make sure stunnel is running okay. If you set foreground = yes in the config file, the console will display the contents of the log. Otherwise, check the log via:

[remote] $ tail -f /var/log/stunnel4/stunnel.log

Note that you still won't be able to reach your stunnel server from the outside until you expose the port with iptables.

Opening Server Firewall to Stunnel

Need to open firewall to incoming connections on the stunnel rsync port (273):

[remote] $ iptables -A INPUT -p tcp --dport 273 -j ACCEPT
[remote] $ iptables -A FORWARD -p tcp -j ACCEPT --dport 273 -m state --state NEW

Client

This describes the client configuration required to get rsync working over stunnel.

Rsync Client

The client's rsync will operate, as usual, on local port 873. Behind the scenes, stunnel will transparently map that port to a remote rsync daemon/server, so it will appear to the rsync client that it is communicating with a local rsync daemon/server.

Configuring Rsync Client

No special rsync setup is needed for the client. All we have to do is point the rsync client to the rsync server, and specify one of the modules in our configuration file. The rsync daemon running on the remote server will take care of the rest.

Running Rsync Client

The client does not need to run the rsync daemon, only the server does. When you are on the client, you can just run rsync from the command line, and pass it the location of the rsync server (which is localhost port 873). Stunnel takes care of the rest, so the rsync daemon on the remote server appears to be just another local service.

The rsync server should appear to be a local service on port 873. You can check the local rsync port is being mapped correctly to the remote rsync daemon/server by running:

[client] $ rsync -vv localhost::

This runs rsync and connects to the server at localhost (since no host is specified, it defaults to 873). This is where stunnel is listening and forwarding traffic between the client and server. If the server is listening you should get a simple success message back:

[client] $ rsync -vv localhost::
opening tcp connection to localhost port 873
sending daemon args: --server --sender -vvde.Lsfx . /  (5 args)
pi             	Raspberry Pi Wifi Data

Here is the rsync script that syncs wifi data between Raspberry Pis and the remote server: https://git.charlesreid1.com/rpi/pi-transmission/src/master/rsync_with_server.sh

The rsync_with_server.sh script wraps the basic functionality of running rsync a single time. This script is ultimately called by check_stunnel.sh, which is a "forever loop" that opens the stunnel connection and runs rsync over stunnel to update the server with the latest wifi data from the Raspberry Pi's wifi sensor. We'll cover that script below.

Adding Tcp Wrappers for Secure Rsync Protocol

Add the following to /etc/services on the client:

ssync 273/tcp

(Optional: you may need to add the following to the client /etc/hosts.allow. This actually caused problems for me.)

ssync : LOCAL

See stunnel for rsync config file for Raspberry Pi clients: https://git.charlesreid1.com/rpi/pi-stunnel/src/master/stunnel.rpiclient.rsync_over_273.conf

Stunnel Client

Configuring Stunnel Client for Rsync

Client stunnel over rsync configuration file goes here.

Running Stunnel Client for Rsync

The usual - run stunnel using the stunnel command.

To monitor what is happening use debug = 7 and foreground = yes.

Forever Loop: Listening for Client Changes

To listen for changes to the folder containing wifi data on the Raspberry Pi clients, we want to have a loop that runs forever: it should keep the stunnel connection open if it fails, and it should run rsync over stunnel to keep the latest wifi data (CSV files) up-to-date on the remote server.

It should also be resilient to failure - if the stunnel connection fails (if the Raspberry Pi can't find a wifi network to connect to), the program should not crash but should continue running in the background and attempt to connect after a periodic wait.

This script is contained in check_stunnel.sh in the Raspberry Pi pi-transmission repository: https://git.charlesreid1.com/rpi/pi-transmission/src/master/check_stunnel.sh

Here's that relatively modest script:

#!/bin/bash
#
# Check if stunnel is running. 
# If not, try to start it.
# In any case, try to rsync.
# Then take a nap.

while true; do

	echo "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" 
	if [[ "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" == "" ]]; then
		echo "Stunnel not running, trying to start it..."
		stunnel
		sleep 10
		if [[ "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" == "" ]]; then
			echo "No die!"
		fi
	fi

	# If stunnel is not running, this will raise an error and keep going.
	# Do rsync here
	./rsync_with_server.sh

	# wait X seconds, then do it again.
	sleep 300;

done

Debugging

Debugging problems with the two interacting stunnel-rsync layers can get tricky. Here's a good workflow.

Debugging stunnel

First, if you want to see what stunnel is actually doing, add the following to the stunnel configuration file:

debug = 7
foreground = yes

These will run stunnel in the foreground and print out lots of information. You can run this in a terminal window, then open another window and run rsync commands. You should see activity in the stunnel window, indicating it is initiating a connection with the server and passing traffic.

You can do the same thing on the server to monitor the server instance of stunnel, so if you need to troubleshoot a problem on the server side, edit the server stunnel configuration file and add the debug and foreground options.

Debugging rsync

If you are confident stunnel is working properly and that the problem is with rsync, you can monitor rsync using the system log. rsync does not log to its own log file.

By running tail -f /var/log/syslog on the server in a window, then running rsync over stunnel commands in another, you should see messages about rsync activity showing up in the syslog. This should also give you more helpful and descriptive information when things go wrong, and help you diagnose the error.