Stunnel/Rsync

Server

• Server:
  • Adding secure rsync protocol
  • Setting up rsync server config (which dir to rsync to)
  • Running rsync service
  • Setting up stunnel config
  • Running stunnel service

Rsync

Configure Rsync Server

On the server, we want to run an rsync daemon that will listen for incoming requests to synchronize files. The daemon will compare incoming files to the current copies of files, and update any changes it does not have.

To run an rsync daemon, edit the rsync config file at /etc/rsyncd.conf.

In the config file you will set the name of your module, which has a specific data directory on the server. You can potentially have multiple clients syncing to multiple locations on the server by using multiple modules, but we are just using one.

See rsyncd.conf in https://charlesreid1.com:3000/rpi/pi-transmission

Running Rsync Server

Once you have set the rsync configuration file, the rsync daemon will run like any standard system service.

$ sudo service rsync start

Adding Tcp Wrappers for Secure Rsync Protocol

The next thing we need to do is give the system a bit more information about the protocol we are using. Rsync is just tcp traffic, so we can define a new service and tell the system what port and protocol it uses. We do this using /etc/services.

Add the following line to the server's /etc/services (match it exactly):

ssyncd 273/tcp # secure rsync over stunnel

(Optional: may need to add this to /etc/hosts.allow but this messed things up for me:

ssyncd : A.B.C.D

where A.B.C.D is the server's IP address.)

See https://charlesreid1.com:3000/docker/d-stunnel/src/master/stunnel.server.rsync_over_273.conf for details

Stunnel Server for Rsync

Configure Stunnel Server for Rsync

The rsync daemon will run on port 873. The stunnel connection we will set up will be over port 273. The stunnel configuration file will accept connections on 273 and connect them to port 873.

Note these require the rsync service to be defined following the above steps.

Server stunnel.conf file for rsync over port 273:

output	= /var/log/stunnel4/stunnel.log
cert	= /etc/stunnel/stunnel.fullchain.pem
key		= /etc/stunnel/stunnel.key.pem
pid		= /var/run/stunnel4/stunnel.pid
client	= no

debug = 7
foreground = yes

[rsync]
accept = 273
connect = 873

See https://charlesreid1.com:3000/docker/d-stunnel/src/master/stunnel.server.rsync_over_273.conf

Running Stunnel Server for Rsync

Fire up stunnel with the stunnel command:

[remote] $ stunnel

Ideally, nothing will happen. Errors may be printed to the screen, or stunnel may fail silently. Check the log to make sure stunnel is running okay. If you set foreground = yes in the config file, the console will display the contents of the log. Otherwise, check the log via:

[remote] $ tail -f /var/log/stunnel4/stunnel.log

Note that you still won't be able to reach your stunnel server from the outside until you expose the port with iptables.

Opening Server Firewall to Stunnel

Need to open firewall to incoming connections on the stunnel rsync port (273):

[remote] $ iptables -A INPUT -p tcp --dport 273 -j ACCEPT
[remote] $ iptables -A FORWARD -p tcp -j ACCEPT --dport 273 -m state --state NEW

Client

This describes the client configuration required to get rsync working over stunnel.

Rsync

The client's rsync will operate, as usual, on local port 873. Behind the scenes, stunnel will transparently map that port to a remote rsync daemon/server, so it will appear to the rsync client that it is communicating with a local rsync daemon/server.

Configuring Rsync

No special rsync setup is needed for the client. All we have to do is point the rsync client to the rsync server, and specify one of the modules in our configuration file. The rsync daemon running on the remote server will take care of the rest.

Running Rsync

The client does not need to run the rsync daemon, only the server does. When you are on the client, you can just run rsync from the command line, and pass it the location of the rsync server (which is localhost port 873). Stunnel takes care of the rest, so the rsync daemon on the remote server appears to be just another local service.

The rsync server should appear to be a local service on port 873. You can check the local rsync port is being mapped correctly to the remote rsync daemon/server by running:

[client] $ rsync -vv localhost::

This runs rsync and connects to the server at localhost (since no host is specified, it defaults to 873). This is where stunnel is listening and forwarding traffic between the client and server. If the server is listening you should get a simple success message back:

[client] $ rsync -vv localhost::
opening tcp connection to localhost port 873
sending daemon args: --server --sender -vvde.Lsfx . /  (5 args)
pi             	Raspberry Pi Wifi Data

Here is the rsync script that syncs wifi data between Raspberry Pis and the remote server: https://git.charlesreid1.com/rpi/pi-transmission/src/master/rsync_with_server.sh

The rsync_with_server.sh script wraps the basic functionality of running rsync a single time. This script is ultimately called by check_stunnel.sh, which is a "forever loop" that opens the stunnel connection and runs rsync over stunnel to update the server with the latest wifi data from the Raspberry Pi's wifi sensor. We'll cover that script below.

Adding Tcp Wrappers for Secure Rsync Protocol

Add the following to /etc/services on the client:

ssync 273/tcp

(Optional: you may need to add the following to the client /etc/hosts.allow. This actually caused problems for me.)

ssync : LOCAL

See stunnel for rsync config file for Raspberry Pi clients: https://charlesreid1.com:3000/rpi/pi-stunnel/src/master/stunnel.rpiclient.rsync_over_273.conf

Stunnel Client for Rsync

Configuring Stunnel Client for Rsync

Client stunnel over rsync configuration file goes here.

Running Stunnel Client for Rsync

The usual - run stunnel using the stunnel command.

To monitor what is happening use debug = 7 and foreground = yes.

Forever Loop: Listening for Client Changes

To listen for changes to the folder containing wifi data on the Raspberry Pi clients, we want to have a loop that runs forever: it should keep the stunnel connection open if it fails, and it should run rsync over stunnel to keep the latest wifi data (CSV files) up-to-date on the remote server.

It should also be resilient to failure - if the stunnel connection fails (if the Raspberry Pi can't find a wifi network to connect to), the program should not crash but should continue running in the background and attempt to connect after a periodic wait.

This script is contained in check_stunnel.sh in the Raspberry Pi pi-transmission repository: https://git.charlesreid1.com/rpi/pi-transmission/src/master/check_stunnel.sh

Here's that relatively modest script:

#!/bin/bash
#
# Check if stunnel is running. 
# If not, try to start it.
# In any case, try to rsync.
# Then take a nap.

while true; do

	echo "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" 
	if [[ "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" == "" ]]; then
		echo "Stunnel not running, trying to start it..."
		stunnel
		sleep 10
		if [[ "$(ps aux | grep [s]tunnel | grep -v check_stunnel)" == "" ]]; then
			echo "No die!"
		fi
	fi

	# If stunnel is not running, this will raise an error and keep going.
	# Do rsync here
	./rsync_with_server.sh

	# wait X seconds, then do it again.
	sleep 300;

done

Debugging

Debugging problems with the two interacting stunnel-rsync layers can get tricky. Here's a good workflow.

Debugging stunnel

First, if you want to see what stunnel is actually doing, add the following to the stunnel configuration file:

debug = 7
foreground = yes

These will run stunnel in the foreground and print out lots of information. You can run this in a terminal window, then open another window and run rsync commands. You should see activity in the stunnel window, indicating it is initiating a connection with the server and passing traffic.

You can do the same thing on the server to monitor the server instance of stunnel, so if you need to troubleshoot a problem on the server side, edit the server stunnel configuration file and add the debug and foreground options.

Debugging rsync

If you are confident stunnel is working properly and that the problem is with rsync, you can monitor rsync using the system log. rsync does not log to its own log file.

By running tail -f /var/log/syslog on the server in a window, then running rsync over stunnel commands in another, you should see messages about rsync activity showing up in the syslog. This should also give you more helpful and descriptive information when things go wrong, and help you diagnose the error.

• Debugging:
  • How to debug stunnel
  • How to debug rsync
  • Workflow for checking connections while running commands