Having issues getting stunnel client and server to connect.

TL;DR: ssh as root.

Also see:

Have been following this Digital Ocean guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-ssl-tunnel-using-stunnel-on-ubuntu

Problem Description

I am trying to create an stunnel connection from client to server. The stunnel client will route traffic from local port 2222 to local port 8000. This traffic will then be encrypted by Stunnel and sent out over the network, to the stunnel server on the remote machine, also listening for traffic and connections on port 8000. Once the traffic reaches the stunnel server, it is decrypted and forwarded to the server's local port 22, the SSH service.

This allows the execution of an SSH command to localhost that ultimately connects to the remote server:

[local] $ ssh -p 2222 zappa@localhost

...login message...

[remote] $

This is useful because:

You can wrap arbitrary traffic from any local port, and send it encrypted with SSL over any other port.
You can bypass any firewall that allows HTTPS traffic only by disguising your traffic using Stunnel.

Server

Server configuration

Here is the server stunnel.conf (Ubuntu):

output  = /var/log/stunnel4/stunnel.log
cert    = /etc/stunnel/stunnel.fullchain.pem
key     = /etc/stunnel/stunnel.key.pem
client  = no
debug   = 7
[ssh]
accept = 8000
connect = 127.0.0.1:22

The debug level of 7 is maximum and gives a more detailed description of what's happening in the log file.

Server behavior

On the server, starting the stunnel server using the following steps:

Check the stunnel conf:

$ cat /etc/stunnel/stunnel.conf

Open port 8000 if needed:

$ iptables -A INPUT -p tcp --dport 8000 -j ACCEPT

Stop previous stunnel instances and start a new one:

$ killall stunnel && stunnel

Look at the last few lines of the log to verify it is running correctly and bound to port 8000:

$ tail /var/log/stunnel4/stunnel.log
...
2017.03.29 04:59:15 LOG5[ui]: Configuration successful
2017.03.29 04:59:15 LOG7[ui]: Listening file descriptor created (FD=7)
2017.03.29 04:59:15 LOG7[ui]: Service [ssh] (FD=7) bound to 0.0.0.0:8000

Verification that server is ok

I can also see the open ports on the server using the netstat utility:

$ netstat -tulpn

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

(The SSH service is available on this machine on port 22, so we could just connect to the machine that way, but consider a scenario in which port 22 is blocked on a local network and port 8000 is not.)

Finally, an nmap scan of localhost and the server's IP also shows ports 22, 80, and 8000 open and listening:

$ nmap localhost; echo; nmap 196.116.112.336

Starting Nmap 7.01 ( https://nmap.org ) at 2017-03-29 05:12 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00016s latency).
Other addresses for localhost (not scanned): ::1
Not shown: 997 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
8000/tcp open  http-alt

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds


Starting Nmap 7.01 ( https://nmap.org ) at 2017-03-29 05:12 UTC
Nmap scan report for 196.116.112.336
Host is up (0.00013s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
8000/tcp open  http-alt

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

Client

Client configuration

Here is the client stunnel.conf (Mac):

output  = /var/log/stunnel4/stunnel.log
cert    = /usr/local/etc/stunnel/stunnel.fullchain.pem
key     = /usr/local/etc/stunnel/stunnel.key.pem
client  = yes
debug   = 7
[ssh]
accept  = 2222
connect = A.B.C.D:8000

Client behavior

On the client, these steps are for starting the stunnel server. These commands are run on Mac, with stunnel installed by homebrew to /usr/local/.

First check the config file:

$ cat /usr/local/etc/stunnel/stunnel.conf

(Not opening any firewalls or opening port 8000. Letting stunnel take care of that.)

Stop previous stunnel instances and start a new one:

$ killall stunnel && stunnel

Check the last few lines of the log to verify that everything is running correctly and bound to port 8000:

$ tail /var/log/stunnel4/stunnel.log
...
2017.03.28 22:11:17 LOG5[ui]: Configuration successful
2017.03.28 22:11:17 LOG7[ui]: Listening file descriptor created (FD=6)
2017.03.28 22:11:17 LOG7[ui]: Service [ssh] (FD=6) bound to 0.0.0.0:2222

Verifying the client is ok

Not sure how to run the equivalent of the above netstat command.

Checking open ports with nmap shows that port 2222

$ nmap localhost; echo; nmap 127.0.0.1

Starting Nmap 6.47 ( http://nmap.org ) at 2017-03-28 22:30 PDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00042s latency).
Not shown: 970 closed ports, 28 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
2222/tcp open  EtherNet/IP-1

Nmap done: 1 IP address (1 host up) scanned in 5.59 seconds


Starting Nmap 6.47 ( http://nmap.org ) at 2017-03-28 22:30 PDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.025s latency).
Not shown: 998 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
2222/tcp open  EtherNet/IP-1

Nmap done: 1 IP address (1 host up) scanned in 10.93 seconds

Can see port 2222 has been opened locally.

At this point we have verified that both the server and the client are configured okay, and listening on the correct ports.

Connect Over Stunnel Connection

Attempt 1

Error: Connection Reset by Peer

When I run an ssh command on the client, using port 2222, it's no die:

$ ssh -p 2222 zappa@localhost
ssh_exchange_identification: read: Connection reset by peer

The error message takes about 15 seconds to show up.

Reproducing Error

I was able to reproduce the error on another Mac running Homebrew stunnel, with the same configuration file.

$ ssh -p 2222 zappa@localhost
ssh_exchange_identification: read: Connection reset by peer

Attempt 2

Trying various configuration variations, in case that's the problem.

It's not.

Tried server configuration file specifying accept the following ways:

accept = A.B.C.D:8000
accept = 8000

I left server connect as 127.0.0.1:port

Tried the following client configuration files:

accept = 127.0.0.1
accept = 2222

I left client connect as A.B.C.D:8000.

NOTE: The correct notation is, in the client:

[ssh]
accept = 2222
connect = A.B.C.D:8000

and in the server:

[ssh]
accept = 8000
connect = 127.0.0.1:22

Attempt 3: Resolution

After scratching my head a while, I figured that stunnel was running on the remote machine as root, so maybe it was a problem with how I was doing ssh. I tried to ssh as root, and bingo:

$ ssh -p 2222 root@localhost
The authenticity of host '[localhost]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is SHA256:9DL2ohTkZFI9oaqUtMlA5X7gTJW/mmWbC+z7DyrZHzo.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:2222' (ECDSA) to the list of known hosts.
root@localhost's password:
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.9.7-x86_64-linode80 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Sat Mar 25 20:07:52 2017 from 104.200.154.81
root@localhost:~#

The lesson: if you are trying to tunnel a protocol through, need to pay attention to username and permissions.

Solution: Use Root to SSH

The solution to the problems I was experiencing was just to use the username "root" when connecting via ssh.

$ ssh -p 2222 root@localhost

This worked.

Flags

Stunnel/Troubleshooting

From charlesreid1

Contents