Docker/Network Debugging

April 19 Debugging

Cannot reach container, no netstat in the container to check if sockets are open.

Troubleshooting: here and here are both useful pages.

To run netstat in the container, even if netstat is not available in the container, run

$ sudo nsenter -t `docker inspect -f '{{.State.Pid}}'` -n netstat

Can also use docker port command to check which ports are open and exposed to the host:

$ docker port container_name

Went through some debugging with Iptables rules, and dropping a bunch of old crusty rules that were not being used/applied. This did not affect the firewall/connection in any way.

Really frustrating. Port 8888 simply will not open up to the outside world. Port 8081 works fine and is open from the container to the outside world (MongoExpress), but has no rules set in the firewall (incoming packets set to accept all). Same thing with port 28017, this port is also open. There is one rule for it, exactly the same as the one rule for port 8888.

Changing the port number to 9990 did not affect the outcome either. The container is still unreachable, even from the localhost. No port 9990 open anywhere, when scanning from a remote machine and when scanning from the local machine.

Nothing working. No idea.

And of course, two minutes later, everything just magically works somehow, after the fifth time trying the exact same set of commands.

Nmap was not helping here: port 9990 was open but was not showing up in any nmap scans. See:

$ nmap reidmachine.party

Starting Nmap 6.47 ( http://nmap.org ) at 2017-04-19 23:02 UTC
Nmap scan report for reidmachine.party (96.126.102.36)
Host is up (0.00082s latency).
rDNS record for 96.126.102.36: li335-36.members.linode.com
Not shown: 998 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
8081/tcp open  blackice-icecap

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

but when port 9990 is specified, it discovers it:

$ nmap -p 9990 reidmachine.party

Starting Nmap 6.47 ( http://nmap.org ) at 2017-04-19 23:03 UTC
Nmap scan report for reidmachine.party (96.126.102.36)
Host is up (0.00046s latency).
rDNS record for 96.126.102.36: li335-36.members.linode.com
PORT     STATE SERVICE
9990/tcp open  osm-appsrvr

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

Lesson learned: if you know what port you're scanning for with nmap, SPECIFY IT!