SSH tunnels part 3: Reverse tunnels

In the previous two posts, I laid out the basics of forward ssh tunnels. Now it’s time to delve into the dark magic of the reverse tunnel. For reference, here are the first two articles:
* Intro to ssh tunnels
* Chaining ssh tunnels together
The problem addressed in this post is the same as the previous two, except that the remote administrator won’t open up *any* ports in their firewall. Machines on the customer’s network can connect out onto the internet, but there is no direct way to connect in from out here. This is remarkably common, either because the admin only has a limited number of public IP addresses, or because most organizations are very private with their machines. This post describes how to set up a pair of tunnels that meet in the middle on my co-located server, allowing me access to machines that don’t even have a public IP address.
Ssh tunnels are directional. When I open a tunnel with “-L”, I am opening a path from the originating machine forward to the target. The other tunnel argument, “-R”, opens a reverse tunnel from a port on the destination machine back to the originator. In the second post I set up a connection letting me see into remote.customer.com from demo.bioteam.net. However, someone sitting at remote.customer.com wouldn’t be able to see back down that tunnel.
It’s worth noting that at this point we run a risk of violating security policies at the customer site. By enlisting a co-conspirator inside the firewall, we’re getting access to resources that are very clearly supposed to be inaccessible. Bioteam’s practice is to be totally above board about what we are doing and why we’re doing it. Many security / networking folks agree that the goal of a network setup like this is to prevent unauthorized access. In that circumstance, we can frequently agree that my access is authorized, and that it’s simpler for short term or one time connections to use ssh tunnels to get the job done. However, if the policy is actually to prevent any remote access, there’s no getting around it. In those cases, I get in a car or on a plane. Many of my long term clients have started off giving me reverse ssh tunnels while we waited for the organizational paperwork on a “real” solution like VPN access.
Short version: There are times when it’s better to ask forgiveness than permission. Dealing with network security is not one of them. Building trust with the people responsible for a computing network requires both honesty and demonstrated competence over a period of time. If you sneak around behind people’s backs, you will eventually get burned.
The target machine is called: customer.private
I have an account on that machine: bioteam
Bioteam’s co-located server is still: demo.bioteam.net
My account on bioteam’s server: cdwan
The customer also gets an account on bioteam’s server: customer
The customer needs to be logged into customer.private and connect from there to demo.bioteam.net like this:
ssh demo.bioteam.net -l customer -R 9001:customer.private:80
That specification has three parts:
* 9001: The port on demo.bioteam.net on which the tunnel listens
* customer.private: The machine on which the tunnel ends. This is the part where terms like “localhost” get really confusing. In this case “localhost” would work just as well, but I’m making it explicit.
* 80: The port where the tunnel ends.
From my laptop, I open up a forward tunnel exactly as before:
ssh demo.bioteam.net -l cdwan -L 9000:localhost:9001
That’s it. We’ve got a tunnel from my side to port 9001 on demo.bioteam.net, and another tunnel from 9001 to port 80 on customer.private. I connect to http://localhost:9000 on my laptop, and I should see the webserver on customer.private. We’ve created the same chained pair of tunnels from the previous post, but with one originating on each side rather than me pushing both of them forward.
The attentive reader may ask “if there’s a tunnel listening at demo.bioteam.net:9001, can’t you just connect your web server directly to http://demo.bioteam.net:9001? This is a good question. The answer is “yes, but you need to explicitly ask for this functionality.” By default, ssh tunnels only listen to requests originating from within the machine itself. The tunnel listens on what is called the “loopback” interface and is not visible from outside the machine. While it’s possible to get around this, I’ve found it to be a universally bad idea in my work with Bioteam. It represents enough of a potential security hole that I just don’t do it.
Notice also that the customer didn’t have to give me an ssh account on their machine. Instead, we both connected to some machine in the middle (demo.bioteam.net) and I was able to see their web server. As before, as soon as either ssh connection closes, the access is gone. This has a pleasing symmetry insofar as I’m asking for access by giving the customer an account on my machine.
Now for the meat of it: Assume that the customer wants to let me log in via ssh. This means that the reverse tunnel needs to terminate at the ssh port (22) rather than the http port (80). Otherwise it’s exactly the same:
ssh demo.bioteam.net -l customer -R 9002:customer.private:22
With that in place, I can log in to demo.bioteam.net and then run this:
ssh localhost -l bioteam -p 9002
If everything has gone correctly, I will get a password prompt, type my password for cutomer.private, and see a shell prompt for a machine without a public internet IP.
Of course, I can create my own tunnels using that connection. For example, with the reverse tunnel from above in place, I can set up my very own tunnel from port 9010 on my laptop, through port 9011 on demo.bioteam.net, to the webserver on remote.customer.com. This is the part where the “localhosts” get almost impenetrably confusing:
ssh demo.bioteam.net -l cdwan -L 9010:localhost:9011
ssh localhost -l bioteam -p 9002 -L 9011:localhost:80