Techexams Preface:
This is a long post. A very long post. I apologize, but I hope the information is good. I'm going to go back to writing a blog, and I'm presently generating content for it, and I'm going to use y'all to vet it. This isn't really Off Topic, but it touches on so many different topics, that this is probably the most appropriate forum for it.
If you have any questions, corrections, or insights that I missed, please feel free to share! This is information I've compiled from years of experience, and what seems obvious to me, may not be obvious to the rest of the world
Introduction
SSH tunneling is something I've seen a lot of examples of over the years, but the explanations and usage guidelines have been somewhat vague. I suspect someone investigated it enough to do what they want, tossed a couple quick notes together for future reference, and figured 'hey, this might make a good blog post!'. The few texts I've seen that do cover it in depth are heavy on the tech and geek speak, but could afford to be a little more practical.
I'm going to endeavor to be a little more complete, and a little more educational, and maybe, just a little more scary. I'm going to explain the commandlines I use, what the options do, and provide some usage examples. And that's it. I'm not going to go into theory, or history, or anything else.
All of my examples will be openSSH based, because that's what I'm familiar with. I'm sure the same thing is possible with other SSH implementations, but don't ask me how - I don't use them, so I don't know. As well, all of my examples will user ports over 1024. In a unix environment, only root can bind to a port lower than 1024.
So what is SSH tunneling?
In a nutshell, you're connecting two endpoints over the SSH protocol so that it appears they're directly connected, even if that's not the case.
Why would you want to do this? The first, and most obvious benefit, is that any traffic which traverses the tunnel is encrypted. This guarantees a reasonable degree of privacy when using network services over a network you don't trust. For example, let's say that you're on a public wifi network. And you want to check your email. The only thing they have open is port 110 and 143, because the guy who configured the AP is unaware of the existence of POPS and IMAPS. That means your email and your password are sent over the network in cleartext. If you're connected to a wifi network that has no encryption, or uses weak encryption, than anyone with a sniffer can see not only your username and password, but the contents of your email as well.
This is bad.
However, if you were setup to tunnel all of your network traffic over SSH, then absolutely NONE of what was caught off the wire would be visible to the sniffers. You could be on an entirely open network, with no security, and no authentication, and no encryption, and the contents of your data would be perfectly safe.
The second use, and the one people don't like to talk about, is that SSH makes it trivial to circumvent firewall restrictions. For example, let's say your company policy is to block access to external email servers. Sure, you could use webmail, but you'd really like to use your favorite email client.
If the company allows a single port open to the outside world, you can tunnel an SSH connection over that and still use your favorite email client.
I'll get into more details of things like this when I get into the practical applications and examples of each instance.
Forwarding a local port to a remote port
First, the commandline
ssh -f dayne@rhaegar -L 2010:localhost:25 -N
- ssh: invokes the cli ssh client
- -f : sends the process to the background. You want to do this, otherwise you need to keep that command window open. If you send it to the background, you can close the window, or whatever, the tunnel will stay active until you either kill the process on purpose, lose network connectivity, or reboot.
- dayne@rhaegar : my username and the host I'm establishing the tunnel with
- -N : Tells SSH not to run a command. This is needed since we're using -f. It won't fork without a command, or unless explicitly told not to run a command.
- -L 2010:localhost:25 : This is where the magic occurs. This says forward my local port of 2010 to your port 25.
So as long as this tunnel is up (ie, as long as the ssh connection stays alive), any time I initiate communication to localhost port 2010, it's the same as if I was connecting to rhaegar on port 25 instead. Did it work? Let's test it!
tyrion:~ Dayne$ telnet localhost 2010
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 rhaegar.rhaegar ESMTP Postfix (Debian/GNU)
Ok, now let's run the same test trying directly to port 25, and let's see if we get the same banner.
tyrion:~ Dayne$ telnet rhaegar 25
Trying 173.230.128.222...
Connected to rhaegar.
Escape character is '^]'.
220 rhaegar.rhaegar ESMTP Postfix (Debian/GNU)
Indeed we do.
Common Use cases:
While this may seem like a stupid computer trick, there is a subtle difference. Connecting to port 25 on the remote host and port 2010 on the localhost yield the same results, superficially. However, if I connect to the remote host, it's a direct connection, and port 25 is not encrypted. If I connect to port 2010 on local, that traffic is forwarded over SSH, and is encrypted until it reaches it's destination. This allows me a degree of privacy and security in my communications. After all, if I'm transmitting the go order to blow up Farawayistan back to my henchmen on Skullcrusher Mountain while I'm sipping a venti latte in Starbucks, I don't want to leave any evidence for the local wifi sniffing script kiddies to turn over to the FBI.
In order to actually make my local email client make use of the tunnel, all I would have to do is go into the settings, modify the SMTP server name to 127.0.0.1, and change the port to 2010. After that, my mail client will send via the tunnel. This example is not limited to only SMTP, obviously. You could just as easily tunnel POP3, IMAP, IMAPS, IRC, etc. If it can connect over TCP, and if you can configure the host and port, you can tunnel it over ssh.
Forwarding a local port to a remote port through an intermediary:
The preceding example only works because the destination server is running ssh. If it was not running ssh, or if the ssh port wasn't open to the world, then the command would have failed.
Let's say I want to check my pop3 mail on mail.earthlink.net, but I want to do it over an ssh tunnel. If I try to use the above example, it will fail.
tyrion:~ Dayne$ ssh -f dayne@mail.earthlink.net -L 2011:localhost:110 -N
ssh: connect to host mail.earthlink.net port 22: Connection refused
Or, it may just hang, indicating that it's being blocked by a firewall. Either way, I cannot directly forward the pop3 traffic over the tunnel. Bummer.
However, there's another way. If I have an ssh connection to anywhere, I can use that as an intermediary. Essentially, what I will do is create a connection that tells the server I'm sshing into that I want it to create a tunnel linking a port on my machine, to a port on a remote machine.
First, what happens if we connect directly to mail.earthlink.net?
tyrion:~ Dayne$ telnet mail.earthlink.net 110
Trying 209.86.93.208...
Connected to mail.earthlink.net.
Escape character is '^]'.
+OK NGPopper vEL_0_1_36_P at earthlink.net ready <21976.1308009286@mp-rally.atl.sa.earthlink.net>
We will use this text here to verify that our tunnel is connecting to the correct endpoint later. (And NGPopper guys? really?)
So here's the commandline:
ssh -f dayne@rhaegar -L 2012:mail.earthlink.net:110 -N
Almost the same commandline as before, with one difference
- -L 2012:mail.earthlink.net:110
Instead of saying localhost between the two ports as up above, I now have the target name of the server I want to forward to. What the commandline basically says is that I want rhaegar to create a tunnel on my local port of 2012, and any traffic that comes over that tunnel, forward it to mail.earthlink.net on port 110. In this way, rhaegar is proxying my connection for me. From mail.earthlink.net's perspective, the connection will be coming from rhaegar, not my local machine. It's important to note that the encryption only applies between my local computer, and the intermediary. So from localhost to rhaegar, the traffic is encrypted. From rhaegar to mail.earthlink.net, the traffic is not encrypted (not by ssh anyway, if it were an SSL connection, then yes, the SSL encryption would still apply).
Does it work?
tyrion:~ Dayne$ telnet localhost 2012
Trying ::1...
Connected to localhost.
Escape character is '^]'.
+OK NGPopper vEL_0_1_36_P at earthlink.net ready <706.1308009717@mp-sieze.atl.sa.earthlink.net>
It does indeed!
Common use cases:
Again, using the wifi hotspot example - If I have an endpoint that I trust, and have SSH access to, I can now forward my traffic encrypted to that host, and have it act as a proxy for all of my connections. I can once again blow up Farawayistan at my whim, and avoid the local script kiddie narcs. The intermediary server can be anything you have ssh access to. A server you pay for with a company, a server running at home, a server you've 'liberated' from it's proper owners, whatever. If you can ssh to it, you can abuse it.
This use case should be obvious. If I don't have direct ssh access to my endpoint, I can use this technique to get around that. This is also useful in cases where outbound ssh access is limited. For example, some companies only allow external outbound ssh access through a particular server, or group of servers, usually referred to as jump servers. So even if I do have direct ssh access to my endpoint, but I'm prevented by firewall policy from accessing it, I can use the jump server as an intermediary. For the more evil minded among you, this means you can basically tunnel anything you like, as long as you can reach one server which has ssh access. Company just put in a sexy new IronPort, and you're unhappy about it intercepting your SSL traffic? Setup squid on a remote server, then use an ssh tunnel to forward a local port to your new squid server. Alter your browser's proxy settings to point to localhost and the port of the tunnel, and boom, you're now surfing the web free of Big Brother's watchful eye.
Reverse SSH tunneling
Hopefully by now, the concepts of forcing all traffic down a tunnel is starting to make sense. If you're intelligent, and mischievous, and you didn't already know everything I've explained, you are now positively cackling with glee.
Reverse SSH tunneling is the same thing as forward tunneling, except in reverse.
Yeah, I know that seems a little obvious, but it may not explain the situation very well.
In forward tunneling, our goal is to take a local port on our machine, and connect it to a remote port on the other machine. This creates a port listening on our end, connected to a port on the other end.
In reverse tunneling, the goal is to take a remote port on a remote machine, and connect it to a port on the local machine.
The commandline:
ssh -f dayne@rhaegar -R 2020:localhost:22 -N
All the other options are the same as before, the -R command is the only difference, it's used instead of -L (-L = local, -R = remote)
What this command says is that I want the remote machine to create a tunnel that connects port 2020 on the REMOTE machine to port 22 on the LOCAL machine. So now, rhaegar could ssh to port 2020 on localhost, and it would connect back to my laptop on port 22. Does it work?
dayne@rhaegar:~$ ssh -p 2020 localhost
Password:
Last login: Mon Jun 13 20:23:25 2011 from localhost
tyrion:~ Dayne$ hostname
tyrion.local
tyrion:~ Dayne$
Indeed it does.
I can also create a reverse tunnel using my local machine as an intermediary for a tunnel between two remote points.
ssh -f dayne@rhaegar -R 2021:mail.earthlink.net:110 -N
This command tells rhaegar that I want it to create a tunnel between it's port of 2021, go through my machine, with an endpoint of mail.earthlink.net on port 110. All traffic would appear to be coming through my computer, not rhaegar.
Again, does it work?
dayne@rhaegar:~$ telnet localhost 2021
Trying 127.0.0.1...
Connected to rhaegar.
Escape character is '^]'.
+OK NGPopper vEL_0_1_36_P at earthlink.net ready <6253.1308011937@mp-sieze.atl.sa.earthlink.net>
It surely does.
Common use cases:
This is commonly used for cases where you want to access an internal network from outside, but are prevented due to other access restrictions. For example, I used to work at a job where all ssh access to computers in the data center was restricted to the NOC's internal network. This is right and proper.
In the datacenter, we had a workstation that was setup with access to our monitoring web pages. So we could go over to the datacenter, and still sees the details of whatever ticket we needed to work on, update the ticket, and communicate with the guys still in the NOC. However, sometimes the problem isn't with the system that's alerting, it's with a system that it's interdependent with. Sure, I could go over and log into the box locally with the root password. But what if it was a problem I needed to research? I'd need to walk back and forth between the monitoring station and the local console. Or I'd need to get one of the guys in the NOC to do it. Or I'd need to go back to my desk in the NOC, which would be nice, unless something else was wrong that required my presence in the data center.
The simplest solution was to be able to work from the data center terminal. But company security policy prevented that kind of access from that terminal.
So what I did was establish a reverse tunnel from my machine in the NOC to the terminal in the data center. Then I could walk over to the remote terminal, log into it, ssh into that machines port I setup for the tunnel, and I was logged into my machine back in the NOC. From there, I had free reign to do whatever I wanted. I potentially created a huge security hole, however, and if management had ever found out about it, it would not have been a pleasant discussion.
Using forward and reverse tunnels in conjunction:
You can use both of these techniques in tandem.
Let's introduce a third machine, daenerys.
dayne@daenerys:~$ ssh -f dayne@rhaegar -R 2020:mail.earthlink.net:110 -N
daenerys has now established a reverse tunnel to rhaegar. rhaegar can now access mail.earthlink.net on port 110 via it's local port of 2020
Now, on my laptop, I establish a forward tunnel from my local port of 2100 to rhaegar's remote port of 2020
tyrion:~ Dayne$ ssh -f dayne@rhaegar -L 2100:localhost:2020 -N
What happens when I telnet to localhost on port 2100 on my laptop?
tyrion:~ Dayne$ telnet localhost 2100
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK NGPopper vEL_0_1_36_P at earthlink.net ready <10480.1308013263@mp-wonder.atl.sa.earthlink.net>
^]
Here's the flow traffic -
My laptop talks to localhost on port 2100, which via the forward tunnel goes to port 2020 on rhaegar. Port 2020 on rhaegar is the reverse tunnel from daenerys. So my traffic goes from the forwarded tunnel, to the reverse tunnel, and ends up coming out of daenerys connected to mail.earthlink.net on port 110.
You can chain this through as many boxes as you'd like. If you wanted, you could create a shadow network of endpoints, all connected by tunnels, all encrypting your traffic.
Defending against unauthorized use of SSH Tunneling:
If you're responsible for network security in your organization, I may have just scared the bejesus out of you. My company just spent a lot of time, and alot of money implementing a data loss prevention plan, and they were very unhappy when I showed them how easy it was for me to circumvent it. Were I a less honest employee, I could have smuggled whatever information out of the company that I wanted, and with the amount of SSH traffic going across the network as part of regular business, a paper trail would have been very difficult to trace back to me, and since all the traffic is encrypted, they would never have been able to prove beyond the shadow of a doubt that I was the one who smuggled it out.
The problem comes from the fact that SSH port forwarding is turned on by default by virtually every modern unix-like distribution. Astute system administrators have turned it off. The problem is, you have to go further than that to secure against a smart and determined user.
I've run into a lot of admins for whom it occurred to turn it off. By they didn't think it all the way through.
dayne@rhaegar:~$ ls -alh /usr/sbin/sshd
-rwxr-xr-x 1 root root 436K Dec 26 13:12 /usr/sbin/sshd
What's wrong with that picture?
The default sshd permissions make it world executable by default. And sshd has a commandline parameter that allows you to specify it's configuration file. It is trivial for me to upload my own sshd_config file with portforwarding turned back on, bind it to a high port, and now all of a sudden, the box has ssh available on another port for me to tunnel through. Unless you've been a total **** with your firewall policies, the security risk is right back where it was.
Ok, so you disable portforwarding, and you removed global execute permission from sshd. I've only encountered an admin that smart once. Are you protected? Maybe. The aforementioned admin was one smart cookie, except he forgot one thing - he didn't remove the compilers and development libraries. So I was able to upload the openssh source, and compile my own version, and then execute it on a high port.
Not many users are willing to go to those lengths to get around your Facebook blocks. But what happens if a user gets their box compromised by someone who is smart enough and determined enough? If you want to secure your infrastructure, you have to think of as many avenues of circumvention as you can, not merely the ones that will help you pass an audit.
Epilogue
My goal here is not to educate people on how to misbehave at work. SSH is a wonderful tool, but like all tools, how you use it makes a difference. I believe in the right to security and privacy, especially in a political environment that seems determined to take it away. I don't care if it makes it harder for the cops to do their job. I have nothing to hide, but my business is my business, and even the most innocuous bit of information can be used to convict someone if it's spun right. I choose to take no chances and jealously guard my privacy, and I recommend others do the same. Politicians can use the 'if we don't do this, the terrorists win!' line as much as they want. It's my opinion that if we allow our personal liberties to be compromised as a reaction against something that might happen, then they've already won.
My personal preaching aside, when it comes to matters of security, I believe in full disclosure. I believe that people have a right to know the threats they're facing, and while demonstrating these techniques may empower some of the bad guys, I believe that most people are good at heart, and it will be used to help guard companies against potential internal threats.
And it's entirely likely that in the vastness of the internet, this will have no noticeable effect at all. Only time will tell.
