Updates 2024

I have started using Cloudflare Tunnel lately. It works great for my own use but still figuring out how to make it available for our team without them need to have a Cloudflare account.

Another great thing about CF Tunnel is you can add authentication page to your tunneled site, making it more private than just some random url.

Updates 2022

We're currently using sirtunnel.py, a Python script to accomplish this. Sirtunnel make it much easier to implement this by helping to invoke Caddy API to create new vhost that will listen to incoming http request and forward it to ssh tunnel that we establish prior to running sirtunnel.py.

Example usage:-

ssh -t kamal@yourhost.ee -R 9000:localhost:9000 sirtunnel.py kamal-site 9000

This will open remote port 9000 on yourhost.ee and forward it to local port 9000. Then we setup new site in caddy at subdomain kamal-site.yourhost.ee and proxying request to port 9000 on the server.

The new site now available as https://kamal-site.yourhost.ee/.

Notes

When first setting up sirtunnel, need to tell caddy the initial server config:-

curl -X POST "http://localhost:2019/load" -H "Content-Type: application/json" -d @caddy_config.json

and caddy_config.json looks like this:-

{
  "apps": {
    "http": {
      "servers": {
        "sirtunnel": {
          "listen": [":443"],
          "routes": [
          ]
        }
      }
    }
  }
}

******** End updates *****

We have a private development server where we do most our work. But like all modern web application these days, it need to talk (or being talked to) with other applications as well. One is through webhook, where your app receive kind of notifications from other application through http request.

To receive a webhook, your application need to be accessible from the remote application. As mentioned above, our dev server is on private network. The only we can access it from our local computer is by doing ssh tunnel. For example, I'll start my day with:-

ssh dev-server -L 8000:localhost:8000

and then when I run such as python manage.py runserver on the remote dev server, I can access that django application from my laptop as http://localhost:8000/. But of course we can't receive a webhook such as from github this way.

One common solution to this problem is by using service such as ngrok, where I can run command on the remote dev server:-

ngrok http 8000

ngrok will give us random subdomain such as https://xxyymmdd.ngrok.com/, and when accessed over Internet, will forward back the request to our django development server where the ngrok command above was running. It's perfect and I use it most of the time for my personal projects.

But for work, I don't feel easy using third party services that we can't control much. One solution is to setup a proxy on a small vps that fully exposed to Internet. We can use Caddy or nginx for this. But this is a permanent setup. I want something ad-hoc similar to ngrok above, where the tunnel will just die after I closed connection to the remote dev server.

Second option will be ssh reverse tunnel with GatewayPorts enabled. So I can run command like this on the dev-server:-

ssh -R :8000:localhost:8000 proxy-server-ip

That allow connection to port 8000 on proxy-server being forwarded to the host where the command above was running. But one problem with this is you can't get TLS connection to port 8000 of the proxy-server.

So to level this up a bit, we need help from a second tools - a tcp proxy. There's a lot actually if you search this up, so I just settled down with one that seem to provide what I need. I picked tcpproxy, a little tool written in Golang. It has all that I need. So my command, when I want to open up a proxy is this (run on the remote dev-server):-

ssh -t -R 9000:localhost:8000 scarif.planet.rocks -l username "tcpproxy -laddr 0.0.0.0:8000 -raddr localhost:9000 -lcert /etc/letsencrypt/live/scarif.planet.rocks/fullchain.pem -lkey /etc/letsencrypt/live/scarif.planet.rocks/privkey.pem -ltls"
Proxying from 0.0.0.0:8000 to 127.0.0.1:9000

This then provide public access to https://scarif.planet.rocks:8000/ which then get proxied to our django development server running on port 8000. Ok, it look pretty confusing at first. But it require me to only understand this command instead lengthy docs to know what it's doing. Let's look this step by step:-

1. Open a reverse tunnel from port 9000 on proxy server to port 8000 at local machine (local here mean where the command executed).
2. Run tcpproxy on proxy server, listen on port 8000 at all network interfaces. Here we also enable tls and use cert provided by letsencrypt.
3. Forward connection to that port 8000 to port 9000 on local interface.
4. Forward connection on port 9000 back to our local machine through the ssh reverse tunnel.

And here not so nice diagram courtesy of asciiflow:-

                                                         XXXXXXX
                                                     XX        XX
+--------+              +------------+               X
|        |              |            |               XX            X
|        <--------------+            | <-------------+X  Internet    X
|        |              |            |                X              XX
+--------+              +------------+                 XX             X
        8000            9000        8000                XX          XXX
 local                    proxy-server                     X     XXX
                                                            X XX

One thing still missing with this approach is the nice request inspector that ngrok has.

Alternative with Caddy

First we generate required Caddyfile on the remote host:-

echo "https://scarif.planet.rocks {\n proxy / http://localhost:9000 {\n header_upstream Host dev.app.int\n }\n}" | ssh scarif.planet.rocks -l ubuntu -R 9000:localhost:80 "cat > Caddyfile"

The command above will immediately ended and we're back at the local host, but it will write the file to remote host. The file will be written as:-

https://scarif.planet.rocks {
 proxy / http://localhost:9000 {
 header_upstream Host dev.app.int
 }
}

Now we can ssh, reverse forward port on remote and run caddy:-

ssh -t scarif.planet.rocks -l ubuntu -R 9000:localhost:80 "caddy"

This ended up better in few ways:-

• We can access the site as https://scarif.planet.rocks/. Notice that caddy bind to port 80, because we use setcap to allow caddy bind to lower port without being root.
• No changes needed on the local apache or django, like adding to ALLOWED_HOSTS or set ServerAlias. This because we set the Host header when caddy proxying the request.

The downside is that we have to do it in 2 steps. I can't get writing the config file and then executing caddy in single ssh command. Got this error:-

Pseudo-terminal will not be allocated because stdin is not a terminal.

Issue

1. Terminate remote tcpproxy process if the ssh connection die. Using ssh -t seem to work if I just Ctrl+C the connection, but if it died, for example due to network issue, tcpproxy on the proxy server will left running and you have to ssh into it and manually kill the process. This question on stackexchange suggest few workaround but I haven't try yet.

FAQ

Q: Do you know ssh GatewayPorts ?
A: Yes, but as mentioned above, it doesn't give us TLS connection from the public.

Q: Have you try pagekite ?
A: Yes, but I have a hard time trying to understand how it work if I want to host my own server.