Now that I’ve got Starlink up and running in a bonded connection with my DSL while still hosting my own webserver, I thought I’d share some details about how. I’ve learned that I’m old and when this all stops working in a year I’m never going to remember what I did or why, so I needed some rough documentation just for me. Additionally, I had a heck of a time figuring out how to do some of this stuff. In particular, the less common uses nftables seem to be poorly documented.

To start with, Starlink promised that during this beta program there would be downtime. I did some research before I got it, and as best as I could tell from what internet sleuths had put together, my little hexagon on the planet would have only 78% of a 24 hour period with have satellite coverage within 25 degrees of the horizon. It turns out that this is either outdated or wrong: I’m only seeing minutes per day without satellites and only a few minutes of what Starlink calls “beta outage”: ie they’re messing with stuff and they’re not sorry they turned your internet off while they were doing that. It’s only been a few days, but I think my DSL disconnects about as often as Starlink does.

In any case, I was convinced I needed the ability to bond my Starlink and DSL connection to have a continuous internet connection; plus an ability to use both connections for maximum bandwidth was also appealing. The only realistic way to do this in today’s world is with a VPN where local software sends any given TCP packet down one pipe or the other to a VPN server where they get mangled and forwarded on to their intended recipient as a coherent stream. The software is clever and when one side goes down, the dropped packets will appear down the other pipe. TCP is resilient and well-suited to this task.

You can probably roll your own with a cloud server, but the simple answer is just to get a Speedify account. They provide Linux (and other OS) software that will bond any sort of connection with any other at the other end of a VPN for $3/month. Sold.

While it runs on Linux, it appears that making it working on DD-WRT is something of a nightmare. And, while I was happy to ditch my AMD K6-400 OpenBSD router a couple of years ago for a spiffy Netgear R6700 running DD-WRT, it turns out that I hate DD-WRT. If it does what you want it to out of the box it’s great, but dealing with embedded Linux is a bit of a nightmare. When Andrew convinced me to stick this blog behind Cloudflare, the DD-WRT dynamic DNS client was ancient, didn’t support cloudflare, and replacing it with a different one involves a miserable build process.

So, I decided to upgrade my trusty 15-year-old Core2Duo webserver with something modern and press it into double duty as router and webserver. I’m pretty that I could have kept using it, but it was a good excuse to upgrade. I built an overkill 6-core AMD Ryzen 5 3600 system (because the cheap Ryzen 3 chips are completely unavailable). I loaded Debian on it, because while OpenBSD was a fun experiment back when I used to like computers, I’m just not that much of a masochist anymore. I bought a $30 4-port NIC to go in it, and now it’s a “router” as well as webserver. With some minor pain, I managed to get it working with DSL with a fairly bog-standard nftables setup.

[Rant: Linux changes the way to do IP masquerading constantly. It remains unclear that anything gets better, but it does change. First, there was ipfwadm. This worked great to share my dial-up SLIP connection with the rest of the house circa 1996. Then there was ipchains. And then when they switched to iptables 20 years ago, I refused to learn it because there was just going to be another new thing. Instead, I switched to OpenBSD and it turns out that pf is fantastic. I finally decided to give up and learn iptables for this project, only to discover that it’s been replaced by nftables.]

Starlink arrived days later than promised after FedEx Ground continues to position themselves as the delivery service with the slogan, “at least we’re more expensive than the post office!” After making Jonah climb on the roof to install it in the dark (I don’t do roofs), I plugged it up to provided wireless router and had it up and running in minutes on its own wifi.

Step 1: Make Starlink work in Linux. This was easy. Unplug the provided router and connect to the Linux router instead. Add the following lines to /etc/network/interfaces:

auto enp39s0
iface enp39s0 inet dhcp

Have I mentioned that the new kernel naming for ethernet devices is terrible? It’s well-intentioned, but I’d rather they just stuck with eth0, eth1 etc and let me write my own persistent udev rules. enp39s0 just rolls off the tongue and is so easy to type, what’s not to like?

Anyway, that’s all that’s needed to get Starlink up and happy on the router. But now I needed to get Speedify working to bond the two.

Speedify’s documentation for Linux is terrible. They have a rough “here’s how to do this on a Raspberry Pi” series of how-to’s, and some screenshots for a UI that wouldn’t run on my Debian installation because it depends on some package that’s long-removed from testing. Whatever, I didn’t want to install an X server ayway.

After getting it installed and logging in with:

/usr/share/speedify/speedify_cli login username password

and then

/usr/share/speedify/speedify_cli connect

I was able to access the internet through the speedify VPN, but only from either Starlink or DSL. I needed to remove the “replacedefaultroute” from my /etc/ppp/peers/dsl-provider file to prevent the ppp connection from blasting over the default route that the Starlink dhcp server provided. Once I did that, ran /usr/share/speedify/speedify_cli startupconnect on , and ensured that the Starlink block appeared *before* the DSL block in my /etc/network/interfaces file, things seemed to work nicely on startup.

It turns out that there’s a completely undocumented file, /etc/speedify/speedify.conf. Here’s mine:

# Set to 1 to enable sharing Speedify to other devices
ENABLE_SHARE=1

# The interface(s) to use for sharing
# ex: SHARE_INTERFACE="eth1"
# When you enable sharing, Speedify will automaticlly set this interface to the NEVER priority so that it is not used as an Internet connection. 
# If you disable sharing for the interface and want to use it again as an Internet connection, you can set it back to the Always priority by doing:
# /usr/share/speedify/speedify_cli adapter priority {interface} always
SHARE_INTERFACE="enp34s0"

# IP to use for the sharing interface
# ex: INTERFACE_IP="192.168.145.1"
INTERFACE_IP="192.168.0.15"

# DNS servers to send over DHCP to clients
# ex: DNS_SERVERS="10.202.0.1,1.1.1.1"
DNS_SERVERS="10.202.0.1,1.1.1.1"

# Set to 1 to allow internet access on other devices when Speedify is disconnected
SHARE_WITHOUT_SPEEDIFY=0

ENABLE_SHARE=1 means that speedify will set up its own nftable rules to enable NAT (masquerade for us ancient folk). It will also ham-fistedly install/enable dnsmasq, which if you’re running isc-dhcp-server will conflict. I removed dnsmasq. The DNS_SERVERS line up there is, I think, used to configure dnsmasq, so I don’t care about that because I’m running my own DNS server as well.

So, this is all you need for a simple shared speedify bonded Starlink/DSL setup. But I run a webserver, IMAP server, want to be able to ssh into my home network from outside, etc. And that’s where it gets nasty.

I spent an embarrassingly long time trying to figure out why ddclient wouldn’t update Cloudflare’s DNS records with my Starlink IP. Because I’m old and don’t keep up with the proceedings of the IETF, I missed this: 100.64.74.22 is *not* a public IP address. Given that it starts with 100 and not 192 or 10, I didn’t register it as a private address. But it really, really is.

So, yeah, this is probably the crappiest thing about Starlink for me so far. We ran out of ipv4 addresses a long time ago, so I shouldn’t be at all surprised that Starlink couldn’t get any. This sucks and means that my webserver is not going to be accessible via ipv4 and Starlink. It remains to be seen if it’s accessible via ipv6, but ipv6 doesn’t work for me yet, and that’s a different headache entirely.

Speedify will gladly support inbound traffic through your VPN if you buy a dedicated server from them for $120/month. Yeah, not so much for me thanks.

That leaves my webserver stuck with DSL. But when connected to speedify with the default nftables, if you hit my webserver it’ll get your request and respond… via the bonded speedify connection. Which means you’re never going to get an established connection.

So what I needed was a way for *most* traffic leaving the webserver/router to do so on the speedify connection, but traffic that is webserver responses needs to go out on the DSL connection only. Yikes.

I spent a long time figuring this out. It’s possible if I’d learned iptables, that the solution would be more obvious, because the nftables documentation does not mention this at all. At first, it looks like you can’t really do routing with nftables at all. And you can’t, really. But you can mark packets and then let iproute direct marked packets to a different routing table. This is poorly documented, but thanks to this random forum post, I was able to figure it out. Here’s what I did.

My /etc/nftables.conf:

#!/usr/sbin/nft -f

flush ruleset

define lan = enp34s0

table inet filter {
        chain input {
                type filter hook input priority 0
        }
        chain forward {
                type filter hook forward priority 0
                oifname "ppp0" tcp flags syn tcp option maxseg size set rt mtu
        }
        chain output {
                type filter hook output priority 0
        }
}

table ip nat {
        chain postrouting {
                type nat hook postrouting priority 0; policy accept
                oifname "enp39s0" masquerade
        }
}
table ip mangle {
         chain output {
                type route hook output priority mangle; policy accept
                tcp sport 80 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 22 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 443 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 993 ip daddr != 192.168.0.0/24 counter mark set 42
        }
}

The ip mangle table at the bottom is where the magic is. The webserver rule says, basically, that all packets being output from this server with a source port of 80 and a source address that is NOT my local network should be marked with “42”. Then, I set up routing for packets marked “42”. I needed to avoid packets on the local network or I couldn’t access my own webserver from the local network, because the responses would be directed out the DSL connection!

First, I had to create the routing table in /etc/iproute2/rt_tables by adding this line:

201 dsl.out

Here’s the command to create the separate dsl routing table. I put this as a post-up line in my network/interfaces for after the ethernet connection associated with my DSL is brought up:

ip rule add fwmark 42 table dsl.out

The actual contents of this routing table is defined by the following line which I put in a script in /etc/ppp/ip-up.d/0addroute:

ip route add default via 0.0.0.0 dev ppp0 table dsl.out

The next missing piece is Starlink statistics. Dishy gives a pretty slick statistics page on the phone app, which is also available from a browser if you hit it at the address 192.168.100.1. This is sort of a hidden network on the Starlink that needs a special rule if you’re not going to use their router. The following line in my /etc/network/interfaces file after the Starlink block makes it accessible:

post-up ip route add 192.168.100.0/24 via 0.0.0.0 dev enp39s0

This makes it accessible from the router, and as far as I’m concerned *should* make it accessible from the rest of the network as well. But it didn’t, and after an hour of being unable to figure why not, I just added a second NAT target that you can see in my nftables.conf above to the enp39s0 interface. This is a lame solution, but seems to work, and I don’t want to spend the rest of my life figuring it out.

So, I think I’m in the place where it works, a router reboot brings up both connections and all the nftables and routing automatically.

A side note: I’ve got Dishy plugged up to the UPS with my router and network gear. This means that when the power goes up I’ll still have Starlink! This is great because our power goes out all the time, and when it does, it takes the DSL with it because apparently CenturyLink can’t be bothered with battery backups.

The only thing that doesn’t work right now is that /etc/resolv.conf is currently getting stomped on by something; I suspect speedify. I’ll figure it out eventually, but I’m pretty happy with the set up so far.

I suspect long term the correct thing to do is just move this webserver in the cloud so it’s actually responsive. I’m pretty disappointed that Starlink didn’t make my webserver any faster for my tens of readers, but maybe there are other options (such as ipv6) that will improve things going forward.