Posts by Berck

Now that I’ve got Starlink up and running in a bonded connection with my DSL while still hosting my own webserver, I thought I’d share some details about how. I’ve learned that I’m old and when this all stops working in a year I’m never going to remember what I did or why, so I needed some rough documentation just for me. Additionally, I had a heck of a time figuring out how to do some of this stuff. In particular, the less common uses nftables seem to be poorly documented.

To start with, Starlink promised that during this beta program there would be downtime. I did some research before I got it, and as best as I could tell from what internet sleuths had put together, my little hexagon on the planet would have only 78% of a 24 hour period with have satellite coverage within 25 degrees of the horizon. It turns out that this is either outdated or wrong: I’m only seeing minutes per day without satellites and only a few minutes of what Starlink calls “beta outage”: ie they’re messing with stuff and they’re not sorry they turned your internet off while they were doing that. It’s only been a few days, but I think my DSL disconnects about as often as Starlink does.

In any case, I was convinced I needed the ability to bond my Starlink and DSL connection to have a continuous internet connection; plus an ability to use both connections for maximum bandwidth was also appealing. The only realistic way to do this in today’s world is with a VPN where local software sends any given TCP packet down one pipe or the other to a VPN server where they get mangled and forwarded on to their intended recipient as a coherent stream. The software is clever and when one side goes down, the dropped packets will appear down the other pipe. TCP is resilient and well-suited to this task.

You can probably roll your own with a cloud server, but the simple answer is just to get a Speedify account. They provide Linux (and other OS) software that will bond any sort of connection with any other at the other end of a VPN for $3/month. Sold.

While it runs on Linux, it appears that making it working on DD-WRT is something of a nightmare. And, while I was happy to ditch my AMD K6-400 OpenBSD router a couple of years ago for a spiffy Netgear R6700 running DD-WRT, it turns out that I hate DD-WRT. If it does what you want it to out of the box it’s great, but dealing with embedded Linux is a bit of a nightmare. When Andrew convinced me to stick this blog behind Cloudflare, the DD-WRT dynamic DNS client was ancient, didn’t support cloudflare, and replacing it with a different one involves a miserable build process.

So, I decided to upgrade my trusty 15-year-old Core2Duo webserver with something modern and press it into double duty as router and webserver. I’m pretty that I could have kept using it, but it was a good excuse to upgrade. I built an overkill 6-core AMD Ryzen 5 3600 system (because the cheap Ryzen 3 chips are completely unavailable). I loaded Debian on it, because while OpenBSD was a fun experiment back when I used to like computers, I’m just not that much of a masochist anymore. I bought a $30 4-port NIC to go in it, and now it’s a “router” as well as webserver. With some minor pain, I managed to get it working with DSL with a fairly bog-standard nftables setup.

[Rant: Linux changes the way to do IP masquerading constantly. It remains unclear that anything gets better, but it does change. First, there was ipfwadm. This worked great to share my dial-up SLIP connection with the rest of the house circa 1996. Then there was ipchains. And then when they switched to iptables 20 years ago, I refused to learn it because there was just going to be another new thing. Instead, I switched to OpenBSD and it turns out that pf is fantastic. I finally decided to give up and learn iptables for this project, only to discover that it’s been replaced by nftables.]

Starlink arrived days later than promised after FedEx Ground continues to position themselves as the delivery service with the slogan, “at least we’re more expensive than the post office!” After making Jonah climb on the roof to install it in the dark (I don’t do roofs), I plugged it up to provided wireless router and had it up and running in minutes on its own wifi.

Step 1: Make Starlink work in Linux. This was easy. Unplug the provided router and connect to the Linux router instead. Add the following lines to /etc/network/interfaces:

auto enp39s0
iface enp39s0 inet dhcp

Have I mentioned that the new kernel naming for ethernet devices is terrible? It’s well-intentioned, but I’d rather they just stuck with eth0, eth1 etc and let me write my own persistent udev rules. enp39s0 just rolls off the tongue and is so easy to type, what’s not to like?

Anyway, that’s all that’s needed to get Starlink up and happy on the router. But now I needed to get Speedify working to bond the two.

Speedify’s documentation for Linux is terrible. They have a rough “here’s how to do this on a Raspberry Pi” series of how-to’s, and some screenshots for a UI that wouldn’t run on my Debian installation because it depends on some package that’s long-removed from testing. Whatever, I didn’t want to install an X server ayway.

After getting it installed and logging in with:

/usr/share/speedify/speedify_cli login username password

and then

/usr/share/speedify/speedify_cli connect

I was able to access the internet through the speedify VPN, but only from either Starlink or DSL. I needed to remove the “replacedefaultroute” from my /etc/ppp/peers/dsl-provider file to prevent the ppp connection from blasting over the default route that the Starlink dhcp server provided. Once I did that, ran /usr/share/speedify/speedify_cli startupconnect on , and ensured that the Starlink block appeared *before* the DSL block in my /etc/network/interfaces file, things seemed to work nicely on startup.

It turns out that there’s a completely undocumented file, /etc/speedify/speedify.conf. Here’s mine:

# Set to 1 to enable sharing Speedify to other devices
ENABLE_SHARE=1

# The interface(s) to use for sharing
# ex: SHARE_INTERFACE="eth1"
# When you enable sharing, Speedify will automaticlly set this interface to the NEVER priority so that it is not used as an Internet connection. 
# If you disable sharing for the interface and want to use it again as an Internet connection, you can set it back to the Always priority by doing:
# /usr/share/speedify/speedify_cli adapter priority {interface} always
SHARE_INTERFACE="enp34s0"

# IP to use for the sharing interface
# ex: INTERFACE_IP="192.168.145.1"
INTERFACE_IP="192.168.0.15"

# DNS servers to send over DHCP to clients
# ex: DNS_SERVERS="10.202.0.1,1.1.1.1"
DNS_SERVERS="10.202.0.1,1.1.1.1"

# Set to 1 to allow internet access on other devices when Speedify is disconnected
SHARE_WITHOUT_SPEEDIFY=0

ENABLE_SHARE=1 means that speedify will set up its own nftable rules to enable NAT (masquerade for us ancient folk). It will also ham-fistedly install/enable dnsmasq, which if you’re running isc-dhcp-server will conflict. I removed dnsmasq. The DNS_SERVERS line up there is, I think, used to configure dnsmasq, so I don’t care about that because I’m running my own DNS server as well.

So, this is all you need for a simple shared speedify bonded Starlink/DSL setup. But I run a webserver, IMAP server, want to be able to ssh into my home network from outside, etc. And that’s where it gets nasty.

I spent an embarrassingly long time trying to figure out why ddclient wouldn’t update Cloudflare’s DNS records with my Starlink IP. Because I’m old and don’t keep up with the proceedings of the IETF, I missed this: 100.64.74.22 is *not* a public IP address. Given that it starts with 100 and not 192 or 10, I didn’t register it as a private address. But it really, really is.

So, yeah, this is probably the crappiest thing about Starlink for me so far. We ran out of ipv4 addresses a long time ago, so I shouldn’t be at all surprised that Starlink couldn’t get any. This sucks and means that my webserver is not going to be accessible via ipv4 and Starlink. It remains to be seen if it’s accessible via ipv6, but ipv6 doesn’t work for me yet, and that’s a different headache entirely.

Speedify will gladly support inbound traffic through your VPN if you buy a dedicated server from them for $120/month. Yeah, not so much for me thanks.

That leaves my webserver stuck with DSL. But when connected to speedify with the default nftables, if you hit my webserver it’ll get your request and respond… via the bonded speedify connection. Which means you’re never going to get an established connection.

So what I needed was a way for *most* traffic leaving the webserver/router to do so on the speedify connection, but traffic that is webserver responses needs to go out on the DSL connection only. Yikes.

I spent a long time figuring this out. It’s possible if I’d learned iptables, that the solution would be more obvious, because the nftables documentation does not mention this at all. At first, it looks like you can’t really do routing with nftables at all. And you can’t, really. But you can mark packets and then let iproute direct marked packets to a different routing table. This is poorly documented, but thanks to this random forum post, I was able to figure it out. Here’s what I did.

My /etc/nftables.conf:

#!/usr/sbin/nft -f

flush ruleset

define lan = enp34s0

table inet filter {
        chain input {
                type filter hook input priority 0
        }
        chain forward {
                type filter hook forward priority 0
                oifname "ppp0" tcp flags syn tcp option maxseg size set rt mtu
        }
        chain output {
                type filter hook output priority 0
        }
}

table ip nat {
        chain postrouting {
                type nat hook postrouting priority 0; policy accept
                oifname "enp39s0" masquerade
        }
}
table ip mangle {
         chain output {
                type route hook output priority mangle; policy accept
                tcp sport 80 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 22 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 443 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 993 ip daddr != 192.168.0.0/24 counter mark set 42
        }
}

The ip mangle table at the bottom is where the magic is. The webserver rule says, basically, that all packets being output from this server with a source port of 80 and a source address that is NOT my local network should be marked with “42”. Then, I set up routing for packets marked “42”. I needed to avoid packets on the local network or I couldn’t access my own webserver from the local network, because the responses would be directed out the DSL connection!

First, I had to create the routing table in /etc/iproute2/rt_tables by adding this line:

201 dsl.out

Here’s the command to create the separate dsl routing table. I put this as a post-up line in my network/interfaces for after the ethernet connection associated with my DSL is brought up:

ip rule add fwmark 42 table dsl.out

The actual contents of this routing table is defined by the following line which I put in a script in /etc/ppp/ip-up.d/0addroute:

ip route add default via 0.0.0.0 dev ppp0 table dsl.out

The next missing piece is Starlink statistics. Dishy gives a pretty slick statistics page on the phone app, which is also available from a browser if you hit it at the address 192.168.100.1. This is sort of a hidden network on the Starlink that needs a special rule if you’re not going to use their router. The following line in my /etc/network/interfaces file after the Starlink block makes it accessible:

post-up ip route add 192.168.100.0/24 via 0.0.0.0 dev enp39s0

This makes it accessible from the router, and as far as I’m concerned *should* make it accessible from the rest of the network as well. But it didn’t, and after an hour of being unable to figure why not, I just added a second NAT target that you can see in my nftables.conf above to the enp39s0 interface. This is a lame solution, but seems to work, and I don’t want to spend the rest of my life figuring it out.

So, I think I’m in the place where it works, a router reboot brings up both connections and all the nftables and routing automatically.

A side note: I’ve got Dishy plugged up to the UPS with my router and network gear. This means that when the power goes up I’ll still have Starlink! This is great because our power goes out all the time, and when it does, it takes the DSL with it because apparently CenturyLink can’t be bothered with battery backups.

The only thing that doesn’t work right now is that /etc/resolv.conf is currently getting stomped on by something; I suspect speedify. I’ll figure it out eventually, but I’m pretty happy with the set up so far.

I suspect long term the correct thing to do is just move this webserver in the cloud so it’s actually responsive. I’m pretty disappointed that Starlink didn’t make my webserver any faster for my tens of readers, but maybe there are other options (such as ipv6) that will improve things going forward.

I think I can say with reasonable confidence that I was wrong in my prior estimations of a super-low herd immunity threshold. Things are not looking good for COVID in most of the world. Colorado is seeing 10 times as many daily new cases as in the summer, and the United States as a whole is looking almost as bad. Colorado was barely untouched, so that doesn’t say much about the herd immunity claims, but then there’s Europe.

Europe looks so much worse than it did in the Spring. The UK, Spain, France and even Italy. The daily new case rate is hard to compare since we maxed out on testing capacity in March, pretty much all over the world. But the deaths do not lie, and they’re ramping up on a curve that looks like it’s going to match Spring. These are hard-hit places I’d expect to see showing some signs of immunity in this wave, but it’s not showing up anywhere I can see it.

But… why? The common claim is behavior has been returning back to normal. That’s maybe true, but what about Sweden? As far as I can tell Sweden has been fairly consistent in their response and they are seeing an enormous uptick in cases that I think we’d be hard-pressed to explain away with behavior changes.

Weather is starting to seem a likely culprit. I think it better explains the rise in the autumn than behavior shifts–the behavior shifts started in early summer and if that were the driving force, I’d have expected the uptick in cases to start way earlier than it did.

Where do we go from here? As long as we’re able to stay under capacity in hospitals, I think that full-scale lockdowns like we saw in the Spring continue to be unwarranted, and probably impossible anyway. But for those of us who can stay at home, now is the time to do it. It’s probably more important now than it was when everyone did it, and furthermore there’s an end in sight. In March, it was ridiculous to suggest that the majority of the world stay at home until there’s a vaccine, but given that we’re close now, this seems a more reasonable request now.

Stay safe and stay at home if you can.

Time for a bit of a personal COVID epistemic update.  To reiterate: I’m no expert, and while there’s no shortage of those, plenty of folks keep asking what I think.

First, let me talk about the things it looks like I may have been wrong about.  I’ve been pretty loudly claiming that the low-threshold herd immunity people are on to something.  I think they still are, but I said early on that I’d seriously reevaluate their claims once we saw significant second-wave infections in places that were hard-hit initially.  And now we have.  First up was Spain:

That alone is enough to make me question. But we’ve also got the UK:

My first thought is that it would be very interesting to break these things down by region. I spent awhile trying to do that for Spain, only realize the Spanish numbers are a giant mess, with a huge shift in record keeping between the two waves. My thought was that it’s possible that the hard-hit regions initially (Barcelona) shifted to a different region (Madrid) in the second wave. Maybe, but I don’t think it’s enough of an effect to matter. I couldn’t find much data to support the theory in Spain, and found nothing to support the theory at all in the UK.

So what about other places that were pretty hard hit, but don’t seem to have a second wave? There’s plenty of those, too:

So, what’s different? My theory at this point is that the severe lockdown in Spain and the UK were effective at the beginning and halted the spread before we got to the herd immunity threshold. There’s some evidence to support this with seroprevalence studies. NYC saw seroprevalence as high as 22% in April, whereas Spain was more like 6% with similar numbers for the UK. Still hardly a smoking gun — we know these seroprevalence numbers aren’t very good. It matters a lot when you look as antibody numbers drop pretty quickly, especially after a mild infection. But, assuming (and it’s a questionable assumption) that the numbers are equally crap across countries, the higher numbers in NYC may well be significant.

So, does this mean that “pursuing herd immunity” is a good strategy? Probably not, though I don’t think any country actually did that. Still, you can pick countries that did near-nothing and you see a curve that’s pretty familiar looking:

I think (though I don’t have much data to support this), that a Brazil-shaped curve is probably far better than a Spain-shaped curve. I think you can get a Brazil shaped curve with a much lower peak by enacting sensible measures that are far short of a lockdown. You know where this is going:

I think it’s starting to be pretty clear that in the long term, the total lockdown strategies should be judged a failure. They absolutely slowed the spread while countries were locked down, but I don’t see a significant benefit when you consider the post-lockdown bounce. And you must consider the post-lockdown bounce.

Why lockdown? The lockdowns were initiated with the idea that they’d keep healthcare systems from being saturated. They would “flatten the curve” — instead of overwhelming healthcare systems with uncontrollable, exponential growth, we could spread the inevitable infections out a bit so the healthcare system could manage it. In retrospect, however, I think it’s pretty clear that lockdowns weren’t actually necessarily to accomplish that goal. It became pretty clear early on that we’d accomplished that goal in the civilized west, but then, inexplicably the call shifted to, “crush the curve!”

The shifting goalposts lost me. It’s certainly possible to lockdown your way out of Coronavirus. It’s been successful in China and New Zealand so far, and it might even work in Australia. (If anyone wants to bet, my money is against it working in Australia long term, though they’ve done an amazing job so far.) But what about the U.S. or Europe? By the time we even considered it, it was too late.

The next-best argument for the lockdown is that we could use the time to stop the spread, set up test/trace/isolate, make lots of PPE, better align hospitals, etc. We certainly did none of this in the U.S. Germany implemented a super impressive sounding test and trace program…. how are they doing?

Yeah. That’s not looking like super-impressive results to me, but I think it’s too early to judge.

What else have I been wrong about? I predicted the whole country would look a lot like New York by October. I meant by the end of October, so technically, I guess I’ve got 30 days, but it’s not looking good.

We know this graph is distorted. The April peaks should probably be closer to the 100k line if we had the testing then we’ve got now. But who cares–we haven’t see a huge testing ramp-up in the month of September when I’d have expected to start seeing a fall-off. The worst-hit states are showing improvement: California, Texas, Florida. But what about places like Illinois? Or Colorado? This is a huge country, and we spent a long time locked down. I think it’s going to take another couple months.

So what should we be doing? We should be investing heavily in rapid tests. Cheap, not-very-accurate rapid tests. Such tests have low false-negative rate, but a higher false-positive rate. But most importantly, they very rarely report negative when someone is highly contagious. Who cares how good PCR tests are if they’re expensive, hard to get, inexplicably involve shoving an unobtanium swab into your brain, and are so sensitive they can detect the virus long after you’re contagious? Imagine how much better the world would be if your fellow coworkers or students or friends could all take a daily test. It doesn’t have to be perfect, but if the White House were doing it regularly (and, you know, acting on the results), we wouldn’t be looking at the disaster they’ve got right now.

What about a vaccine? I’m amused that everyone who thinks this virus is a deadly, terrifying scourge also believes that we should NOT rush a vaccine. Really? I think the worse this thing is, the more we should discount the risks of a bad vaccine in favor of the potential benefits. I’m not entirely sure where the right balance is, but the dissonance is deafening. Also, we’ve got thousands of young people who have volunteered for human challenge trials, but a world that’s too queasy to take them up on it. Why? The modern world has never seen anything like this, and I think it’s silly to be so sure that standard vaccine trials are necessarily the right answer here.

Okay, but what about individual decisions? I think, at this point and for the near future, theaters, cinema and live music are dead and should stay dead. I think any gathering of more than a dozen people indoors is unnecessary and shouldn’t happen. I think at-risk folks should avoid everything. But what about us younger folk?

It depends. An 18-year-old living at home with a 65-year-old has, I think, a duty to not expose themselves to anything that the 65-year-old shouldn’t be exposed to. But what about people like me? I live with an old woman, it’s true, but she’s not yet in the at-risk category. Sure, it’s possible either of us could have long-term health consequences from a run-in with COVID. But those risks are quite rare, and we ride motorcycles, which is far more likely to have long-term health consequences.

Personally, I’m mostly going about my life as normal, but my life isn’t that normal. I’ll forgo my annual trip to the movie theater, but I really miss live music. I’m going to restaurants less frequently, but a lot of that is just because going to restaurants sucks right now. I see small groups of friends now and again, and that’s all I really want, anyway.

But I’m also not visiting my mother, and this is rough for her. She’s lonely and hasn’t socialized in forever, and unlike me, she likes people. But she’s old, moth-eaten, and high-risk. I’m just not going to be the one that kills her (unless she asks me to). I’m considering a 2-week self-quarantine and then hopping in my plane to fetch her for a visit…

I’m convinced of a couple of things: At least for the U.S., it’s going to get better from here, not worse. Also, we’re all getting it eventually.