Our Dumb Blog

Cliff Bennett died today. He and his wife had just sold their home in Georgia and bought one in Arizona to retire and be near their youngest daughter and youngest grandchild. But both of them caught COVID-19 around the beginning of the year. His wife recovered. Cliff was admitted to the hospital, eventually intubated and put on a ventilator, then given a tracheotomy. In February he was transferred to a rehabilitation hospital to try to wean him off the ventilator. But then he had to return to the hospital with a fungal infection and sepsis. He recovered from the infections, but his lungs were too far gone to be able to survive off the ventilator for long. His family made the difficult decision to take him off the machines keeping him alive on Saturday. He had been so healthy before catching Covid that his body lasted until today.

Cliff was my dad’s best friend, the principal of my small church school, later my pastor, and the officiant at my wedding. He and his wife agreed to be in my mom and dad’s will to take care of me and my siblings in case something happened to them while we were still children. Cliff had an astonishing ability to talk to and listen to anyone. He loved shopping for antiques, entranced by the story each little treasure held. He seemed to love all things old: old music, old movies, old baseball, and old pickup trucks. But maybe most of all old friends.

My favorite part of school as an elementary student was when the whole school would gather in assembly (there weren’t that many of us), and Cliff (back then Mr. Bennett) would read to us. He would read long engaging books to us, one chapter at a time. Sometimes he would read shorter pieces, and I’m pretty sure that’s where I first heard “Casey at the Bat”. The job of principal obviously called for being serious a lot of the time, but when he’d get to an especially funny part in a book, he’d try to keep reading while starting to uncontrollably laugh, before eventually having to stop, catch his breath, and exclaim, “Oh, me!”

I think it was Cliff’s first e-mail address that was tinkertoeverstochance, and he loved explaining its meaning. It was a baseball announcer call whenever Joe Tinker, Johnny Evers, and Frank Chance performed their signature double-play while playing for the Cubs between 1902 and 1912.

Baseball’s Sad Lexicon Franklin Pierce Adams

These are the saddest of possible words:
“Tinker to Evers to Chance.”
Trio of bear cubs, and fleeter than birds,
Tinker and Evers and Chance.
Ruthlessly pricking our gonfalon bubble,
Making a Giant hit into a double-
Words that are heavy with nothing but trouble:
“Tinker to Evers to Chance.”

I think I have to disagree. These are not the saddest of possible words.

Now that I’ve got Starlink up and running in a bonded connection with my DSL while still hosting my own webserver, I thought I’d share some details about how. I’ve learned that I’m old and when this all stops working in a year I’m never going to remember what I did or why, so I needed some rough documentation just for me. Additionally, I had a heck of a time figuring out how to do some of this stuff. In particular, the less common uses nftables seem to be poorly documented.

To start with, Starlink promised that during this beta program there would be downtime. I did some research before I got it, and as best as I could tell from what internet sleuths had put together, my little hexagon on the planet would have only 78% of a 24 hour period with have satellite coverage within 25 degrees of the horizon. It turns out that this is either outdated or wrong: I’m only seeing minutes per day without satellites and only a few minutes of what Starlink calls “beta outage”: ie they’re messing with stuff and they’re not sorry they turned your internet off while they were doing that. It’s only been a few days, but I think my DSL disconnects about as often as Starlink does.

In any case, I was convinced I needed the ability to bond my Starlink and DSL connection to have a continuous internet connection; plus an ability to use both connections for maximum bandwidth was also appealing. The only realistic way to do this in today’s world is with a VPN where local software sends any given TCP packet down one pipe or the other to a VPN server where they get mangled and forwarded on to their intended recipient as a coherent stream. The software is clever and when one side goes down, the dropped packets will appear down the other pipe. TCP is resilient and well-suited to this task.

You can probably roll your own with a cloud server, but the simple answer is just to get a Speedify account. They provide Linux (and other OS) software that will bond any sort of connection with any other at the other end of a VPN for $3/month. Sold.

While it runs on Linux, it appears that making it working on DD-WRT is something of a nightmare. And, while I was happy to ditch my AMD K6-400 OpenBSD router a couple of years ago for a spiffy Netgear R6700 running DD-WRT, it turns out that I hate DD-WRT. If it does what you want it to out of the box it’s great, but dealing with embedded Linux is a bit of a nightmare. When Andrew convinced me to stick this blog behind Cloudflare, the DD-WRT dynamic DNS client was ancient, didn’t support cloudflare, and replacing it with a different one involves a miserable build process.

So, I decided to upgrade my trusty 15-year-old Core2Duo webserver with something modern and press it into double duty as router and webserver. I’m pretty that I could have kept using it, but it was a good excuse to upgrade. I built an overkill 6-core AMD Ryzen 5 3600 system (because the cheap Ryzen 3 chips are completely unavailable). I loaded Debian on it, because while OpenBSD was a fun experiment back when I used to like computers, I’m just not that much of a masochist anymore. I bought a $30 4-port NIC to go in it, and now it’s a “router” as well as webserver. With some minor pain, I managed to get it working with DSL with a fairly bog-standard nftables setup.

[Rant: Linux changes the way to do IP masquerading constantly. It remains unclear that anything gets better, but it does change. First, there was ipfwadm. This worked great to share my dial-up SLIP connection with the rest of the house circa 1996. Then there was ipchains. And then when they switched to iptables 20 years ago, I refused to learn it because there was just going to be another new thing. Instead, I switched to OpenBSD and it turns out that pf is fantastic. I finally decided to give up and learn iptables for this project, only to discover that it’s been replaced by nftables.]

Starlink arrived days later than promised after FedEx Ground continues to position themselves as the delivery service with the slogan, “at least we’re more expensive than the post office!” After making Jonah climb on the roof to install it in the dark (I don’t do roofs), I plugged it up to provided wireless router and had it up and running in minutes on its own wifi.

Step 1: Make Starlink work in Linux. This was easy. Unplug the provided router and connect to the Linux router instead. Add the following lines to /etc/network/interfaces:

auto enp39s0
iface enp39s0 inet dhcp

Have I mentioned that the new kernel naming for ethernet devices is terrible? It’s well-intentioned, but I’d rather they just stuck with eth0, eth1 etc and let me write my own persistent udev rules. enp39s0 just rolls off the tongue and is so easy to type, what’s not to like?

Anyway, that’s all that’s needed to get Starlink up and happy on the router. But now I needed to get Speedify working to bond the two.

Speedify’s documentation for Linux is terrible. They have a rough “here’s how to do this on a Raspberry Pi” series of how-to’s, and some screenshots for a UI that wouldn’t run on my Debian installation because it depends on some package that’s long-removed from testing. Whatever, I didn’t want to install an X server ayway.

After getting it installed and logging in with:

/usr/share/speedify/speedify_cli login username password

and then

/usr/share/speedify/speedify_cli connect

I was able to access the internet through the speedify VPN, but only from either Starlink or DSL. I needed to remove the “replacedefaultroute” from my /etc/ppp/peers/dsl-provider file to prevent the ppp connection from blasting over the default route that the Starlink dhcp server provided. Once I did that, ran /usr/share/speedify/speedify_cli startupconnect on , and ensured that the Starlink block appeared *before* the DSL block in my /etc/network/interfaces file, things seemed to work nicely on startup.

It turns out that there’s a completely undocumented file, /etc/speedify/speedify.conf. Here’s mine:

# Set to 1 to enable sharing Speedify to other devices
ENABLE_SHARE=1

# The interface(s) to use for sharing
# ex: SHARE_INTERFACE="eth1"
# When you enable sharing, Speedify will automaticlly set this interface to the NEVER priority so that it is not used as an Internet connection. 
# If you disable sharing for the interface and want to use it again as an Internet connection, you can set it back to the Always priority by doing:
# /usr/share/speedify/speedify_cli adapter priority {interface} always
SHARE_INTERFACE="enp34s0"

# IP to use for the sharing interface
# ex: INTERFACE_IP="192.168.145.1"
INTERFACE_IP="192.168.0.15"

# DNS servers to send over DHCP to clients
# ex: DNS_SERVERS="10.202.0.1,1.1.1.1"
DNS_SERVERS="10.202.0.1,1.1.1.1"

# Set to 1 to allow internet access on other devices when Speedify is disconnected
SHARE_WITHOUT_SPEEDIFY=0

ENABLE_SHARE=1 means that speedify will set up its own nftable rules to enable NAT (masquerade for us ancient folk). It will also ham-fistedly install/enable dnsmasq, which if you’re running isc-dhcp-server will conflict. I removed dnsmasq. The DNS_SERVERS line up there is, I think, used to configure dnsmasq, so I don’t care about that because I’m running my own DNS server as well.

So, this is all you need for a simple shared speedify bonded Starlink/DSL setup. But I run a webserver, IMAP server, want to be able to ssh into my home network from outside, etc. And that’s where it gets nasty.

I spent an embarrassingly long time trying to figure out why ddclient wouldn’t update Cloudflare’s DNS records with my Starlink IP. Because I’m old and don’t keep up with the proceedings of the IETF, I missed this: 100.64.74.22 is *not* a public IP address. Given that it starts with 100 and not 192 or 10, I didn’t register it as a private address. But it really, really is.

So, yeah, this is probably the crappiest thing about Starlink for me so far. We ran out of ipv4 addresses a long time ago, so I shouldn’t be at all surprised that Starlink couldn’t get any. This sucks and means that my webserver is not going to be accessible via ipv4 and Starlink. It remains to be seen if it’s accessible via ipv6, but ipv6 doesn’t work for me yet, and that’s a different headache entirely.

Speedify will gladly support inbound traffic through your VPN if you buy a dedicated server from them for $120/month. Yeah, not so much for me thanks.

That leaves my webserver stuck with DSL. But when connected to speedify with the default nftables, if you hit my webserver it’ll get your request and respond… via the bonded speedify connection. Which means you’re never going to get an established connection.

So what I needed was a way for *most* traffic leaving the webserver/router to do so on the speedify connection, but traffic that is webserver responses needs to go out on the DSL connection only. Yikes.

I spent a long time figuring this out. It’s possible if I’d learned iptables, that the solution would be more obvious, because the nftables documentation does not mention this at all. At first, it looks like you can’t really do routing with nftables at all. And you can’t, really. But you can mark packets and then let iproute direct marked packets to a different routing table. This is poorly documented, but thanks to this random forum post, I was able to figure it out. Here’s what I did.

My /etc/nftables.conf:

#!/usr/sbin/nft -f

flush ruleset

define lan = enp34s0

table inet filter {
        chain input {
                type filter hook input priority 0
        }
        chain forward {
                type filter hook forward priority 0
                oifname "ppp0" tcp flags syn tcp option maxseg size set rt mtu
        }
        chain output {
                type filter hook output priority 0
        }
}

table ip nat {
        chain postrouting {
                type nat hook postrouting priority 0; policy accept
                oifname "enp39s0" masquerade
        }
}
table ip mangle {
         chain output {
                type route hook output priority mangle; policy accept
                tcp sport 80 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 22 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 443 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 993 ip daddr != 192.168.0.0/24 counter mark set 42
        }
}

The ip mangle table at the bottom is where the magic is. The webserver rule says, basically, that all packets being output from this server with a source port of 80 and a source address that is NOT my local network should be marked with “42”. Then, I set up routing for packets marked “42”. I needed to avoid packets on the local network or I couldn’t access my own webserver from the local network, because the responses would be directed out the DSL connection!

First, I had to create the routing table in /etc/iproute2/rt_tables by adding this line:

201 dsl.out

Here’s the command to create the separate dsl routing table. I put this as a post-up line in my network/interfaces for after the ethernet connection associated with my DSL is brought up:

ip rule add fwmark 42 table dsl.out

The actual contents of this routing table is defined by the following line which I put in a script in /etc/ppp/ip-up.d/0addroute:

ip route add default via 0.0.0.0 dev ppp0 table dsl.out

The next missing piece is Starlink statistics. Dishy gives a pretty slick statistics page on the phone app, which is also available from a browser if you hit it at the address 192.168.100.1. This is sort of a hidden network on the Starlink that needs a special rule if you’re not going to use their router. The following line in my /etc/network/interfaces file after the Starlink block makes it accessible:

post-up ip route add 192.168.100.0/24 via 0.0.0.0 dev enp39s0

This makes it accessible from the router, and as far as I’m concerned *should* make it accessible from the rest of the network as well. But it didn’t, and after an hour of being unable to figure why not, I just added a second NAT target that you can see in my nftables.conf above to the enp39s0 interface. This is a lame solution, but seems to work, and I don’t want to spend the rest of my life figuring it out.

So, I think I’m in the place where it works, a router reboot brings up both connections and all the nftables and routing automatically.

A side note: I’ve got Dishy plugged up to the UPS with my router and network gear. This means that when the power goes up I’ll still have Starlink! This is great because our power goes out all the time, and when it does, it takes the DSL with it because apparently CenturyLink can’t be bothered with battery backups.

The only thing that doesn’t work right now is that /etc/resolv.conf is currently getting stomped on by something; I suspect speedify. I’ll figure it out eventually, but I’m pretty happy with the set up so far.

I suspect long term the correct thing to do is just move this webserver in the cloud so it’s actually responsive. I’m pretty disappointed that Starlink didn’t make my webserver any faster for my tens of readers, but maybe there are other options (such as ipv6) that will improve things going forward.

I had another visit to the Lynn Institute today. This time was just to give them some blood, two and a half vials, which is apparently all my vein wanted give up, according to my phlebotomist Angelica.

My study administrator told me she had some very interesting news. She said she would know more on Friday but she had heard it on good authority that our vaccine trial was going to be phased into a crossover study, meaning that we would be receiving two injections of whatever we hadn’t received the first time around. So if I got the real vaccine, I’d be injected with two shots of the placebo. But more importantly, if I’d received the placebo, I’d get the real thing. She said she thought this would happen around the end of April. Since the timeline of me getting a vaccine through normal procedures means I might not get one until summer or even fall, that’s pretty darn good news!

Also, Novavax’s previous trial showed nearly 90% efficacy, which is really, really good!

And now I have a correction to post about a previous blog entry. Another member of this household has also entered into the Novavax trial, and I was informed that it turns out the syringe I was injected with was covered in sparkly blue tape. The stuff inside is clear. So no protomolecule for me after all.

Today I had an appointment for my second injection for the Novavax vaccine trial. I go to the clinic a couple minutes early, but I had to wait quite a while because, as they told all of us waiting in the waiting room sitting as far apart as possible, one of the participants undergoing the injection had passed out. But it wasn’t because of the injection. She passed out when they were trying to draw blood. According to one of the employees, “She said she looked away but then she heard the needle pop through her skin, and that’s when she fainted.”

This time I got an injection in the other arm, but it was the same injection as last time: a 2/3rds chance I got the vaccine, a 1/3 chance I got saline. I gave urine and blood again and had the world’s shortest physical by the same doctor as last time, who said, “She looks excellent,” to which I replied, “Why, thank you!” This time I couldn’t see inside the syringe to see if it was the same weird blue color. This time Bobbie seemed to really jab it into my arm, which hurt pretty good. Just like last time, my arm started burning right about five minutes after the injection.

Tonight my arm is sore, though not a whole lot. It’s throbbing a little.

I go back in two weeks to give more blood.

I asked when the trial would be unmasked, and they said they didn’t know. They said that if I get a chance to get a real vaccine, to call them first and they’d let the cat out of the bag.

This clinic has also been conducting Moderna trials. In the waiting room beforehand and in the clinic break room where we had to wait for half an hour before we could leave, just in case we had an adverse effects, there were other people there who had been in the Moderna trial last year. They had received their first injection in August and their second in September. Today they were informed that they had received placebos and so they were given the real Moderna vaccine today. I asked a lady who had just gotten hers if the real one felt any different than the placebo injections, but she said no. She had been convinced that she had received the real vaccine last year, but it turned out she didn’t after all.

My phone’s web browser Chrome likes to show me news stories it thinks I’d be interested in reading (it’s convinced I’m obsessed with Star Wars, The Expanse, Phoebe Bridgers, and asteroids, so what’s what most of the articles are about). Day before yesterday it popped up an article from KRDO saying that phase three trials were being conducted in Colorado Springs for a new COVID-19 vaccine.

So I Googled the name of the company Novavax and Colorado Springs, and the website that popped up offered to let me take a survey to see if I were eligible to participate. For this particular trial they only want participants who meet certain criteria of having a greater risk of becoming infected with COVID-19. Examples include:

Racial minority (nope)

Live in crowded conditions such as shared housing (nope)

Be an essential worker in close contact with the public or other workers (not really)

Are 65 or older (not yet)

Have underlying medical conditions

I checked the list of underlying medical conditions, since I’m pretty healthy. Turns out one of the conditions is being FAT!

At the end of the survey, it said that I appeared to qualify and to call a number with a (405) area code.

The next morning at 8 a.m. I called the number. The woman who answered asked me the same questions that I had answered in the online questionnaire and told me that I qualified for the trial. “Can you come in tomorrow?” “What time?” I answered. She said 11 a.m. and gave me the address. I asked what the sign on the door said, and she paused and answered, “I’m sorry, I don’t know. I’m in Oklahoma City.” I guess I didn’t have to wait until 8 a.m. to call.

This morning I showed up at a doctor’s office at 11 a.m. I had to fill out a bunch of paperwork, then I was ushered into an exam room. For the next two hours, a stream of different people came in to ask me detailed questions about my medical history, perform a physical exam, request urine, measure my height, take my weight, take my blood pressure, take my blood, and finally administer an injection.

A woman in blue scrubs came in. “Hi, I’m Bobbie!” she greeted me. She asked me which arm I wanted the injection. I chose my right arm, and she said my second injection in three weeks would be in my opposite arm. “You are participant number 25,” she said, reading from my chart. Then she uncapped a syringe holding what appeared to be filled with a substance that I can only describe to be a florescent shade of navy blue. It reminded me of the protomolecule from The Expanse (maybe I am obsessed after all). She swabbed my right upper arm just below my shoulder and gave me a quick jab. I hardly felt anything at all.

This vaccine trial uses a lab engineered spike protein to try to teach your body that spike proteins like the one in SARS-COV-2 are bad. 2/3rds of the participants are given the vaccine, and 1/3rd are given a placebo of saline. I got paid $150 for my visit today, and I get more money for each additional visit (one in three weeks for my second injection, one a week after that, and then once every six months for the next two years). Well, at least I got a gift card that I was told money would be added to.

At this point, I finally got to leave my personal exam room and was taken to the clinic’s break room where a clinic worker was sitting with a laptop on one side of a conference table and a stylishly dressed middle aged black woman was sitting on the other side. I took a seat as well, and the clinic worker instructed me to download an app onto my phone called Patient Cloud and register an account and read the instructions in the app. This was tricky, because right then the clinic worker started recounting a hilarious TikTok video to the other trial participant with scene by scene detail while I was trying to read some very dry instructions.

“My arm stings!” I exclaimed, interrupting the TikTok recantation. “How badly does it hurt?” asked the clinic worker, clinically, suddenly typing into her laptop. I glanced at the other trial participant to see if any sign of agreement appeared in her eyes, but she just continued to sit there pleasantly. “It only stings a little,” I answered. A bit later I added, “My muscle hurts!” “How badly does it hurt?” asked the clinic worker again. “Just a little,” I answered.

The clinic worker took the blood pressure of the other trial participant and then told her she had not had an adverse reaction to her injection in the last 30 minutes so she was free to go. Then the clinic worker made sure I had completed my first two surveys on the downloaded app. She gave me a thermometer and a ruler and told me to take my temperature and measure any rash that may appear at my injection site and record the results once a day in the app. She took my temperature and blood pressure again and then, satisfied with the results, told me that I could leave.

This particular vaccine is probably not as effective as the Pfizer of Moderna vaccines. On the other hand, it only has to be refrigerated, not kept at super cold temperatures. At this point I think I probably got the vaccine instead of the placebo, but I don’t know when they’ll tell me. Supposedly, according to Colorado vaccine plan, I get to be in phase 2 if I did get the placebo (though it is unclear if I’m an essential enough worker to be there anyway).

Tonight my right upper arm hurts, like when I get a flu shot, except worse. Not enough to bother me, just enough to notice.

I have never been so excited to have a sore arm.

(No, Bobbie didn’t have a New Zealander accent.)