Archive for March, 2021

Science

30 March 2021 at 8:03 pm
by Jonah

The governor has announced that everyone in the state is eligible to get a vaccine as of Friday.

On the other hand, we have appointments to receive our third injections in the Novavax crossover trial in three/four weeks.

Do we abandon the trial and get some real shots? (The Lynn Institute instructed us to let them know if we had the opportunity to get an approved vaccine, and they would un-blind us to let us know if we are already full vaccinated or not.)

Or do we stick to the schedule? I put myself on a waiting list for a vaccine the other day because I figured I qualified under two or three qualifications under tier 1B.4. But I didn’t get a notification for an appointment.

I wonder how long it would take to get a notification for a vaccine appointment being in tier 1B.4. It seems like I might get an appointment before the general populace, which can start trying to get an appointment in three days. Would that be before my next Novavax appointment on April 26 at 9 a.m.?

When I signed up for the Novavax trial, it seemed like a good way to cut in line. I either got a vaccine months early or I qualified to be in tier 1B.4, one step ahead of everyone else. And it made me look like a hero! It didn’t seem like a hard decision.

Also, science!

But now that it comes down to it, I’ve been doing what I’ve been doing for a year and two weeks now. I could keep doing it for four more weeks and help scientists who are developing a highly effective and easily storable/transportable vaccine and help protect the world.

Or I could take my place in line, rightfully so according to the state, and get a “real” vaccine (which I might or might not actually be able get in the next four weeks anyway). And I could give up my status as a hero, as little as my part to play is.

So I took myself off the wait list.

Science.

Of course, I might have been fully vaccinated since January.

(Berck will tell you this is not a hard decision.)

The Flaming Lips Bubble Concert

24 March 2021 at 10:21 pm
by Jonah

The first year we were married, I worked in an un-airconditioned warehouse in Oklahoma City while my husband worked his way through flight school. Whenever I was having a bad day, my husband would put on Yoshimi Battles the Pink Robots, and it never failed to cheer me up. Despite being in the same city as The Flaming Lips, we never saw them live; we certainly never could have afforded tickets to one of their concerts, as I made just over the amount that would have qualified us for food stamps.

Seventeen years later, after moving to Colorado and after going a year without attending a live concert due to the COVID pandemic, we finally bought tickets to a Flaming Lips in Oklahoma City for their final space bubble concert, for a night we happened to be in Oklahoma for a Formula Vee race.

We’d reserved a hotel room within walking distance of the venue so we didn’t have to worry about parking our truck and trailer hauling the race car. Normally, this would have meant we could both drink at the concert and then walk back to the hotel, but it turns out there is no open bar at space bubble concerts. We did walk to a nearby restaurant and enjoyed several craft beers ahead of time. My husband ordered an enormous antipasto platter that we couldn’t finish and two pizzas, including a surprisingly good brisket and poblano pepper pizza. (“Tyler would tell us to order it,” he said, referring to the economist Tyler Cowen’s advice to order the most seemingly out of place item on a menu, because there is probably a good reason it is on there. Fortunately, the pizzas only came in one size, so that I didn’t antagonize my husband by order the largest size, per engineers’ advice.) The antipasto platter included some roasted broccoli, which I couldn’t help trying. I will point out that I resisted eating all of the roasted broccoli, which I really wanted to do. More on that later.

The concert tickets said that doors opened at 7, so we made our way over to the venue around 7:30. “Doors open” meant we could collect an envelope with wrist bands and then stand in our assigned rectangle chalked marked into the sidewalk out front. We were in row 2, bubble 2. I have to say that having a place to stand and not having people on either side of you in a regular line was nice.

chalk square
BUBBLE 2

We each got wanded. I was instructed to just take the things out of my pocket that would set the wand off, and I responded that probably all of it would set it off. Out came the usual (wallet, sunglasses case, money clip, fingernail clippers, lip balm, phone) and two items in the deep pockets of my BDU trousers brought especially for this concert: a pulse ox meter and a CO2 meter.

Around 8:15 a guy came out and explained to all of us what would happen next and encouraged us to use the bathroom NOW. At 8:30 we finally went inside. The floor of the venue was covered in deflated plastic bubbles. Row 1 were still getting into theirs.

deflated bubbles

We followed the people in front of us almost all the way to the far side of the room and found our bubble. We stepped through the unzipped slit and pulled the plastic up around us and over our heads. This was pretty claustrophobic. But right then a staff member with a leaf blower started inflating our bubble. We zipped up the zipper so only the tip of the leaf blower was sticking in our bubble.

just the tip

The bubble filled up with air quite quickly, and we zipped up the zipper the rest of the way as soon as the guy with the leaf blower pulled it out.

leaf blower dude

We were now alone in a bubble with some stuff at the bottom of it. We could take off our masks!

goodies

This whole time an instructional video was playing on a screen on the stage. It explained that we had a little battery operated fan, a bottle of water, a commemorative bandana, a towel, and a sign (on the other side it said, “IT’S HOT IN HERE”),

I didn't really

There were some also some glasses that made the lights look cool.

cool glasses

The last item in the bubble was a wireless speaker with which the fan caused interference when run next to it.

crappy speaker

Pretty much immediately after closing the bubble, it became uncomfortably hot, and we took off our flannel shirts. At this point, the carbon dioxide meter began alarming, so we had to turn the alarm off. Pretty soon the carbon dioxide stopped displaying anything but zeros. We finally figured out that it wasn’t malfunctioning but that it had reached a CO2 level so high that it couldn’t show a reading.

0000
26° C is 78.8° F

But once the concert started, it was easier to put up with how uncomfortable it was in the bubble. The band started playing from inside their individual bubbles, and confetti and beach balls dropped from above.

confetti and beach balls

An advantage of being in the bubble was that we didn’t get confetti in our hair or get bonked in the nose with beach balls. We did have fun trying to punch our bubble at exactly the right time a beach ball was landing on it to launch the beach ball back into the air.

The band started off with a great song to start the set, “Race for the Prize.”

Two scientists are racing
For the good of all mankind
Both of them side-by-side
So determined

Locked in heated battle
For the cure that is their prize
But it’s so dangerous
But they’re determined

Theirs is to win
If it kills them
They’re just humans
With wives and children

Upwards to the vanguard
Where the pressure is too high
Under the microscope
Hope against hope

Forging for the future
But to sacrifice their lives
Both of them side-by-side
So determined

This song was released in 1999, and I’ve listened to it multiple times, but tonight it made me think of Dr. Katalin Karikó and Dr. Lisa Jackson.

Wayne Coyne, the lead singer, emerged from his bubble to hold up a couple of custom balloons.

Oklahoma City

Wayne had to get a new bubble every so often to sing in because his bubble kept fogging up. Ours started collecting condensation as well, thus the towel provided.

Wayne's world

The leaf blower guys came by regularly, and they would mercifully aim the leaf blower at each of our faces, which felt so good. We only requested more air by holding up our sign a couple of times.

our savior

It was during one of these fresh air feeds that I couldn’t hold it in any longer and let loose the result of eating the broccoli earlier. So, yes, we were then trapped in a bubble with my fart.

The worst part was when the concert ended and we had to wait for all of the people in the back to exit first, which made me rethink the wisdom of getting a bubble in the second row. At some point our little fan had stopped, so I tried fanning with the sign, until my husband said, “That just causes you to produce more carbon dioxide.” Still, we were checking our oxygen levels with the pulse ox meter, and our oxygen levels were fine.

Still, it was an amazing experience, and I’m so glad to have done it. I also never want to do it again.

The band played a song off their new album, but most of the concert was their old hits, including several songs from Yoshimi. The sound quality was predictably not good, and I’m not sure that our portable speaker played anything but canned applause (you couldn’t really hear the people in the other bubbles). I don’t know if I can explain why I enjoy Yoshimi so much. A bit part is the lyrics, which are often sad but hopeful at the same time. (They also played “She Don’t Use Jelly”, which possibly has the most vapid lyrics of the entire 1990s.) The hit song off Yoshimi is “Do You Realize,” which, of course, they were going to play. But having lost someone important in my life five days previously, was especially poignant.

One, two, three, four

Do you realize
That you have the most beautiful face?
Do you realize
We’re floating in space?
Do you realize
That happiness makes you cry?
Do you realize
That everyone you know someday will die?

And instead of saying all of your goodbyes
Let them know you realize that life goes fast
It’s hard to make the good things last
You realize the sun doesn’t go down
It’s just an illusion caused by the world spinning round

Do you realize?

Ever to chance

17 March 2021 at 10:46 pm
by Jonah

Cliff Bennett died today. He and his wife had just sold their home in Georgia and bought one in Arizona to retire and be near their youngest daughter and youngest grandchild. But both of them caught COVID-19 around the beginning of the year. His wife recovered. Cliff was admitted to the hospital, eventually intubated and put on a ventilator, then given a tracheotomy. In February he was transferred to a rehabilitation hospital to try to wean him off the ventilator. But then he had to return to the hospital with a fungal infection and sepsis. He recovered from the infections, but his lungs were too far gone to be able to survive off the ventilator for long. His family made the difficult decision to take him off the machines keeping him alive on Saturday. He had been so healthy before catching Covid that his body lasted until today.

Cliff was my dad’s best friend, the principal of my small church school, later my pastor, and the officiant at my wedding. He and his wife agreed to be in my mom and dad’s will to take care of me and my siblings in case something happened to them while we were still children. Cliff had an astonishing ability to talk to and listen to anyone. He loved shopping for antiques, entranced by the story each little treasure held. He seemed to love all things old: old music, old movies, old baseball, and old pickup trucks. But maybe most of all old friends.

My favorite part of school as an elementary student was when the whole school would gather in assembly (there weren’t that many of us), and Cliff (back then Mr. Bennett) would read to us. He would read long engaging books to us, one chapter at a time. Sometimes he would read shorter pieces, and I’m pretty sure that’s where I first heard “Casey at the Bat”. The job of principal obviously called for being serious a lot of the time, but when he’d get to an especially funny part in a book, he’d try to keep reading while starting to uncontrollably laugh, before eventually having to stop, catch his breath, and exclaim, “Oh, me!”

I think it was Cliff’s first e-mail address that was tinkertoeverstochance, and he loved explaining its meaning. It was a baseball announcer call whenever Joe Tinker, Johnny Evers, and Frank Chance performed their signature double-play while playing for the Cubs between 1902 and 1912.

Baseball’s Sad Lexicon Franklin Pierce Adams

These are the saddest of possible words:
“Tinker to Evers to Chance.”
Trio of bear cubs, and fleeter than birds,
Tinker and Evers and Chance.
Ruthlessly pricking our gonfalon bubble,
Making a Giant hit into a double-
Words that are heavy with nothing but trouble:
“Tinker to Evers to Chance.”

I think I have to disagree. These are not the saddest of possible words.

Geeky Details of Starlink/DSL Bonding

4 March 2021 at 12:11 am
by Berck

Now that I’ve got Starlink up and running in a bonded connection with my DSL while still hosting my own webserver, I thought I’d share some details about how. I’ve learned that I’m old and when this all stops working in a year I’m never going to remember what I did or why, so I needed some rough documentation just for me. Additionally, I had a heck of a time figuring out how to do some of this stuff. In particular, the less common uses nftables seem to be poorly documented.

To start with, Starlink promised that during this beta program there would be downtime. I did some research before I got it, and as best as I could tell from what internet sleuths had put together, my little hexagon on the planet would have only 78% of a 24 hour period with have satellite coverage within 25 degrees of the horizon. It turns out that this is either outdated or wrong: I’m only seeing minutes per day without satellites and only a few minutes of what Starlink calls “beta outage”: ie they’re messing with stuff and they’re not sorry they turned your internet off while they were doing that. It’s only been a few days, but I think my DSL disconnects about as often as Starlink does.

In any case, I was convinced I needed the ability to bond my Starlink and DSL connection to have a continuous internet connection; plus an ability to use both connections for maximum bandwidth was also appealing. The only realistic way to do this in today’s world is with a VPN where local software sends any given TCP packet down one pipe or the other to a VPN server where they get mangled and forwarded on to their intended recipient as a coherent stream. The software is clever and when one side goes down, the dropped packets will appear down the other pipe. TCP is resilient and well-suited to this task.

You can probably roll your own with a cloud server, but the simple answer is just to get a Speedify account. They provide Linux (and other OS) software that will bond any sort of connection with any other at the other end of a VPN for $3/month. Sold.

While it runs on Linux, it appears that making it working on DD-WRT is something of a nightmare. And, while I was happy to ditch my AMD K6-400 OpenBSD router a couple of years ago for a spiffy Netgear R6700 running DD-WRT, it turns out that I hate DD-WRT. If it does what you want it to out of the box it’s great, but dealing with embedded Linux is a bit of a nightmare. When Andrew convinced me to stick this blog behind Cloudflare, the DD-WRT dynamic DNS client was ancient, didn’t support cloudflare, and replacing it with a different one involves a miserable build process.

So, I decided to upgrade my trusty 15-year-old Core2Duo webserver with something modern and press it into double duty as router and webserver. I’m pretty that I could have kept using it, but it was a good excuse to upgrade. I built an overkill 6-core AMD Ryzen 5 3600 system (because the cheap Ryzen 3 chips are completely unavailable). I loaded Debian on it, because while OpenBSD was a fun experiment back when I used to like computers, I’m just not that much of a masochist anymore. I bought a $30 4-port NIC to go in it, and now it’s a “router” as well as webserver. With some minor pain, I managed to get it working with DSL with a fairly bog-standard nftables setup.

[Rant: Linux changes the way to do IP masquerading constantly. It remains unclear that anything gets better, but it does change. First, there was ipfwadm. This worked great to share my dial-up SLIP connection with the rest of the house circa 1996. Then there was ipchains. And then when they switched to iptables 20 years ago, I refused to learn it because there was just going to be another new thing. Instead, I switched to OpenBSD and it turns out that pf is fantastic. I finally decided to give up and learn iptables for this project, only to discover that it’s been replaced by nftables.]

Starlink arrived days later than promised after FedEx Ground continues to position themselves as the delivery service with the slogan, “at least we’re more expensive than the post office!” After making Jonah climb on the roof to install it in the dark (I don’t do roofs), I plugged it up to provided wireless router and had it up and running in minutes on its own wifi.

Step 1: Make Starlink work in Linux. This was easy. Unplug the provided router and connect to the Linux router instead. Add the following lines to /etc/network/interfaces:

auto enp39s0
iface enp39s0 inet dhcp

Have I mentioned that the new kernel naming for ethernet devices is terrible? It’s well-intentioned, but I’d rather they just stuck with eth0, eth1 etc and let me write my own persistent udev rules. enp39s0 just rolls off the tongue and is so easy to type, what’s not to like?

Anyway, that’s all that’s needed to get Starlink up and happy on the router. But now I needed to get Speedify working to bond the two.

Speedify’s documentation for Linux is terrible. They have a rough “here’s how to do this on a Raspberry Pi” series of how-to’s, and some screenshots for a UI that wouldn’t run on my Debian installation because it depends on some package that’s long-removed from testing. Whatever, I didn’t want to install an X server ayway.

After getting it installed and logging in with:

/usr/share/speedify/speedify_cli login username password

and then

/usr/share/speedify/speedify_cli connect

I was able to access the internet through the speedify VPN, but only from either Starlink or DSL. I needed to remove the “replacedefaultroute” from my /etc/ppp/peers/dsl-provider file to prevent the ppp connection from blasting over the default route that the Starlink dhcp server provided. Once I did that, ran /usr/share/speedify/speedify_cli startupconnect on , and ensured that the Starlink block appeared *before* the DSL block in my /etc/network/interfaces file, things seemed to work nicely on startup.

It turns out that there’s a completely undocumented file, /etc/speedify/speedify.conf. Here’s mine:

# Set to 1 to enable sharing Speedify to other devices
ENABLE_SHARE=1

# The interface(s) to use for sharing
# ex: SHARE_INTERFACE="eth1"
# When you enable sharing, Speedify will automaticlly set this interface to the NEVER priority so that it is not used as an Internet connection. 
# If you disable sharing for the interface and want to use it again as an Internet connection, you can set it back to the Always priority by doing:
# /usr/share/speedify/speedify_cli adapter priority {interface} always
SHARE_INTERFACE="enp34s0"

# IP to use for the sharing interface
# ex: INTERFACE_IP="192.168.145.1"
INTERFACE_IP="192.168.0.15"

# DNS servers to send over DHCP to clients
# ex: DNS_SERVERS="10.202.0.1,1.1.1.1"
DNS_SERVERS="10.202.0.1,1.1.1.1"

# Set to 1 to allow internet access on other devices when Speedify is disconnected
SHARE_WITHOUT_SPEEDIFY=0

ENABLE_SHARE=1 means that speedify will set up its own nftable rules to enable NAT (masquerade for us ancient folk). It will also ham-fistedly install/enable dnsmasq, which if you’re running isc-dhcp-server will conflict. I removed dnsmasq. The DNS_SERVERS line up there is, I think, used to configure dnsmasq, so I don’t care about that because I’m running my own DNS server as well.

So, this is all you need for a simple shared speedify bonded Starlink/DSL setup. But I run a webserver, IMAP server, want to be able to ssh into my home network from outside, etc. And that’s where it gets nasty.

I spent an embarrassingly long time trying to figure out why ddclient wouldn’t update Cloudflare’s DNS records with my Starlink IP. Because I’m old and don’t keep up with the proceedings of the IETF, I missed this: 100.64.74.22 is *not* a public IP address. Given that it starts with 100 and not 192 or 10, I didn’t register it as a private address. But it really, really is.

So, yeah, this is probably the crappiest thing about Starlink for me so far. We ran out of ipv4 addresses a long time ago, so I shouldn’t be at all surprised that Starlink couldn’t get any. This sucks and means that my webserver is not going to be accessible via ipv4 and Starlink. It remains to be seen if it’s accessible via ipv6, but ipv6 doesn’t work for me yet, and that’s a different headache entirely.

Speedify will gladly support inbound traffic through your VPN if you buy a dedicated server from them for $120/month. Yeah, not so much for me thanks.

That leaves my webserver stuck with DSL. But when connected to speedify with the default nftables, if you hit my webserver it’ll get your request and respond… via the bonded speedify connection. Which means you’re never going to get an established connection.

So what I needed was a way for *most* traffic leaving the webserver/router to do so on the speedify connection, but traffic that is webserver responses needs to go out on the DSL connection only. Yikes.

I spent a long time figuring this out. It’s possible if I’d learned iptables, that the solution would be more obvious, because the nftables documentation does not mention this at all. At first, it looks like you can’t really do routing with nftables at all. And you can’t, really. But you can mark packets and then let iproute direct marked packets to a different routing table. This is poorly documented, but thanks to this random forum post, I was able to figure it out. Here’s what I did.

My /etc/nftables.conf:

#!/usr/sbin/nft -f

flush ruleset

define lan = enp34s0

table inet filter {
        chain input {
                type filter hook input priority 0
        }
        chain forward {
                type filter hook forward priority 0
                oifname "ppp0" tcp flags syn tcp option maxseg size set rt mtu
        }
        chain output {
                type filter hook output priority 0
        }
}

table ip nat {
        chain postrouting {
                type nat hook postrouting priority 0; policy accept
                oifname "enp39s0" masquerade
        }
}
table ip mangle {
         chain output {
                type route hook output priority mangle; policy accept
                tcp sport 80 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 22 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 443 ip daddr != 192.168.0.0/24 counter mark set 42
                tcp sport 993 ip daddr != 192.168.0.0/24 counter mark set 42
        }
}

The ip mangle table at the bottom is where the magic is. The webserver rule says, basically, that all packets being output from this server with a source port of 80 and a source address that is NOT my local network should be marked with “42”. Then, I set up routing for packets marked “42”. I needed to avoid packets on the local network or I couldn’t access my own webserver from the local network, because the responses would be directed out the DSL connection!

First, I had to create the routing table in /etc/iproute2/rt_tables by adding this line:

201 dsl.out

Here’s the command to create the separate dsl routing table. I put this as a post-up line in my network/interfaces for after the ethernet connection associated with my DSL is brought up:

ip rule add fwmark 42 table dsl.out

The actual contents of this routing table is defined by the following line which I put in a script in /etc/ppp/ip-up.d/0addroute:

ip route add default via 0.0.0.0 dev ppp0 table dsl.out

The next missing piece is Starlink statistics. Dishy gives a pretty slick statistics page on the phone app, which is also available from a browser if you hit it at the address 192.168.100.1. This is sort of a hidden network on the Starlink that needs a special rule if you’re not going to use their router. The following line in my /etc/network/interfaces file after the Starlink block makes it accessible:

post-up ip route add 192.168.100.0/24 via 0.0.0.0 dev enp39s0

This makes it accessible from the router, and as far as I’m concerned *should* make it accessible from the rest of the network as well. But it didn’t, and after an hour of being unable to figure why not, I just added a second NAT target that you can see in my nftables.conf above to the enp39s0 interface. This is a lame solution, but seems to work, and I don’t want to spend the rest of my life figuring it out.

So, I think I’m in the place where it works, a router reboot brings up both connections and all the nftables and routing automatically.

A side note: I’ve got Dishy plugged up to the UPS with my router and network gear. This means that when the power goes up I’ll still have Starlink! This is great because our power goes out all the time, and when it does, it takes the DSL with it because apparently CenturyLink can’t be bothered with battery backups.

The only thing that doesn’t work right now is that /etc/resolv.conf is currently getting stomped on by something; I suspect speedify. I’ll figure it out eventually, but I’m pretty happy with the set up so far.

I suspect long term the correct thing to do is just move this webserver in the cloud so it’s actually responsive. I’m pretty disappointed that Starlink didn’t make my webserver any faster for my tens of readers, but maybe there are other options (such as ipv6) that will improve things going forward.