Our Dumb Blog

Someone goes postal on my server.

by

Berck

Posted in:

I’m not the best web server admin, mostly because it tends to bore me. My little server spends most of its time idle, and 90% of the requests it gets are from Google’s bot. (I’ve actually just used Google’s web-admin tools to seriously reduce the crawl rate on my server, because its continually re-examining images that haven’t changed in a decade… To the tune of 2GB per month. Things are much more acceptable, now.)

Anyway, I was listening to my new old Dan Bern album (New American Language) and it started stuttering. I quickly determined the problem wasn’t with the rip of the used CD, but that my server was busy thrashing its swap space. Several iterations of manually invoking the OOM-killer proved ineffective, and I managed to determine there were a zillion instances of php5 running.

I looked at the logs, and found this:

56.0.143.25 – – [08/Mar/2012:11:25:58 -0700] “GET /?p=3335 HTTP/1.1” 200 36014 “http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd
=1&cts=1331231157738&ved=0CCMQFjAA&url=http%3A%2F%2Fnachzen.net%2F%3Fp%3D3335&ei=s_lYT_3-BKqOigKD0ei8Cw&usg=AFQjCNFJvrOwt-dQ-iD6Y5bl6fsiQLZc
pQ&sig2=2JwwkvbKO_70wnK-o7UU8A” “Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /blog/wp-content/themes/berck-steam/style.css HTTP/1.1” 200 8018 “-” “Mozilla/4.0 (compati
ble;)”
56.0.143.25 – – [08/Mar/2012:11:26:00 -0700] “GET /favicon.ico HTTP/1.1” 200 622 “-” “Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101
Firefox/10.0.2”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /blog/wp-includes/wlwmanifest.xml HTTP/1.1” 200 1350 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200901 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200912 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201107 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201106 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?feed=rss HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201012 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201008 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199508 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199702 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199501 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=200003 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199612 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199804 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199602 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199809 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?p=3340 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199510 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /blog/xmlrpc.php?rsd HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?p=3333 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199701 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?p=3335 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199405 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199609 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199408 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199803 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199411 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:03 -0700] “GET /?m=199805 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201010 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201004 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200905 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200806 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201005 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201003 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200812 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200911 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201001 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200811 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?feed=rss2 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200904 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200902 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200805 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201105 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200903 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:26:06 -0700] “GET /?m=199603 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201112 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201110 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200810 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200908 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200910 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201111 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?feed=atom HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200907 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201104 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201103 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201109 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200808 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201006 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201002 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201009 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=200909 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201011 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201202 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201007 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”
56.0.143.25 – – [08/Mar/2012:11:25:59 -0700] “GET /?m=201101 HTTP/1.1” 504 550 “-” “Mozilla/4.0 (compatible;)”

It goes on, but I’ve truncated the paste. It’s a recursive request for everything on the front page of my site, which isn’t nice, but wouldn’t have caused problems except that they were all concurrent. Which is evil. My first thought it was intentionally malicious, but this is weird:

host 56.0.143.25
25.143.0.56.in-addr.arpa domain name pointer gk-central-25.srvs.usps.gov.

The USPS, not China. And the first entry appears to be someone who found the page from a Google search. I’m wondering if maybe there’s some evil Firefox extension that spawns a million connections and tries to download an entire website before you ever click on anything, just in case you want something?

Anyway, I learned my lesson about default values. Max Connections of 150 is no good for dynamic page content on a P4 with 1GB of RAM. Some testing reveals I can support 32 connections without death and destruction resulting (though it’s certainly not fast). So, changed that value to 32 and I’m good to go for now.

Of course, it brings up all kinds of silliness with dynamic page content. There’s probably lots of optimization I could do, but I have homework, and future instances of this will at least keep my music playing.

6 responses to “Someone goes postal on my server.”

  1. Berck Avatar
    Berck

    Ah hah. I think it’s the “FasterFox” extension, which appears to set max concurrent connections to something insane, then prefetches everything on the page. Some people should be shot.

  2. nana Avatar
    nana

    Um, Berck… if/when you have the desire, could you translate this post for simpletons?

  3. Berck Avatar
    Berck

    You weren’t really the intended audience…

  4. nana Avatar
    nana

    Well, yeah, I know that… but the blog is “public.” That’s why I said, “if you wish to.” And if you don’t, don’t!

  5. Berck Avatar
    Berck

    My server crashed. I think it happened as a result of a poorly-designed firefox extension that claims to “speed up” browsing, combined with the fact that my server was poorly configured.

  6. nana Avatar
    nana

    Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.