Elliott C. Back: Internet & Technology

I’ve just added two little cute features to the main page of my website. They don’t do much more than improve the usability and aesthetic of the front page by a tiny margin. The first is quite practical–it alerts you and sets the 404 status code if you loaded my site through a domain or subdomain:

The second is a Flickr badge across the top of my page, with a custom-made Flickr logo to take you to my Flick page:

But this is Web 2.0, and I use the Thickbox script in other places on my site, so why not here too?

It’s fun tinkering around with your main page. I need to add a cookie-rotator to the image on the front page, rather than make it time based. Then people can see different versions of me everytime they come back, rather than the current “new elliott at 1 AM” business.

I got a lovely email just now threatening me for being a notorious spammer:

Your doing it to drive up your Google Rank is pitiful, though I’ve informed Google of your attempts to game their system. Further evidence of scraping will be dealt with through the legal system. Perhaps a note to [your employer] will be of use as well.

I sent back my reply, which indicates that no I am not a spammer, thank you very much:

I’m terribly sorry you are experiencing web scrapers, but honest-to-god it’s not me. I wrote a plugin a long time ago for Wordpress called “WP Autoblog” that can take an RSS feed and import them as a series of posts. The posts get branded with attribution like “Post by XYZ and software by me” which you’re probably mistaking for something I’m actively a part of. I wrote the plugin to aggregate some of my family blogs (ericback.com, elliottback.com) together into a single feed, but it quickly became abused by spammers so I pulled it. You can read more here.

All this in spite of people making photo-aggregators, sitewide tagging, and making Planet sites. I can’t believe how much grief a hacky Wordpress plugin has given me over the years. Hopefully as it gets more and more out of date, this query count will start to drop from 400k (not that much) to a few hundred. Then I will smile.

Today I had the pleasure of a random guy in Mexico recursively downloading as much of my site as he could, which sent my CPU load to 2.0, a level that Dreamhost would find acceptable but which I personally freak out about. The r-dns and IP of this guy are:

dsl-189-171-15-59.prod-infinitum.com.mx
189.171.15.59

He started at 04/Nov/2007:12:04:36 and ended (by iptables ban) at 04/Nov/2007:20:17:03. In those 8 hours and thirteen minutes, he made over 250,000 requests. That’s an extra 8.5 requests per second from a single IP, which is clearly unacceptable behavior:

[root@fc624389 ~]# cat access_log | grep 189.171.15.59 | wc -l
251923

If you don’t believe me, the next biggest offender over the last 24 hours made only 4,400 requests:

[root@fc624389 ~]# cat access_log | cut -d’ ‘ -f1 | sort -n | uniq -c | sort -nr | more
251923 189.171.15.59
4403 66.249.73.116
2012 76.88.78.239
1646 70.141.105.233

The user agent of this guy doesn’t tell *me* anything about him, but maybe one of you readers has an idea?

189.171.15.59 – - [04/Nov/2007:12:04:38 -0500] “GET /wp-content/themes/greenmarinee/images/links_bullet.gif HTTP/1.1″ 200 467 “http://celebrity-photos.elliottback.com/” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Media Center PC 3.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)”

Another thing that bugs me is he requested each URL about 7 times. WTF? Do you really need to spider my site as fast as you can seven times?

[root@fc624389 ~]# cat access_log | grep 189.171.15.59 | cut -d’ ‘ -f11 | sort | uniq | wc -l
35414

I am either thinking of writing a very evil script to confuse non-google/msn/live/ask/yahoo bots by writing in an infinite number of invisible links into my websites, or installing some kind of mod_throttle into my apache. It looks like mod_limitipconn might help here, too.