Category Archives: Hosting

Smarthost.net: Ultimately Just a Waste of Time

Smarthost.net has some nice “storage server” deals with some very configurable options. If you want a VPS with 1TB of disk space, their offerings are pretty attractive.

For my search engine, I need hosting with a good chunk of disk space in order to hold the index. It doesn’t need to be fast storage, and it doesn’t need a lot of CPU and RAM — retrieval of index entries is fast and efficient.

This made Smarthost look pretty ideal, so I signed up and got a web server going. It worked well for about a month. So I decided to set up a second one to do some light web crawling (you don’t get enough cores on their plans to do anything heavy).

After about a week, both machines were unreachable. I contacted support and found out that the drive array had failed on the machine hosting both of them. Support tried to recover it, but ultimately it was a total failure. So the little bit of web crawling data and the search engine log data for about 2 weeks (since the last time I pulled it) was destroyed.

Annoying, but hardware failures happen.

A week later I found my crawler machine suspended because of a false positive on Spamhaus. Apparently their system is so badly-written that just visiting a domain with a web crawler can get you on a “bad list” for supposedly hosting a virus/malware. Many hosting providers, Smarthost included, will auto-suspend service for any box that gets on that list.

I got that machine removed from Spamhaus the same day and had it reactivated a few hours later to download the 200k pages or so that had been crawled, but support was pretty snarky about it. Clearly Smarthost is not a service that is compatible with what I do.

I ended up moving the web server to 1tbvps, which is slightly more expensive, but has more CPU cores and RAM, which is always nice. I moved the crawler to Digital Ocean, which is a very data-science-friendly service. We’ll see whether I have issues with those, but I suspect they will work better for my purposes.

Ultimately my 2 month experience with Smarthost ended up being a complete waste of time.

 

Blocking Spammy or Malicious IPs with Nginx

Over the years I’ve added a bunch of sites and networks to my IP block list. They’re sources of spam, hack scripts, and traffic that just wastes my time and resources.

Not all of these sites are spam or malicious at this very moment (some had viruses or were otherwise compromised as part of a botnet), it’s just what I’ve found useful to block. The names are optional. I just added comments with what the network looked up as (or why it was blocked).

Here’s what I’ve blocked. To implement this in Nginx, edit the nginx.conf and in the main block add “include blockips.conf” with this file saved as blockips.conf in the same directory.

 

# Every one of these networks has been blocked because they have
# behaved in a spammy, botlike, no-account hackscript way.
# Add this to your /etc/nginx/nginx.conf like so:
# include blockips.conf;
# (assuming it's in the same directory)
deny 27.16.0.0/12; # CHINANET Hubei province network
deny 27.148.0.0/14; # CHINANET FUJIAN PROVINCE NETWORK
deny 27.152.0.0/13; # CHINANET FUJIAN PROVINCE NETWORK
deny 27.184.0.0/13; # CHINANET hebei province network
deny 36.248.0.0/14; # China Unicom Fujian Province Network
deny 39.64.0.0/11; # China Unicom Shandong province network
deny 42.224.0.0/12; # China Unicom Henan province network
deny 46.105.127.195; # OVH SAS - For spamming searches on WbSrch.
deny 50.21.178.17; # >30k spam queries from securedservers.net
deny 50.21.178.18; # >15k spam queries from securedservers.net
deny 58.20.0.0/16; # China Unicom HuNan province network
deny 58.22.0.0/15; # China Unicom FuJian province network
deny 58.240.0.0/15; # China Unicom Jiangsu province network
deny 58.248.0.0/13; # China Unicom Guangdong province network
deny 59.56.0.0/14; # CHINANET fujian province network
deny 59.60.0.0/15; # CHINANET fujian province network
deny 60.0.0.0/13; # China Unicom Hebei Province Network
deny 60.10.0.0/16; # China Unicom Hebei Province Network
deny 60.11.0.0/16; # China Unicom Heilongjiang Province Network
deny 60.12.0.0/16; # China Unicom Zhejiang province network
deny 60.168.0.0/13; # CHINANET anhui province network
deny 60.176.0.0/12; # CHINANET Zhejiang province network
deny 60.195.248.0/21; # Beijing Bitone United Networks Technology Service Co., Ltd.
deny 60.220.0.0/14; # China Unicom Shanxi Province Network
deny 60.252.192.0/19; # Daqing Zhongji Petroleum Communication
deny 61.140.0.0/14; # CHINANET Guangdong province network
deny 61.144.0.0/15; # CHINANET Guangdong province network
deny 61.147.123.0/24; # CHINANET jiangsu province network
deny 61.235.0.0/16; # China TieTong Telecommunications Corporation
deny 101.16.0.0/12; # China Unicom Hebei province network
deny 101.64.0.0/13; # UNICOM ZheJiang Province Network
deny 101.224.0.0/13; # CHINANET SHANGHAI PROVINCE NETWORK
deny 104.238.93.98; # GoDaddy site excitesubmit.com/excitesubmit.net/excitesubmit.org spamming domain searches (perhaps acting like they're a site submit service)
deny 110.80.0.0/13; # CHINANET FUJIAN PROVINCE NETWORK
deny 110.88.0.0/14; # CHINANET FUJIAN PROVINCE NETWORK
deny 110.152.0.0/14; # CHINANET xinjiang province network
deny 110.176.0.0/13; # CHINANET SHANXI PROVINCE NETWORK
deny 110.228.0.0/14; # China Unicom Hebei province network
deny 110.240.0.0/12; # China Unicom Hebei province network
deny 111.0.0.0/10; # China Mobile Communications Corporation
deny 112.0.0.0/10; # China Mobile Communications Corporation
deny 112.111.0.0/16; # China Unicom Fujian Province Network
deny 111.128.0.0/11; # North Star Information Hi.tech Ltd. Co. (China)
deny 112.224.0.0/11; # China Unicom Shandong province networ
deny 113.64.0.0/11; # CHINANET Guangdong province network
deny 113.96.0.0/12; # CHINANET Guangdong province network
deny 113.240.0.0/13; # CHINANET HUNAN PROVINCE NETWORK
deny 115.48.0.0/12; # China Unicom Henan province network
deny 114.80.0.0/16; # CHINANET SHANGHAI PROVINCE NETWORK
deny 115.192.0.0/11; # CHINANET Zhejiang province network
deny 115.224.0.0/12; # CHINANET Zhejiang province network
deny 166.62.101.30; # >50k Spam queries from secureserver.net.
deny 117.24.0.0/13; # CHINANET Fujian province network
deny 117.32.0.0/13; # CHINANET Shanxi(SN) province network
deny 117.128.0.0/10; # China Mobile Communications Corporation
deny 118.186.0.0/15; # Building D, No.2 Shangdi Xinxi Road Pioneering Park,
deny 118.213.0.0/16; # CHINANET Qinghai Province Network
deny 119.4.0.0/14; # China Unicom SiChuan province network
deny 119.39.0.0/16; # China Unicom HuNan province network
deny 119.48.0.0/13; # China Unicom Jilin province network
deny 119.233.128.0/17; # Xiamen Broadcasting & TV Network Transmit Co.Ltd
deny 119.248.0.0/14; # China Unicom Heibei Province Network
deny 120.0.0.0/12; # China Unicom Heibei Province Network
deny 120.32.0.0/13; # CHINANET FUJIAN PROVINCE NETWORK
deny 120.40.0.0/14; # CHINANET FUJIAN PROVINCE NETWORK
deny 120.192.0.0/10; # China Mobile Communications Corporation
deny 121.204.0.0/14; # CHINANET Fujian province network
deny 121.224.0.0/12; # CHINANET jiangsu province network
deny 123.4.0.0/14; # China Unicom Henan province network
deny 123.128.0.0/13; # China Unicom Shandong Province Network
deny 123.168.0.0/14; # CHINANET SHANDONG PROVINCE NETWORK
deny 124.128.0.0/13; # China Unicom Shandong province network
deny 125.88.0.0/13; # CHINANET Guangdong province network
deny 153.3.0.0/16; # China Unicom Jiangsu province network
deny 175.0.0.0/12; # CHINANET HUNAN PROVINCE NETWORK
deny 175.42.0.0/15; # China Unicom Fujian Province Network
deny 175.44.0.0/16; # China Unicom Fujian Province Network
deny 180.76.0.0/16; # Beijing Baidu Netcom Science and Technology Co., Ltd.
deny 180.96.0.0/11; # Chinanet Jiangsu Province Network
deny 182.128.0.0/12; # CHINANET Sichuan province network
deny 183.64.0.0/13; # CHINANET Chongqing Province Network
deny 183.128.0.0/11; # CHINANET Zhejiang province network
deny 183.192.0.0/10; # China Mobile Communications Corporation
deny 202.228.204.204; # server.hiraishoten.net
deny 210.73.208.0/20; # Beijing Blue Ocean information technology co.LTD
deny 218.6.0.0/17; # CHINANET fujian province network
deny 218.56.0.0/15; # China Unicom Shandong province network
deny 218.80.0.0/16; # CHINANET Shanghai province network
deny 218.204.0.0/14; # China Mobile Communications Corporation
deny 219.128.0.0/13; # CHINANET Guangdong province network
deny 220.160.0.0/15; # CHINANET Fujian province network
deny 220.168.0.0/15; # CHINANET Hunan province network
deny 221.0.0.0/15; # China Unicom Shandong province network
deny 221.10.0.0/16; # China Unicom SiChuan province network
deny 221.224.0.0/13; # CHINANET jiangsu province network
deny 222.76.0.0/14; # CHINANET fujian province network
deny 222.136.0.0/13; # China Unicom Henan province network
deny 222.176.0.0/13; # CHINANET Chongqing province network
deny 222.240.0.0/13; # CHINANET Hunan province network
deny 223.8.0.0/13; # CHINANET SHANXI PROVINCE NETWORK
deny 223.64.0.0/11; # China Mobile Communications Corporation
deny 222.84.0.0/16; # CHINANET Guangxi province network

Server Updated

Over the course of a lunch hour yesterday I updated the operating system on the Basternae box.  It was running Ubuntu 10.04.  It’s now on Ubuntu 12.04.  It took about 45 minutes to finish the whole process including testing and fixes.  Only one thing broke in the process.  The Django web framework, which runs about half of my websites, was uninstalled by the upgrade process.  I had to manually reinstall it, which took about 10 minutes.

I remember upgrading a Red Hat 5 box to Red Hat 9 many years back.  The process took about 10 hours to finish and get everything working again.  Easy upgrades are a good thing.

Startup Script Created

For the past few days, the test version of the MUD has only been up when I was actively running it from my Linux shell account.

I’ve created a startup/restart script so it should be running all the time now, barring any catastrophic crash-explosions.

Telnet basternae.org port 4502 to connect.

Resource Usage For Basternae 3

Here’s the resource usage for Basternae running on Mono on Ubuntu with 1 active user and 6 zones loaded:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     24624  0.2  6.9  39284 25548 pts/0    SNl  21:40   0:08 mono Basternae.exe

I have to imagine with a full complement of zones and a few active players that the memory usage will reach 64MB VSZ and 48MB RSS with probably about 1.2% CPU usage.  This would be well within the abilities of the current hosting and would use about 15% of the overall available resources.

What I am a little afraid of is memory leakage.  Contrary to popular folklore, running C# does not completely eliminate memory leakage, it just makes it easier to avoid.  The nice thing is that I’ve designed all of the game objects with instance counters, so at any time I can get a count of how many rooms, objects, mobiles, etc. exist in memory at any given time, whether they’re actually active in the game or not.  This will make it easier to track down leaks if any happen.

Worst case is if resource usage expands drastically we may have to upgrade from the Linode 360 to the Linode 540.  In that case, running Basternae would cost a total of $120 per year.  I’ve been spending that much on lunch every two weeks (I really need to cut down, but I work next to all of the best restaurants in my city), so it’s not exactly a huge number in the grand scheme of things.

It should also be mentioned that on Windows, memory usage is only 14MB.  Running on Linux/Mono requires a lot more resources.  Part of this is because some of the things used by .NET are automatically loaded by the Windows OS so they don’t contribute to memory size, and part of it is because in a way Mono is actually a “Windows software emulator” and is less optimal than something like a C executable compiled to run natively.

22 Hours Uptime

Mind you, only about 5 players have logged into the Magma 3.04 demo site, but it’s been up 22 hours so far.

I remember back when Magma was first being developed. An hour of uptime was considered a good thing.

Funny how things progress. I remember my first MUD in 1999, Illustrium Arcana, running on a Cyrix 6×86-PR200 on a 32MB machine co-located on a friend’s ISP (stax.net, apparently shut down around the end of 2006). Having T1 access was so amazing ten years ago. Now I’d be angry if my cable modem slowed down to 1.5 MBPS.

The original Cyrix machine ran at about 200 BogoMIPS, which was screaming fast. Right now I have what is effectively 1/10 of a Xeon 2.5 GHz processor. That gives me 500 BogoMIPS and 320 MB of RAM. That’s more than enough to run a MUD and a few websites.

Moved To A New Host

Basternae.org has moved to a new host.  We are no longer on Hostmonster and are now on Linode.

Why Move?

Hostmonster is a great host and you get a *LOT* for the money considering that it’s less than $10 per month if you prepay. I’ve just simply outgrown it. I how have ~30 domains, ~30 databases, 6GB of data, and run all manner of things — WordPress, Drupal, Django, and custom Python and PHP scripts.

Hostmonster can almost support all of that. If I had 20 domains or less I *almost* wouldn’t have any trouble. However, here are the limitations that I’ve outgrown:

1. 32MB PHP memory limit. This is a wall that larger Drupal sites will hit regularly.
2. If you’re doing something they don’t like, your account can be suspended instantly. This happened to me once when some Chinese bot was spamming a database script and was performing what they called “long-running queries”. A call to tech support straightened it out (I had to delete a few files, but wasn’t using them anyway).
3. Support for Python and Django is pretty sparse. Since I’ve consumed the Python kool-aid, this has suddenly become important.
4. Lack of customization ability. Since it’s shared hosting, they don’t want me possibly screwing it up for other people.
5. No support for long-running processes. I couldn’t run a MUD on it.

These are things that most mere mortals wouldn’t run into, and if you’re just starting off with web design/development, I do recommend starting with Hostmonster. Everyone eventually outgrows shared hosting. It took me three years.

The New Host

Why would I choose Linode? I’ve mentioned in previous posts that I like Slicehost, and Linode is almost exactly the same thing — Linux-based Xen VPS (Virtual Private Server) hosting.

What pushed me over the top is that for the basic $20 per month plan I get 320MB of RAM at Linode instead of 256MB I would get at Slicehost *and* Linode allows you to run a 32-bit OS, while Slicehost is 64-bit only. That’s not just 64MB more memory, it’s more like 128MB more according to statistics I’ve seen on memory use for 64-bit Ubuntu.

I am extremely RAM-hungry. I eat it like candy.

Setting Up The New Server

Friday evening I signed up for Linode. I had an Ubuntu 9.04 VPS up and running in 10 minutes. There was nothing to do for basic configuration — network, DNS, ssh, it was all good out of the box. It was a naked server with pretty much nothing on it (not even a web server), but that was exactly what I wanted since I was going to configure the machine from scratch.

Knowing what a memory hog the Apache web server is, I went something faster and more lightweight — nginx (“Engine X”). Nginx isn’t very well-documented if you don’t speak Russian, so I had to spend some time trolling blogs and forums to get the info needed to set up PHP and FastCGI and a few dozen virtual hosts.

This is also the first time I’ve tried using Drupal multi-site. It’s been such a hassle updating Drupal modules on every site each time there’s an update or security fix, so I wanted to use one core codebase. It was far easier to set up than I had ever imagined (just add a separate directory in your sites folder for each domain).

I had to install a boatload of modules, move a ton of content, set up scads of virtual host files, edit piles of configuration files, configure and export/import dozens of databases, and all of that other craziness involved with setting up a web server from scratch. This took part of Friday, all of Saturday, and half of Sunday.

In the process I also upgraded a few Drupal installs from v5 to v6. It’s not a very pretty process if you have complex custom views.

The most annoying part of the update was getting the nginx rewrite rule for WordPress set up so that the CSS file would actually load. For a few hours this site was just plain text on a white background.

I still have a few sites to move (the *big* ones) and a few kinks to work out, but I’m already pretty happy with the server — it is much more snappy and responsive, and I can edit sites faster thanks to quicker page loads.

If you’ve outgrown shared hosting and know Linux well, I recommend moving to Linode.

Mass Refactoring

In the interest of making sure the MUD can be modified easily well into the future I’ve broken the heck out of the codebase.

Yep.

I’ve been refactoring, rearranging, renaming, and restructuring things so they make a lot more sense. That has inevitably broken a few things, but it will be far easier to add new types, flags, and enumerations in the future — things like adding a new terrain type without breaking existing terrain types or having to recompile the editor and re-convert all of the zones.

Believe me, we had quite a headache with zone file format changes on Basternae 2 and I don’t want to repeat it.

This means that it’ll take a few days before the codebase will build again and I’ll have some work to do on the converter and the editor, but it’ll be worth it.

I’m still looking at putting up a test server in May and so far Slicehost is the most likely candidate. Feel free to offer suggestions. Since it’s out-of-pocket and non-income-producing, my budget isn’t any more than $20 per month (should I be accepting donations for this?)

Got The .NET

Not much to report today, but I picked up the basternae.net domain.  Now the full trilogy points here (.com, .net, and .org).  Yippee!

Got The Dotcom

It seems that the domain Basternae.com recently became available.  I bought it, so both basternae.org and basternae.com point here.  Yippee!

A Likely Host

After spending a while shopping around it looks like I’ve found the place to put a Basternae server when the time comes for on-line testing: Slicehost (http://www.slicehost.com)

Their virtual private server (VPS) hosting starts at $20 a month and is easy to upgrade. I don’t want to spend much since MUDs typically don’t generate any income whatsoever, and Slicehost ought to be a relatively painless way to go.

The head web application programmer here at work uses them and only has good things to say.

Server Options

I know it’s a little premature at this point, but I’ve started to think a bit about server options. There are a lot of different things I could do, with various cost/reliability/control considerations.

The primary decision is broken into one of four options:

1. Host it from home.
This would require configuring dynamic DNS and would probably be a slow connection. I have 512Kbps upstream, so that wouldn’t be too slow unless a significant number of players connected. What it would save in hosting costs it would certainly eat up in electricity costs. I’m not sure the exact amount, but leaving a computer on 24 hours a day costs me somewhere betwen 25 and 40 dollars per month in electricity. All in all a generally bad idea.

2. Sign up for co-location or a dedicated server.
An expensive route, but there is no shortage of control. At somewhere like aplus.com I could get a nice dedicated server for $99/month and a passable (celeron 1.7GHz w/512MB) server for $49 per month. I would be able to host all of my other domains and webspace all in one place. Other services offer similar packages, but pretty much none start under $50. This would probably be overkill for the web presence I have.

3. Try to get “free” hosting.
I might be able to find someone I know with server space available or find someone willing to host the MUD. This would be a good idea right until the point where whoever was hosting got sick of doing so. Too dangerous considering you get what you pay for.

4. Sign up for shared MUD hosting. This is probably the most sensible route for now. Here’s a comparison of what can be had at the various hosting services for $20 per month:

MudMagic: 300MB disk, 50MB RAM, 4% CPU, unlimited connections.
GenesisMUDs: 400MB disk, 55MB RAM, 15% CPU, unlimited connections. [actually $19.50/month]
Silverden: 100MBdisk, 20MB RAM, (CPU not listed), unlimited connections.
MUDDrake: 500MB disk, 48MB RAM, (CPU not listed), unlimited connections.
Dune Internet: 500MB disk, 60MB RAM, (CPU not listed), unlimited connections. [actually $16.00/month]
InfoLaunch: 600MB disk, 45MB RAM, 10% CPU, unlimited connections.
Mu-Host.com: 500MB disk, 75MB RAM, (CPU not listed), connections not listed.

Silverden is offering the most ridiculous of packages, woefully inadequate to any sort of task, and since they are offering a Pentium 200 dedicated server I’ll have to assume that they’ve gone out of business and just hadn’t bothered to take their site down.

Dune internet and Mu-Host look the best, especially since the major consideration has always been RAM more than processor in any MUD I’ve worked on. Right now, with only a couple zones connected, the server uses 18 megabytes of RAM. This is the debugger-profiler version, so it’s a bit more bloated, but it’s not unrealistic to expect to be using about 40-48 megs when fully assembled.

The irritating thing, though, is that most of the companies that list CPU percentage don’t bother to list the TYPE of CPU you’re getting a percentage of. For all I know MUDMagic could be offering 4% of a Pentium 200 while GenesisMUDs is offering 15% of Pentium Core Duo e6700. All CPUs are not the same.

If I were to pick a service I would probably go with Dune.