| September 21, 2011 7:57 pm

For the past year and a half, I’ve been using DreamHost, a company based out of California, to host this website. In most ways, it’s been a fantastic experience. The customer service is prompt and knowledgeable [1], they go out of their way to use renewable energy, and they give you all sorts of additional goodies when you sign up for an account [2].

But even though the experience has been mostly fantastic, I have one serious gripe: the actual hosting experience is terrible.

Let me see if I can explain why. With my previous host, Brinkster, it was rare to have a website outage. Things, for the most part, Just Worked ™. The website was up and people came to read it. That’s generally all you can ask from a hosting company. (To be clear, the hosting experience with Brinkster was great, everything else was middling to awful.)

With DreamHost, though, it’s another story. If you’ve come to this site lately (especially in the mornings or late evenings) it will be unresponsive. At other times, you might see a strange message in your browser, saying “Error 500: Internal Server Error.”

As the person paying for the hosting space, this is somewhat frightening.  HTTP errors (of which 500 is one) can be cryptic, and figuring out why your website (which was working five minutes ago and costing you money) is broken problem can require some serious time.

Wait, I've Almost Got It

500 Errors in General

The 500 error, in particular, is a true terror. It means that the web server encountered an unexpected condition and can’t return the web page to you. It’s a sort of catch-all error that gets generated when nothing more detailed is available. If you want additional information (something which might actually be helpful to fix the problem), you have to work your way through cryptic error logs and pray that some divinity is listening to your pleas. It’s not fun.

When hosting with Brinkster, I never had 500 errors. Not once. Occasionally there would be an outage, but these were repaired quickly and with little fuss. WIth Dreamhost, though, 500 errors happen all the time: at least daily, and sometimes hourly. They crop up when trying to do administrative tasks, when making posts, when trying to load the home page for the website, or when uploading files. The errors can be intermittent, or they might last for hours. It’s all very frustrating.

Lately, they’ve been so bad that I had essentially convinced myself to move away from DreamHost to another provider. Something I consider a true shame. It would mean giving up the goodies [3], probably paying more money, and losing shell access to the server. So, before I decided to move away forever, I decided to get in touch with DreamHost support (again) and work out the problems with my website.

Happily, after a protracted email message thread and a great deal of searching online, I’ve I’ve learned a few things about optimizing websites for DreamHost. I’ve also found a strategy (I think) for solving the dreaded “Internal server error” problem.

The Problem

In the world of web hosts, there are several different options you can go with. If you’re rich, and powerful, or you just want to go with the best, then you will opt for a dedicated server. This means that you have a single computer all to yourself to do with as you see fit. For most people, though, a dedicated server is out of reach. It’s simply too expensive. For these people, you have to make do with a second option: shared hosting.

When you have a shared hosting account, it means that there are many websites on a single server. All of the resources (CPU, RAM, hard disk) are shared amongst every person who has an account and all of the websites the server is responsible for.

Because I’m not rich (and far too neurotic to pay for a dedicated server), I have a shared account. On that account, I have three websites. This one, Oak-Tree.us, and a website that I run for a local writing group.

And that is where the problem starts. DreamHost makes its money by putting as many people on a single machine as they can [4]. A certain number are required to pay for the machine, after that, every additional user is pure profit. The operating mantra might be, “Charge ‘em cheap, stack ‘em high, stack ‘em deep.”

There’s just one problem with that philosophy, though. For everyone’s websites to work, there have to be limitations. No single user/site can hog all of the CPU or RAM. It would bring the whole server to a standstill and be unfair to the (many) paying customers.

For this reason, DreamHost makes use of a script that shuts down any website that uses too much memory or too much CPU time. If the site crosses a certain threshold, it gets killed. It doesn’t matter what the process is, either.

It turns out that my websites were exceeding this invisible limit. WordPress, the content management system I use for my sites, can use a lot of memory.

The Content Management Sytem I run, WordPress can be very heavy on system resources.

The Solution

The solution to the problem, then, was to make WordPress use less CPU and RAM. In a word, to optimize the sites. Here’s the approach that I took.

  1. I aggressively looked at all of the plugins on my site and removed any that I wasn’t using. At the same time, I did a bit of research and removed plugins known to cause problems.
  2. By pestering customer support and carefully reading about other people’s experiences, I learned how the DreamHost memory script worked. I then made changes to how my account was configured so that I was less likely to feel its wrath.
  3. I installed a special caching plugin called “W3 Total Cache” that creates static HTML files for all of the pages in my site.
  4. I moved all images and other static files to the Amazon CloudFront content delivery network. (W3 Total Cache can help simplify this process.)
  5. I simplified my theme files in order to help them load faster. I also tweaked the templates to work with WordPress 3.0.

Taken together, I think all of the changes have got the problem under control. In the past week, I’ve noticed that the website is more responsive and page load times have improved. I’ve also had less trouble with 500 errors. Which is to say, success!

In the rest of this post, I’ll walk you through the specifics of the first two changes. The last three: installing and configuring W3 Total Cache, enabling Amazon Cloudfront, and the tweaks to the theme are all posts in their own right.

Purging Plugins

The first step I took was to purge any plugins that I wasn’t using.

WordPress plugins are fantastic things, but they’re also a bit like the pounds you put on around Christmas and Thanksgiving: easy to acquire, but unseemly and unsightly. Worse, many plugins are poorly written, which makes them bloated and fat in other ways. They can easily double or treble your use of RAM resources.

When I started, I had 35 plugins enabled on this site. Most of which were not needed, so I removed them. I’m now down to just 19, and things appear to be running much more smoothly.

While pruning my plugins, I noticed that one, in particular, was hard on the server. This was the “All in One SEO Pack” which optimizes the site for search engines. Apparently, the SEO pack is a real turd and has a huge impact on performance. DreamHost even expressly prohibits its use in shared accounts. It’s in their Terms of Service:

Customer agrees to not engage in activities pertaining to Black Hat SEO, Spamdexing, and so-called “Scraper sites.” These can all have a severely detrimental effect on server performance and are not permitted. Pursuant to this policy, the poorly-written WordPress plugin “All in One SEO Pack” is expressly prohibited on DreamHost shared hosting servers and may only be installed on DreamHost Private Servers (VPS).

Who would have known? (Do people actually read the 50 page documents companies post as TOS?) Disabling it greatly helped the performance of my sites. The effect was almost instantaneous.

Configuration Tweaks

The second action was also pretty straightforward. One of the interesting things about DreamHost is that they allow you to have multiple user accounts active under a single hosting account. This may not make much sense if you’re not a DreamHost user, but I’ll try and explain.

Most hosting companies give you a single account on a server somewhere. This computer account corresponds to your hosting account in a one-to-one fashion. (It probably even has the same log-in name.) That’s not how DreamHost works, though. Rather than limit you to a single account on the server, they allow you to have multiple computer accounts for every hosting account.

I’ve got several: one for version control, one for the websites, one for creating backups, and one for securely sharing files.

It turns out, though, that the amount of memory and CPU is limited by the computer account. Since I had just one account for all of my websites, they were all influenced by the account limit [5].

With multiple user accounts on the computer, though, each account gets its own separate allotment of memory and CPU. So, there was no reason to put all of the sites under a single user.  I made a unique user for each domain. blog.oak-tree.us got one account, oak-tree.us got another, and writing.oak-tree.us got a third. Since doing so, both the performance problems and the 500 errors appear to have dried up.

A simple change, but it’s had a big impact on the websites. I sincerely hope that the solution is permanent. (Or at least long-term.)

Conclusion

Clearly, I’m frustrated with DreamHost. Even so, I’ve been very happy with certain aspects of their service. The goodies they include with a hosting account are awesome, the access to the shell is amazing, and I appreciate the patient and understanding customer support.

Even with the troubles, I’d recommend the company to others. But that recommendation comes with a significant caveat. If you’re willing to spend time and optimize your site, they are a great solution. But you need to spend the time and effort. Otherwise, you might find yourself staring at an indecipherable error.

 ServerError500_5436e95f-b1b5-46eb-80ab-0769a875117d

Hopefully this post will give you some easy to implement ideas for the former.


______________________________________________

[1] Anyone who has suffered through a customer service interaction with a hosting company knows how rare this is. The individual on the other end of the line will either be under-qualified or socially inept. All of the competent and friendly seem to be working for DreamHost.

[2] The goodies include things such as free access to WebDAV folder shares, access to version control (SVN, Git, and Bzr), the ability to work with your website via a secure shell (SSH), CRON, an interface to Amazon CloudFront, and others. These goodies (particularly SVN) are the primary reason that I signed up with DreamHost and why I will probably stay. No one else quite matches the features/price ratio.

[3] Moving websites to another provider is a gigantic pain in the ass. Especially if you’ll be moving your database. Moving from Brinkster to here took the better part of a week and I still didn’t get all parts of the website transferred. I have no desire to go through that again.

[4] Brinkster currently advertises 1,000,000 domains (as of September 21, 2011) and 1500 machines. This means that there may be as many as 600 domains (websites) per machine.

[5] They weren’t just influencing them, they were causing the other sites to go down. Because pages wouldn’t load correctly, the server would try to diagnose the problem, and use more memory in the process. A feedback loop of computer memory death.

Comments

One Response to “Internal Server Errors (500) and DreamHost”

Flor wrote a comment on March 20, 2012

I am moving out from Dreamhost, after being a client since 2005 because of 500 errors and even a massive hack attack to all my sites, splited in 5 different users, with long long long passwords (25 letters+number in average) and totally up to date software (wordpress or gallery or joomla).

I even tried to go to VPS, but then all the sites started to give an Apache error (bad_httpd_conf). After 6 emails during the last week, the problem was still not totally fixed. So, I am FED UP with them.

I hope your problems stop, but I have the feeling that they want to get rid of clients with big sites, to not annoy their need for profit. I am moving now to InMotion… will give a try. The price is ok, they limit the number of domains one can have (what does not annoy me), so, I think this is enough “limitation”. They don´t advertise “all unlimited” and then start to kill proccess without even telling why-what-how to fix.

Care to comment?