David Ramsden – Network engineer, general geek, petrol + drum and bass head

Disabling WordPress XML-RPC and banning offenders with fail2ban

This isn't something new. SANS ISC reported on this 2 years ago. The bad guys love anything that can be used in a reflection DoS and the WordPress XML-RPC functionality is a prime candidate. There are various ways to disable it, through WordPress plugins for example, or by hacking away at code. All of these are fine if you're in control over what gets installed on the web server. In a shared hosting environment you've got to rely on your users.

Running Apache you can disable XML-RPC globally and simply with the following:

The configuration should be placed as part of the global Apache configuration. When any file called xmlrpc.php is requested, on any vhost, from an IP address not listed by the Require ip line, an Error 403 Forbidden will be served instead. This configuration should ensure that WordPress plugins like Jetpack continue to work.

I've seen a few examples where even after doing this the bad guys still continuously request xmlrpc.php even though they're being served a 403 error. To further protect the web server fail2ban can be deployed.

Firstly create a filter definition:

Then create the jail:

Now when someone requests xmlrpc.php 3 times within the defined findtime their IP address will be blocked.


Banning repeat offenders with fail2ban

More and more I see fail2ban banning the same hosts repeatedly. One way to tackle this could be to increase the ban time but you could also have fail2ban monitor itself to find "repeat offenders" and then ban them for an extended period of time.

Firstly, create a filter definition:

This will be used against the fail2ban log and will find any hosts that have been unbanned. We don't want to monitor hosts that have been banned because, er, they're already banned. We also want to ignore any log entries that are generated by the jail itself.

Next edit jail.local to add a new jail:

This jail will monitor the /var/log/fail2ban.log file and use the repeat-offender filter that was defined earlier. If 3 unban's are seen within 5 hours, the host will be banned for 48 hours. You could adjust the banaction to use the route action which may give some performance benefits on a very busy server.


loadbalancer.org – Linux feedback agent

I've been working with some loadbalancer.org appliances recently, load balancing traffic over MySQL and Apache servers running Linux. The load balancer supports a feedback agent where it can query the real server to gauge how utilised it is based on, for example, CPU load and then distribute the request to the real server that should perform the best.

Over on the loadbalancer.org blog is an article about the feedback agent and how to implement it for Linux servers. The shell script suggested is:

Although this does give you a 1 second average CPU, it's a bit too accurate and doesn't give much headroom. If the CPU suddenly spiked very quickly and then returned to normal the load balancer wouldn't know this. And indeed watching the real server weighting change on the load balancer v's what top is reporting confirms this. The weighting on a real server can drastically jump up and down.

A better feedback agent script is:

This will take a 3 second average reading and then report back the overall average. This prevents the real server weighting on the load balancer from fluctuating so much. Change the NUM_CHECKS variable to take more or less readings as required.


Root shell on DrayTek AP 800

The DrayTek AP 800 is a 2.4Ghz 802.11n Access Point with the ability to make it dual band, 2.4Ghz and 5Ghz, with an optional USB dongle. It supports multi-SSID with VLAN tagging, built in RADIUS server, per-SSID/station bandwidth control and can act as a bridge, repeater etc.

As with all of these SOHO products it'd built on Linux. Which means somewhere there is a root shell lurking.

The DrayTek AP 800 has telnet enabled out of the box. Establish a telnet connection and login as the admin user. You'll be dropped in to a restricted busybox shell. To make it slightly less restrictive type rddebug. This will let you use commands such as ps and echo.

Now spawn telnetd on a different port and invoke a full shell, with:

Telnet to the AP on port 2323 to be dropped in to a root shell.

This will likely also work with the AP 900 and the 2860 series. Leave a comment if you've tried it.

Tagged as: , , No Comments

Custom kernel on a DigitalOcean droplet – the right way

A few days ago I decided to create a VPS, known as a "droplet", with DigitalOcean. They claim a deployment time of 55 seconds. And 55 seconds after hitting the button I had a Debian 7 x64 droplet running. The plan was to migrate my current VPS to this DigitalOcean droplet. The first task I always undertake with any Linux deployment is to create a custom stripped down kernel patched with grsecurity. However, unknown to me, the way DigitalOcean boot your droplet with KVM means that you can only use a kernel of their choice.

But don't panic because there is a workaround by utilising kexec. kexec is used to speed up reboots. When you power on any machine it goes through a POST procedure to initialise the hardware. When you reboot a machine it goes through this POST procedure again which really isn't needed if the hardware is already initialised. Normally kexec is invoked during the reboot runlevel to load a kernel in to memory and jump straight in to it, bypassing the hardware initialisation. The trick is to reverse this and have the machine utilise kexec during the startup runlevel, therefore jumping in to the custom kernel straight away and only utilising DigitalOcean's choice of kernel as a bootstrap.

I've seen a lot of hacks where people suggest modifying the init scripts such as /etc/init.d/rcS to kexec the custom kernel on boot before any init scripts are executed. But this is very hacky and you could easily end up in a situation where the custom kernel doesn't boot but because there's no way to stop kexec running you could have a completely unusable droplet.

In my opinion the correct way to do this is to write a proper LSB init script. Note that the below is specific to Debian and Ubuntu. The init script should also give you the option to abort the kexec process in case something goes wrong. At least that way the droplet will still boot with DigitalOcean's kernel and you can get in and fix things.

The LSB init script is as follows:

The above should be placed in /etc/init.d/droplet-kernel and chmod 755.

In addition you need /etc/default/droplet-kernel:

Now use update-rc.d to install the init script:

Now when the droplet boots, one of the first thing it does is look at running kexec to load and boot a custom kernel. The /etc/defaults/droplet-kernel file contains all the customisable options, such as easily enabling/disabling this process, where the custom kernel and initrd images are and the option to override the kernel arguments (such as rootfs, verbosity etc).

The script runs interactive and will hold the boot routine for 10 seconds, giving you the option to press Ctrl+C to abort the kexec process. This is especially useful if something has gone wrong with the kernel that will be kexec'd. Pressing Ctrl+C will mean the droplet continues to boot using DigitalOcean's kernel and you should be able to fix things up.

You'll notice that when the droplet is kexec'd, it appends "kexeced" to the kernel cmdline argument. It does this to prevent any loops from occurring. The init script checks if "kexeced" is the last argument and if it is won't try to kexec the kernel. Otherwise the droplet would get in to an endless reboot.

When the droplet is rebooted (runlevel 6), it removes the "kexeced" argument so that we get the option to Ctrl+C during the bootup again. This avoids the need to shutdown the droplet and power it on again for kexec to kick in.

Here it is in action:


Virtual Hosting With mod_proxy

The other day I had someone ask if there's a nice solution to the following problem:

Multiple web development virtual machines but only one external IP address.

The quick solution is to port forward on different ports to each virtual machine. For example 81 goes to VM1, 82 goes to VM2, 83 goes to VM3 etc. Which granted, would work, but isn't a "neat" solution.

Using mod_proxy under Apache is a much better solution to this problem.

Deploy a "front-end" server running Apache and mod_proxy. Create a virtual host for each virtual server and then using mod_proxy, reverse proxy to the virtual server. Port forward from the WAN to your front-end Apache server running mod_proxy.

Here's what an example config would look like on the front-end Apache server:

Requests for cust1.dev.domain.com would be reverse proxied to and requests for cust2.dev.domain.com would be reverse proxied to All with one external IP address and one port forward rule.

Just one of the many uses of mod_proxy. You can also use it for SSL bridging and SSL offloading. Neat!

Tagged as: , 1 Comment

Creating a HA iSCSI Target Using Linux

Some time ago I created a High Availability iSCSI target using Ubuntu Linux, iscsi-target, DRBD and heartbeat. The HA cluster consisted of two nodes and the iSCSI initiators were Windows Server 2008. I was able to mount the LUN and copy a video to it, play it back and then pull the power from the primary iSCSI target. A few seconds later the second iSCSI target took over and video continued to play.

Pretty cool, huh?

Here is my guide if you want to try this. Although I've not gone back through the guide to make sure it's correct. But if you spot anything that's wrong or not very clear, please leave a comment.

Tagged as: , , 8 Comments

Nitter 2.0.1 Available – Fixes DMs

A very quick (and rare) update on my blog!

Since 25th May 2011, my Nitter script has been broke due to a change with the Twitter API. OK, so the problem was actually Net::Twitter::Lite so the changes to my script have been minimal as I've switched over to Net::Twitter (3.18001), which supports the new way of requesting friend and followers IDs.

You can grab Nitter 2.0.1 from the usual place.

Happy New Year! Hopefully you won't update Nitter and then be bombarded with alerts from your NOC over the holidays.

Tagged as: , No Comments

Catching Up With An Old Friend, Ubuntu.

Since getting my iPhone some 18 months ago, I hardly turn on my desktop PC. I can do almost anything that I need to do on my iPhone. As a result my desktop PC had fallen in to a state of ruin. Last night I decided to try to tidy it up a little.

My desktop PC started out with Ubuntu 8.04 and I've upgraded each time a new release came out. As a result it had accumulated a lot of crap throughout the years. So I removed X and a lot of the CLI bits and bobs that I'd installed. Stripped it back to as much of a bare metal install as possible. Then used tasksel to install Ubuntu desktop and went from there.

It's all back up and running and I'm quite impressed with Ubuntu 10.04. I can't talk for other distributions but Ubuntu has made massive steps in the right direction over the years. I can now plug in my iPhone 4 and Rhythmbox will pop up and allow me to play my iTunes library. And the Gwibber social client is a great replacement for TweetDeck.

It's these simple things that will appeal to your average desktop user. Great work Ubuntu!

Tagged as: No Comments