David Ramsden – Network engineer, general geek, petrol + drum and bass head
27Aug/14

Custom kernel on a DigitalOcean droplet – the right way

A few days ago I decided to create a VPS, known as a "droplet", with DigitalOcean. They claim a deployment time of 55 seconds. And 55 seconds after hitting the button I had a Debian 7 x64 droplet running. The plan was to migrate my current VPS to this DigitalOcean droplet. The first task I always undertake with any Linux deployment is to create a custom stripped down kernel patched with grsecurity. However, unknown to me, the way DigitalOcean boot your droplet with KVM means that you can only use a kernel of their choice.

But don't panic because there is a workaround by utilising kexec. kexec is used to speed up reboots. When you power on any machine it goes through a POST procedure to initialise the hardware. When you reboot a machine it goes through this POST procedure again which really isn't needed if the hardware is already initialised. Normally kexec is invoked during the reboot runlevel to load a kernel in to memory and jump straight in to it, bypassing the hardware initialisation. The trick is to reverse this and have the machine utilise kexec during the startup runlevel, therefore jumping in to the custom kernel straight away and only utilising DigitalOcean's choice of kernel as a bootstrap.

I've seen a lot of hacks where people suggest modifying the init scripts such as /etc/init.d/rcS to kexec the custom kernel on boot before any init scripts are executed. But this is very hacky and you could easily end up in a situation where the custom kernel doesn't boot but because there's no way to stop kexec running you could have a completely unusable droplet.

In my opinion the correct way to do this is to write a proper LSB init script. Note that the below is specific to Debian and Ubuntu. The init script should also give you the option to abort the kexec process in case something goes wrong. At least that way the droplet will still boot with DigitalOcean's kernel and you can get in and fix things.

The LSB init script is as follows:

The above should be placed in /etc/init.d/droplet-kernel and chmod 755.

In addition you need /etc/default/droplet-kernel:

Now use update-rc.d to install the init script:

Now when the droplet boots, one of the first thing it does is look at running kexec to load and boot a custom kernel. The /etc/defaults/droplet-kernel file contains all the customisable options, such as easily enabling/disabling this process, where the custom kernel and initrd images are and the option to override the kernel arguments (such as rootfs, verbosity etc).

The script runs interactive and will hold the boot routine for 10 seconds, giving you the option to press Ctrl+C to abort the kexec process. This is especially useful if something has gone wrong with the kernel that will be kexec'd. Pressing Ctrl+C will mean the droplet continues to boot using DigitalOcean's kernel and you should be able to fix things up.

You'll notice that when the droplet is kexec'd, it appends "kexeced" to the kernel cmdline argument. It does this to prevent any loops from occurring. The init script checks if "kexeced" is the last argument and if it is won't try to kexec the kernel. Otherwise the droplet would get in to an endless reboot.

When the droplet is rebooted (runlevel 6), it removes the "kexeced" argument so that we get the option to Ctrl+C during the bootup again. This avoids the need to shutdown the droplet and power it on again for kexec to kick in.

Here it is in action:

2Feb/12

Virtual Hosting With mod_proxy

The other day I had someone ask if there's a nice solution to the following problem:

Multiple web development virtual machines but only one external IP address.

The quick solution is to port forward on different ports to each virtual machine. For example 81 goes to VM1, 82 goes to VM2, 83 goes to VM3 etc. Which granted, would work, but isn't a "neat" solution.

Using mod_proxy under Apache is a much better solution to this problem.

Deploy a "front-end" server running Apache and mod_proxy. Create a virtual host for each virtual server and then using mod_proxy, reverse proxy to the virtual server. Port forward from the WAN to your front-end Apache server running mod_proxy.

Here's what an example config would look like on the front-end Apache server:

Requests for cust1.dev.domain.com would be reverse proxied to 192.168.0.100 and requests for cust2.dev.domain.com would be reverse proxied to 192.168.0.101. All with one external IP address and one port forward rule.

Just one of the many uses of mod_proxy. You can also use it for SSL bridging and SSL offloading. Neat!

Tagged as: , 1 Comment
2Jan/12

Creating a HA iSCSI Target Using Linux

Some time ago I created a High Availability iSCSI target using Ubuntu Linux, iscsi-target, DRBD and heartbeat. The HA cluster consisted of two nodes and the iSCSI initiators were Windows Server 2008. I was able to mount the LUN and copy a video to it, play it back and then pull the power from the primary iSCSI target. A few seconds later the second iSCSI target took over and video continued to play.

Pretty cool, huh?

Here is my guide if you want to try this. Although I've not gone back through the guide to make sure it's correct. But if you spot anything that's wrong or not very clear, please leave a comment.

Tagged as: , , 8 Comments
30Dec/11

Nitter 2.0.1 Available – Fixes DMs

A very quick (and rare) update on my blog!

Since 25th May 2011, my Nitter script has been broke due to a change with the Twitter API. OK, so the problem was actually Net::Twitter::Lite so the changes to my script have been minimal as I've switched over to Net::Twitter (3.18001), which supports the new way of requesting friend and followers IDs.

You can grab Nitter 2.0.1 from the usual place.

Happy New Year! Hopefully you won't update Nitter and then be bombarded with alerts from your NOC over the holidays.

Tagged as: , No Comments
14Aug/10

Catching Up With An Old Friend, Ubuntu.

Since getting my iPhone some 18 months ago, I hardly turn on my desktop PC. I can do almost anything that I need to do on my iPhone. As a result my desktop PC had fallen in to a state of ruin. Last night I decided to try to tidy it up a little.

My desktop PC started out with Ubuntu 8.04 and I've upgraded each time a new release came out. As a result it had accumulated a lot of crap throughout the years. So I removed X and a lot of the CLI bits and bobs that I'd installed. Stripped it back to as much of a bare metal install as possible. Then used tasksel to install Ubuntu desktop and went from there.

It's all back up and running and I'm quite impressed with Ubuntu 10.04. I can't talk for other distributions but Ubuntu has made massive steps in the right direction over the years. I can now plug in my iPhone 4 and Rhythmbox will pop up and allow me to play my iTunes library. And the Gwibber social client is a great replacement for TweetDeck.

It's these simple things that will appeal to your average desktop user. Great work Ubuntu!

Tagged as: No Comments