David Ramsden

Custom kernel on a DigitalOcean droplet – the right way

A few days ago I decided to create a VPS, known as a "droplet", with DigitalOcean. They claim a deployment time of 55 seconds. And 55 seconds after hitting the button I had a Debian 7 x64 droplet running. The plan was to migrate my current VPS to this DigitalOcean droplet. The first task I always undertake with any Linux deployment is to create a custom stripped down kernel patched with grsecurity. However, unknown to me, the way DigitalOcean boot your droplet with KVM means that you can only use a kernel of their choice.

But don't panic because there is a workaround by utilising kexec. kexec is used to speed up reboots. When you power on any machine it goes through a POST procedure to initialise the hardware. When you reboot a machine it goes through this POST procedure again which really isn't needed if the hardware is already initialised. Normally kexec is invoked during the reboot runlevel to load a kernel in to memory and jump straight in to it, bypassing the hardware initialisation. The trick is to reverse this and have the machine utilise kexec during the startup runlevel, therefore jumping in to the custom kernel straight away and only utilising DigitalOcean's choice of kernel as a bootstrap.

I've seen a lot of hacks where people suggest modifying the init scripts such as /etc/init.d/rcS to kexec the custom kernel on boot before any init scripts are executed. But this is very hacky and you could easily end up in a situation where the custom kernel doesn't boot but because there's no way to stop kexec running you could have a completely unusable droplet.

In my opinion the correct way to do this is to write a proper LSB init script. Note that the below is specific to Debian and Ubuntu. The init script should also give you the option to abort the kexec process in case something goes wrong. At least that way the droplet will still boot with DigitalOcean's kernel and you can get in and fix things.

The LSB init script is as follows:

The above should be placed in /etc/init.d/droplet-kernel and chmod 755.

In addition you need /etc/default/droplet-kernel:

Now use update-rc.d to install the init script:

Now when the droplet boots, one of the first thing it does is look at running kexec to load and boot a custom kernel. The /etc/defaults/droplet-kernel file contains all the customisable options, such as easily enabling/disabling this process, where the custom kernel and initrd images are and the option to override the kernel arguments (such as rootfs, verbosity etc).

The script runs interactive and will hold the boot routine for 10 seconds, giving you the option to press Ctrl+C to abort the kexec process. This is especially useful if something has gone wrong with the kernel that will be kexec'd. Pressing Ctrl+C will mean the droplet continues to boot using DigitalOcean's kernel and you should be able to fix things up.

You'll notice that when the droplet is kexec'd, it appends "kexeced" to the kernel cmdline argument. It does this to prevent any loops from occurring. The init script checks if "kexeced" is the last argument and if it is won't try to kexec the kernel. Otherwise the droplet would get in to an endless reboot.

When the droplet is rebooted (runlevel 6), it removes the "kexeced" argument so that we get the option to Ctrl+C during the bootup again. This avoids the need to shutdown the droplet and power it on again for kexec to kick in.

Here it is in action: