Linux,  Work

DHCPd failover

Last week, I set up two dhcpd servers in a fail-over configuration. The goal is that when one DHCP server goes down, the other one takes over so that clients don’t lose their network connection. I read different tutorials on the web, such as this one of a fellow blogger and this documentation published by IBM.

First of all, you need two machines with exactly the same version of ISC dhcpd. Failover is unsupported when the two systems are running a different dhcpd version, as the implementation may have been changed.

On the primary dhcpd, you put this in dhcpd.conf:

failover peer "dhcp" {
    primary;
    address 123.456.789.001; # address of this server
    port 519;
    peer address 123.456.789.002; # address of the secondary dhcpd
    peer port 519;
   max-response-delay 60;
   max-unacked-updates 10;
   mclt 600;
   split 128;
   load balance max seconds 3;
}

key primaryhost {
    algorithm hmac-md5;
    secret "secret";
};

omapi-key primaryhost;
omapi-port 7911;

include "/etc/dhcp/dhcpd.master";

You have to change the address of the primary and the secondary dhcpd in the configuration file. In this example, both the primary and the secondary dhcpd are listening on port 519, so you will have to open this port in the firewall on both systems so that the two dhcp daemons can contact each other. A split value of 128 means that both the primary and secondary server will manage half of the available IP addresses. By setting a higher number (max 255), the primary server will manage more addresses, by lowering it, the secondary will take care of a bigger part. With this option, you can distribute the load over the two servers. mclt is the duration of leases in seconds that either of the DHCP servers will offer to the peer’s DHCP clients when the peer is down. max-response-delay defines the number of seconds after which a dhcpd will consider its peer as dead if it does not get any reply; load balance max seconds is the time which a load balancing is disabled: if a dhcpd fails to respond within this time to a DHCPDISCOVER packet, the peer dhcpd will respond in its place.

Now you need to create a key for connecting to OMAPI, which is dhcpd’s interface which can be used to send it commands at run-time. You will need the dnssec-keygen command, part of the bind9utils package in Debian. To create the key, first cd to the /etc/dhcp/ directory, and run
# dnssec-keygen -a HMAC-MD5 -b 512 -n HOST primaryhost

“primaryhost” should be the name of the key you used in the dhcpd.conf file, you can use use the hostname if you want. This will create two files, one with the extension key and one with the extension private. Open the key file, and copy the last part of the line in that file (i.e. everything after the last space). This is the key’s secret: you have to to put it in your dhcpd.conf file after the secret keyword.

This is how dhcpd.conf looks like on the secondary dhcpd:

failover peer "dhcp" {
    secondary;
    address 123.456.789.002;
    port 519;
    peer address 123.456.789.001;
    peer port 519;
    max-response-delay 60;
    max-unacked-updates 10;
    load balance max seconds 3;
}

key secondaryhost {
    algorithm hmac-md5;
    secret "secret";
};

omapi-key secondaryhost;
omapi-port 7911;
include "/etc/dhcp/dhcpd.master";

Create an OMAPI key on the secondary dhcpd host, just like you did on the primary host, and copy the key’s secret in the dhcpd.conf.

Now we have to create the dhcpd.master file. This file should be at all time identical on both hosts! It can look something like this:

default-lease-time 7200;
max-lease-time 14400;
option domain-name-servers 123.456.789.001, 123.456.789.002;
option domain-name "my.domain";
subnet 123.456.789.0 netmask 255.255.255.0 {
    authoritative;
    option subnet-mask 255.255.255.0;
    option broadcast-address 123.456.789.255;
    option routers 123.456.789.5;
    pool {
        failover peer "dhcp";
        range 123.456.789.50 123.456.789.150;
    }
}

Basically the only difference with your non-failover dhcpd.conf file, is that you add a line

failover peer "dhcp"

in every pool section. If you used another name than dhcp in the failover peer section in dhcpd.conf, replace dhcp by the name you have chosen.

Now we have to find a way to make sure that dhcpd.master on the secondary dhcpd is always in sync with the one on the primary dhcpd. In the future, I might start using something like cfengine to manage configuration files on my servers, but for now I wanted something simple to set up. Incron was the perfect solution. On the primary server, I created a /etc/incron.d/dhcpd file with these contents:

/etc/dhcp IN_CLOSE_WRITE /usr/local/sbin/dhcpd.master.incron.sh $@/$#

This will execute the given shell script for every time a file is modified and closed in the directory /etc/dhcp. Note that you cannot just put an inotify watch on /etc/dhcp/dhcpd.master itself, as the watch will be lost whenever dhcpd.master is deleted, and this is exactly what happens when you use vim to edit the file: when saving an existing file, vim creates a new temporary file, then deletes the original old file, and renames the temporary file to the original name.

/usr/local/sbin/dhcpd.master.incron looks like this:

#!/bin/bash
if [ $1"x" == "/etc/dhcp/dhcpd.masterx" ]
then
    scp /etc/dhcp/dhcpd.master dhcp@secondarydhcp.my.domain:/var/lib/dhcpd/dhcpd.master/
fi

We check whether the saved file is dhcpd.master, and in that case, we scp it to the secondary server.

On the secondary server, I created a user dhcp and I created the directory /var/lib/dhcpd/dhcpd.master/, writable by the dhcp users. I added root’s ssh certificate of the primary server to ~dhcp/.ssh/authorized_keys on the secondary server.

The install incron on the secondary server and create this incron configuration file:

/var/lib/dhcpd/dhcpd.master IN_CLOSE_WRITE /usr/local/sbin/dhcpd.master.incron.sh $@/$#

Then /usr/local/sbin/dhcpd.master.incron.sh looks like this:

#!/bin/bash
if [ $1"x" == "/var/lib/dhcpd/dhcpd.master/dhcpd.masterx" ]
then
    cp $1 /etc/dhcp/
    /etc/init.d/dhcpd restart
fi

Now as soon as you have modified dhcpd.master on the primary dhcp server, the file will be copied to the secondary server and the secondary dhcp daemon will be restarted immediately. You still have to take care of restarting the primary dhcp daemon yourself. I prefer not to do this automatically at every dhcpd.master modification, because otherwise you will have no dhcpd running at all anymore if there is a syntax error in dhcpd.master.

If you plan to do maintenance on one of the dhcp servers, which will take longer than the MCLT time, it can be interesting to let the other dhcpd know that its partner is down, so that it will immediately will claim the complete DHCP pool. You can also use this when you start up one of the dhcp servers while the other one is down. If you do not do this, dhcpd will refuse to offer new IP addresses until MCLT has passed.

For example, if the secondary server is down, run this on the primary server:

omshell < < EOF
server localhost
port 7911
key primaryhost "secret"
connect
new failover-state
set name = "dhcp"
open
set local-state = 4
update
EOF

Keep in mind this note of Barry Pederson's blog however:

When the downed server comes back up, the two servers automatically start communicating and eventually get themselves back into a normal state. But only after the recovering server has spent mclt time in recover-wait state, where it renews existing leases but won't offer up new ones. So you probably wouldn't want to go into a partner-down state if the other server will be down for less than that amount of time.