4. The CARP protocol

CARP (Common Address Redundancy Protocol) is the protocol that achieves system redundancy, by having a group of hosts on the same network segment (redundancy group) to share an IP address, so that, in case of a machine going down, another host in the redundancy group can take over its tasks. CARP also allows a degree of load sharing between systems.

Although creating redundant firewalls is one of its most common uses, CARP isn't a firewall-specific protocol. It can be used to ensure service continuity and/or load sharing to a number of network services.

Initially, the OpenBSD team wanted to produce a free implementation of the IETF standard protocols, VRRP (Virtual Router Redundancy Protocol), defined in [RFC3768], and HSRP (Hot Standby Router Protocol), defined in [RFC2281]; but Cisco, claiming patent rights on it, firmly informed the OpenBSD community that Cisco would defend its patents for VRRP implementation (see [CARP] for more details), thus forcing the OpenBSD developers to create a new, competing protocol designed to be fundamentally different from VRRP.

CARP is a multicast protocol, grouping several physical computers together under one or more virtual addresses. Of these, one system is the master and responds to all packets destined for the group; the other systems (backups) just stand by, waiting for any problem to take its place (as it happens between co-workers...).

At configurable intervals, the master advertises its operation on IP protocol number 112. If the master goes offline, the other hosts in the redundancy group begin to advertise. The host that's able to advertise most frequently becomes the new master. When the main system comes back up, it becomes a backup host by default, although it can be configured to try to become master again.

As you can see, CARP only creates and manages the virtual network interface; it's up to the system administrator to synchronize data between applications, using pfsync(4) (which we'll discuss in the next chapter), rsync or whatever protocol is appropriate for the specific application.

4.1 Configuration parameters

CARP configuration is done via the sysctl(8) and ifconfig(8) commands. There are three relevant sysctl(3) variables:

net.inet.carp.allow
defines whether the host handles CARP packets or not. It is enabled by default;
net.inet.carp.log
defines whether to log CARP errors or not. It may be a value between 0 and 7, corresponding to the syslog(3) priorities, and defaults to 2 (i.e. only CARP state changes are logged);
net.inet.carp.preempt
if set to 0 (default), the host won't try to become master if it receives CARP advertisements from another master. Otherwise, it will try to become master if it is able to advertise more frequently than the current master. This option also enables failing over all interfaces in the event that one interface goes down. In fact, if one physical CARP-enabled interface goes down, CARP will increase by 1 the demotion counter (see below) for all groups that the interface belongs to, thus allowing the election of new masters on all subnets.

The syntax for configuring CARP with ifconfig(8) is:

    ifconfig carpN create

    ifconfig carpN [advbase n] [advskew n] [balancing mode]   \
    [carpnodes vhid:advskew,vhid:advskew,...] [carpdev iface] \
    [[-]carppeer peer_address] [pass passphrase] [state state] [vhid host-id]
carpN
the name of the carp(4) virtual interface.
advbase, advskew
these values determine the interval between two consecutive CARP advertisements. This interval (in seconds) is given by the formula (advbase + (advskew / 255)); increasing advbase will decrease network traffic but increase the delay in electing the new master. Small advskew values allow a host to advertise more frequently, increasing its probability to become master. The values of advbase and advskew must be in the range 0 to 254 and default to 1 and 0 respectively;
balancing
sets the load balancing mode (which will be discussed later); valid modes are arp, ip, ip-stealth and ip-unicast;
carpnodes
a comma-separated list of vhid:advskew pairs to actually define how the load should be shared among the configured carp nodes (see below for further details);
carpdev
specifies the physical interface that belongs to this redundancy group. By default, CARP uses the physical interface that belongs to the same subnet as the virtual interface;
[-]carppeer
allows you to specify the IP address of the other CARP peer(s), instead of using the default multicast group; this allows to use ipsec(4) to protect carp(4) traffic;
pass
the authentication password to use when talking to other CARP-enabled hosts in the redundancy group. This must be the same on all members of the group;
state
force a carp(4) interface into a specific state (init, backup or master);
vhid
the Virtual Host ID. This is a unique number (between 1 and 255) that is used to identify the redundancy group to the other nodes on the network.

4.1.1 The demotion counter

Besides basic configuration, the ifconfig(8) command also allows you to tweak the CARP demotion counter, which is a measure of how "ready" a host is to become master of a CARP group [CARPFAQ] (the higher the counter, the less ready the host). Let's see it in more detail.

CARP interfaces are divided in groups (by default all carp(4) interfaces are members of the "carp" interface group) and each group is assigned a demotion counter, whose value can be viewed by running the following command:

$ ifconfig -g carp
carp: carp demote count 0

The demotion counter comes in handy mainly when:

For further details on the CARP demotion counter, please refer to [CARPFAQ].

4.1.2 Load balancing

CARP provides two different methods for load balancing incoming network traffic among a set of CARP-enabled hosts: ARP balancing and IP balancing.

Both methods require that you first create a load balancing group by configuring, on each balanced carp(4) interface, as many VHIDs as hosts in the balancing group; the advskew on each VHID will be configured so that each host will be the master on a separate VHID (see below for a practical example).

ARP balancing works by applying a hash function to the source MAC address of ARP requests to determine which VHID should handle the request. The ARP request will be answered solely by the host whose carp(4) interface is master for that VHID. ARP load balancing can be enabled through ifconfig(8) by setting the value of the balancing option to "arp" on all hosts; for instance:

# ifconfig carp0 balancing arp carpnodes 1:0,2:100

IP load balancing works in a very similar manner to ARP balancing, but uses the hash of the source and destination addresses of the IP packet to determine which VHID (and therefore which host) should accept the packet.

IP balancing requires that traffic destined to the CARP address be received by all CARP hosts. It can be enabled using ifconfig(8), by setting the balancing option to "ip"; this will cause CARP to use a multicast MAC address, forcing the switch to send incoming traffic to all nodes in the redundancy group. For example:

# ifconfig carp0 balancing ip carpnodes 1:0,2:100

Alternatively, you can set the balancing option to "ip-stealth" (stealth mode), in order to prevent hosts from sending packets with their virtual MAC address as source; this will prevent the switch from learning the virtual MAC address, forcing it to flood the traffic to all its ports. Last, if you're using a hub or a switch that supports some kind of monitoring mode, you can set balancing to "ip-unicast".

The choice between the two load balancing mechanisms mostly depends on the network environment in which the systems will be placed: ARP balancing only works for clients in the local network and cannot balance traffic that crosses a router, as routed traffic always contains the MAC address of the router as its source address. Therefore, if clients are on remote networks, IP balancing is the only option; the only drawback of IP balancing is that traffic destined towards the load balanced IP addresses must be received by all CARP-enabled hosts, resulting in a higher network load.

4.2 Parameters configuration

Now it's time to configure CARP on our firewalls. To examine two slightly different CARP configurations, we will set up the two internal firewalls (Mickey and Minnie, between LAN and DMZ) in active/stand-by mode, with only one system filtering the whole network traffic and the other one acting as a hot spare; the two external firewalls (Donald and Daisy, separating the DMZ from the internet), instead, will be in active/active mode, sharing the traffic load.

So let's recap the firewalls adresses, as we have seen them in the network diagram:

Mickey Minnie Virtual address
LAN 172.16.0.200 172.16.0.201 172.16.0.202
DMZ 172.16.240.200 172.16.240.201 172.16.240.202
pfsync 192.168.2.200 192.168.2.201
Donald Daisy Virtual address
DMZ 172.16.240.100 172.16.240.101 172.16.240.102
 Internet  172.16.250.100 172.16.250.101 172.16.250.102
pfsync 192.168.1.100 192.168.1.101

4.2.1 Active/standby configuration

Let's start with Mickey and Minnie: first, we need to create the carp* devices and configure them with ifconfig(8):

mickey# ifconfig carp0 172.16.0.202/24 vhid 1 pass password1 advbase 1 advskew 0
mickey# ifconfig carp1 172.16.240.202/24 vhid 2 pass password2 advbase 1 advskew 0
minnie# ifconfig carp0 172.16.0.202/24 vhid 1 pass password1 advbase 1 advskew 100
minnie# ifconfig carp1 172.16.240.202/24 vhid 2 pass password2 advbase 1 advskew 100

We have just created the interfaces, assigned them an IP address, a virtual host ID (1 on the LAN, 2 on the DMZ) and a password (probably not the most secure...) for authentication. We also decided that, whenever possible, Mickey will be the master; this is done by giving Minnie a higher advskew value (100), thus making the interval between its advertisements (1 + 100 / 255) higher than the interval between Mickey's advertisements (1 + 0 / 255). And, as we've seen above, the host that's able to advertise most frequently becomes master.

Furthermore, by setting net.inet.carp.preempt to "1" on Mickey, we ensure that Mickey will always try to become the master:

mickey# sysctl net.inet.carp.preempt=1
net.inet.carp.preempt: 0 -> 1

To make these settings permanent after reboot, we just need to edit the /etc/hostname.carp* and /etc/sysctl.conf files on Mickey:

/etc/hostname.carp0
inet 172.16.0.202 255.255.255.0 172.16.0.255 vhid 1 pass password1 advbase 1 advskew 0
/etc/hostname.carp1
inet 172.16.240.202 255.255.255.0 172.16.240.255 vhid 2 pass password2 advbase 1 advskew 0
/etc/sysctl.conf
[...]
net.inet.carp.preempt=1

and on Minnie:

/etc/hostname.carp0
inet 172.16.0.202 255.255.255.0 172.16.0.255 vhid 1 pass password1 advbase 1 advskew 100
/etc/hostname.carp1
inet 172.16.240.202 255.255.255.0 172.16.240.255 vhid 2 pass password2 advbase 1 advskew 100

Note: to make the adoption of CARP easier on pre-existing networks, CARP allows using the physical address of a host as the virtual address of the whole redundancy group.

4.2.2 Active/active configuration

Now let's get on to Donald and Daisy and start by configuring their DMZ interfaces. As before, we will create the carp0 device on each machine, but this time, to enable load balancing, we will use the carpnodes option to assign two different Virtual Host IDs to the interface (VHIDs 3 and 4).

On VHID 3, we will set the advskew of Donald and Daisy to 0 and 100 respectively: this will ensure that Donald becomes master for that VHID; on VHID 4, instead, we will do the opposite, by setting the advskew of Donald and Daisy to 100 and 0 respectively, in order to force Daisy to become master for VHID 4:

donald# ifconfig carp0 172.16.240.102/24 balancing ip carpnodes 3:0,4:100 \
> pass password3
donald# sysctl net.inet.carp.preempt=1
net.inet.carp.preempt: 0 -> 1
daisy# ifconfig carp0 172.16.240.102/24 balancing ip carpnodes 3:100,4:0 \
> pass password3
daisy# sysctl net.inet.carp.preempt=1
net.inet.carp.preempt: 0 -> 1

We now have two redundancy groups with the same IP address, but each with a different master:

donald# ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 01:00:5e:00:01:01
        carp: carpdev rl1 advbase 1 balancing ip
                state MASTER vhid 3 advskew 0
                state BACKUP vhid 4 advskew 100
        groups: carp
        inet 172.16.240.102 netmask 0xffffff00 broadcast 172.16.240.255
        inet6 fe80::2c0:a8ff:fe8e:b112%carp0 prefixlen 64 scopeid 0x5
daisy# ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 01:00:5e:00:01:01
        carp: carpdev rl1 advbase 1 balancing ip
                state BACKUP vhid 3 advskew 100
                state MASTER vhid 4 advskew 0
        groups: carp
        inet 172.16.240.102 netmask 0xffffff00 broadcast 172.16.240.255
        inet6 fe80::219:d2ff:fe02:6469%carp0 prefixlen 64 scopeid 0x5

To make these settings permanent across reboots, we need to edit the startup files on Donald:

/etc/hostname.carp0
inet 172.16.240.102 255.255.255.0 172.16.240.255 balancing ip carpnodes 3:0,4:100 pass password3
/etc/sysctl.conf
[...]
net.inet.carp.preempt=1

and Daisy:

/etc/hostname.carp0
inet 172.16.240.102 255.255.255.0 172.16.240.255 balancing ip carpnodes 3:100,4:0 pass password3
/etc/sysctl.conf
[...]
net.inet.carp.preempt=1

Now we just have to do the same on the external network interfaces, with another two Virtual Host IDs (VHIDs 5 and 6):

donald# ifconfig carp1 172.16.250.102/24 balancing ip carpnodes 5:0,6:100 \
> pass password5
daisy# ifconfig carp1 172.16.250.102/24 balancing ip carpnodes 5:100,6:0 \
> pass password5

and edit the startup files on Donald:

/etc/hostname.carp1
inet 172.16.250.102 255.255.255.0 172.16.250.255 balancing ip carpnodes 5:0,6:100 pass password5

and Daisy:

/etc/hostname.carp1
inet 172.16.250.102 255.255.255.0 172.16.250.255 balancing ip carpnodes 5:100,6:0 pass password5

Though the above configuration involves only a couple of machines, it can be easily extended to up to 32 hosts. One last note: load sharing won't probably achieve a perfect 50/50 distribution between the two machines, since CARP uses a hash of the source and destination IP addresses to determine which system should accept a packet, not the actual load.