Complex routing with OpenBSD

We had to solve a routing problem for a non common network setup. I won’t get into details on why we chose to do it that way, let’s say it was the simplest way we imagined to solve our problem.

We have 2 servers having exactly the same IP address (192.168.1.50) and we wanted both of them to see our router with the same IP address (192.168.1.60), both have their default route on e.g. 192.168.1.1 and for some reason we want not change the routing configuration. We want to be able to interrogate both servers using 2 distinct IP addresses from client-side subnet (10.10.10.100 for accessing server1, 10.10.10.200 for accessing server2), the router itself will be accessed with another IP address (10.10.10.10). In addition to that, the server may initiate a connection on client side and when doing so they should obtain their corresponding IP on client side (10.10.10.100 & 10.10.10.200). The following picture illustrates the whole setup.

The problems identified compared to a basic router setup:

1. interface vr1 on which server1 is connected has the same IP address as vr2 on which server2 is connected
2. server1 and server2 have the same IP address
3. client has to be able to connect to each server without changing the routing table of the servers
4. the servers has to connect to some fixed hosts on client side

To accomplish this we are using bidirectional NAT with pf and routing domains on OpenBSD. Similar configuration may be done on GNU/Linux using iptables and policy routing with the iproute2 utilities.

A routing domain has its own routing table and each network interface can be in only one routing domain. By default everything is routing domain id 0. Ifconfig can be used to change an interface routing domain it belongs to. Excerpt of OpenBSD’s IFCONFIG(8) man page

rdomain route-id
     Attach the interface to the routing table with the
     specified route-id.  Interfaces in different routing
     domains are separated and can not directly pass traffic
     between each other.  By default all interfaces belong to
     routing table 0.

So first we configure each interface to be in its own routing domain, vr1 in routing domain 1, vr2 in routing domain 2 and vr0 stays in default routing domain 0. As specified above vr0 is setup with 3 IP addresses, so the 2 addresses forwarded to the server are defined as aliases. Here is the content of the 3 interfaces config files

/etc/hostname.vr0

inet 10.10.10.10 255.255.255.0
inet alias 10.10.10.100 255.255.255.0
inet alias 10.10.10.200 255.255.255.0

/etc/hostname.vr1

rdomain 1
inet 192.168.1.60 255.255.255.0
!route -T1 add default 192.168.1.60

/etc/hostname.vr2

rdomain 2
inet 192.168.1.60 255.255.255.0
!route -T2 add default 192.168.1.60

I will explain the route entries later. As stated in the ifconfig man page “Interfaces in different routing domains are separated and can not directly pass traffic between each other”, specific pf rules are needed to allow traffic between routing domains using the rtable keyword, excerpt of PF.CONF(5) man page

rtable <number>
   Used to select an alternate routing table for the routing lookup.
   Only effective before the route lookup happened, i.e. when
   filtering inbound.

As stated, an inbound rule may specify that a matching packet jumps to a different routing domain with the rtable keyword, the <number> specifies the target routing domain.

/etc/pf.conf excerpt:

# biNAT and routing domain traversal client->server
match out on vr1 nat-to vr1
match out on vr2 nat-to vr2
match in on vr0 to 10.10.10.100 rdr-to 192.168.1.50 rtable 1
match in on vr0 to 10.10.10.200 rdr-to 192.168.1.50 rtable 2

The nat-to rules are standard rules for doing SNAT. The rdr-to rules are standard rules for doing DNAT with rtable specifying the target routing domain. Omitting it would make the packets being redirected to 192.168.1.50 on interface vr0 as it is belonging to default routing domain 0 (we didn’t test that, but it probably won’t work at all, pf.conf man page says Redirections cannot reflect packets back through the interface they arrive on). This works fine but now I have to explain the route entries in the hostname.vr1 and hostname.vr2 files. What happens when the router receives a packet with 10.10.10.100 as destination and with 10.10.10.1 as source on interface vr0:

• the match in rule does the DNAT, rewrites the destination address to be 192.168.1.50, creates an entry in the DNAT table and does the jumping to routing domain 1.
• the match out rule does the SNAT, rewrites the source address to be 192.168.1.60 (vr0’s address) and creates an entry in the SNAT table.
• the packet is sent to server 1 as expected
• server 1 responds to 192.168.1.60 as expected too
• the router gets the response on vr1, it hits the DNAT and SNAT table, does the translation, but OpenBSD’s kernel emits an ICMP Destination unreachable error

What happened? The response packet was translated back, its destination is 10.10.10.1 who initiated the communication and routing domain 1 has no route for this destination, this is why it emits the ICMP error. It happens because the routing domain traversal is done after the kernel routing lookup. To circumvent that a fake default route to itself is added to the routing domain 1. Itself can be either vr1’s address 192.168.1.60 or the address of the loopback interface. The latter may seem to be a better choice (or more logical), but it needs more configuration. Loopback interface lo0 is in routing domain 0, and as said above, an interface can be in only one routing domain, thus it would be needed to create and second loopback interface lo1 and a third lo2 for the routing domains 1 and 2. Pf would also need extra configuration for allowing lo1 and lo2 and it wouldn’t be possible to use the set skip on lo1 as pf would exit when hitting this rule and not accomplish the SNAT/DNAT/rdomain hopping. Because of that extra configuration we chose the former possibility. Finally it doesn’t really matter because, the kernel doesn’t complain anymore about an unreachable destination, the packet gets back to routing domain 0 and uses its routing table (thus never uses default route in routing domain 1).

Everything is working as planned, client can reach both servers, the translation is done correctly. Routing domains allows us to have identical IP addresses on two interfaces and allows us to reach both servers having the same address. SNAT/DNAT rewrites everything that would require extra routing entries on the servers which we don’t want. Last point is still open, the servers has to reach some services on client side. So let’s do the same pf rules from server-side to client-side, pf.conf excerpt:

# biNAT and routing domain traversal server->client
match out on vr0 received-on vr1 nat-to 10.10.10.100
match out on vr0 received-on vr2 nat-to 10.10.10.200
match in on vr1 to 192.168.1.60 rdr-to 10.10.10.1 rtable 0
match in on vr2 to 192.168.1.60 rdr-to 10.10.10.1 rtable 0

We saw above that the rtable keyword is only effective on inbound rules, thus we cannot write a outbound rule that matches the source routing domain, and because of that both servers would be translated to source address 10.10.10.100. The received-on keyword allows us to detect from where the packet was coming from and so doing the SNAT as we wish. For the DNAT it is the same as above allowing to packets to jump to routing domain 0, however the to 192.168.1.60 could be omitted because vr1 and vr2 have no aliases.

So we resolved our 4th problem. As you may have noticed, we used only match rules for SNAT/DNAT/rdomain hopping, this would work with a pass everything rule, but a good practice is to authorize only needed traffic, here is a more complete /etc/pf.conf file

set skip on lo
block log all

# allow ssh traffic to router
pass in on vr0 proto tcp to 10.10.10.10 port 22
pass out on vr0 proto tcp from 10.10.10.10 port 22

# allow incoming connections destinated to vr0 aliases
# (will be redirected to server1 and server2)
pass in on vr0 proto tcp to 10.10.10.100 port { 22, 443 }
pass in on vr0 proto tcp to 10.10.10.200 port { 22. 443 }

# allow outgoing traffic on vr1 and vr2
# to connect to server1 and server2
pass out on vr1 to 192.168.1.50 port { 22, 443 }
pass out on vr2 to 192.168.1.50 port { 22, 443 }

# allow traffic from servers to specific client service
pass in on vr1 proto udp to 192.168.1.60 port 514
pass in on vr2 proto udp to 192.168.1.60 port 514
pass out on vr0 proto udp to 10.10.10.1 port 514

# biNAT and routing domain traversal client->server
match out on vr1 nat-to vr1
match out on vr2 nat-to vr2
match in on vr0 to 10.10.10.100 rdr-to 192.168.1.50 rtable 1
match in on vr0 to 10.10.10.200 rdr-to 192.168.1.50 rtable 2

# biNAT and routing domain traversal server->client
match out on vr0 received-on vr1 nat-to 10.10.10.100
match out on vr0 received-on vr2 nat-to 10.10.10.200
match in on vr1 to 192.168.1.60 rdr-to 10.10.10.1 rtable 0
match in on vr2 to 192.168.1.60 rdr-to 10.10.10.1 rtable 0