Policy based routing on Linux

Policy based routing on Linux

To make a PBR on linux you have to be aware of a basic roting concepts, iptables (mangle) and be able to write ip rule and more routing tables.
For this purpose I tried one simple virtualbox network.

So first client (host 1) with IP 192.168.100.2 is communicating to the host on the other side (host 2) with IP 192.168.200.2. Because both host have default route set to the router in the middle the communication flows through the router. This router is linux so, let’s try to redirect all the traffic between these two hosts to the „watcher“ on the top of the chart with IP 10.100.0.2 or 10.0.0.2.

Basically if you had some python script/C program in the watcher machine which would be able to simply process the packets and resend it back after processing there is no need to have two interfaces there.

We can run ping on host number 1 to host number 2 and check with tcpdump if the packets are flowing as expected (of course they are). So now let’s redirect first direction between router and host 1.

echo 201 to_w >> /etc/iproute2/rt_tables
ip route add defautl gw via 10.0.0.2 dev enp0s9 table to_w
ip rule add fwmark 11 table to_w

And mark the packets with iptables to match previously created ip rule with fwmark 11.
-A PREROUTING -i enp0s8 -s 192.168.100.2 -d 192.168.200.2 -j MARK --set-mark 11

After this we need to allow forwarding in kernel so put these lines into /etc/sysctl.conf :
net.ipv4.conf.all.forwarding=1
net.ipv4.conf.default.forwarding=1

And
sysctl --system

And with tcpdump you should see redirected traffic on the watcher with tcpdump.
The same procedure we need to repeat on the watcher to resend packets back with the interface nr.2.
echo 201 from_w >> /etc/iproute2/rt_tables
ip route add defautl gw via 10.100.0.1 dev enp0s9 table from_w
ip rule add fwmark 22 table from_w
# put into iptables
-A PREROUTING -i enp0s8 -s 192.168.100.2 -d 192.168.200.2 -j MARK --set-mark 22

Allow forwarding
net.ipv4.conf.all.forwarding=1
net.ipv4.conf.default.forwarding=1

Than next procedure is to check the mark rule is matching on the iptables and that we see second time redirected packet on the router interface.

Now you will see the packet is on the ingress interface of the router but the router doesn’t send the packet althought it has the destination as a directly connected network. That is because of reverse path filtering. So we need to turn off rp_filter in the kernel.
Same procedure as when enabling forwarding.

After that on the client 2 you will see incoming echo request and outgoing echo reply. Next we need to match the flow in the opposite direction so again implement some rules on the router first (between router and client 2).

ip rule add fwmark 12 table to_w
# iptables for returning packets
-A PREROUTING -i enp0s3 -s 192.168.200.2 -d 192.168.100.2 -j MARK --set-mark 12

and now the part between the watcher and the router to redirect the traffic back to router:
ip rule add fwmark 23 table from_w
# iptables for returning packets
-A PREROUTING -i enp0s8 -s 192.168.200.2 -d 192.168.100.2 -j MARK --set-mark 23

Now you have succesfuly redirected the traffic and you will be able see it on the watcher.
The complete excerpts are here:

And from router:

You can see whole the traffic but as a evidence let’s try to make curl to the webserver running on the host 2.
Tcpdump on the host 2:

Tcpdump on the watcher:


Done.