If you have ever used VMWare, you will no doubt be familiar with the concept of a *virtual switch*. In our application we had our VM Hosts in one private vlan, a few things like a PBX and document sharing in another, a public VLAN for mail/etc and a NAT firewall for public/private translation. While VMWare ended up not working out for us, we really liked the clean abstraction that this setup provided so we decided to replicate it with FreeBSD jails for virtualization.
Physical A is using network 10.0.3.x (vlan3), cassandra is using 10.0.2.x (vlan2), and mail/web is using public space of 123.1.2.x (vlan1). We’ll leave VIMAGE jails for another article and make Physical B a non-virtualized firewall with all 3 vlans.
Without further configuration, mail/web will fail, cassandra won’t be hitting the right gateway, cassandra and mail/web can connect right to one another without traversing the firewall, and all 3 of them can connect to Physical A. This obviously isn’t our ideal setup since the whole point of private vlans is security and that’s pretty well shot if the firewall doesn’t even get touched. A common solution to having a private jail on a public host is to NAT directly on the host but that requires a public IP for each host when in reality, we just want a single, normal firewall elsewhere to do it’s thing.
Step 1: Route cassandra traffic
Using pf, we can create a rule that will route all of our cassandra traffic to 10.0.2.1 instead of the default gateway of Physical A which is 10.0.3.1.
pass out on !vlan2 route-to (vlan2 10.0.2.1) from 10.0.2.0/24 to any keep state
This rule says “if vlan2 traffic is going out of any interface that isn’t vlan2, send it to vlan2 through 10.0.2.1″. While we probably could’ve used a static route here instead, we have to use pf for a few other things so I just like to keep all of the logic tidy in one config file. While this will send most outbound traffic through the firewall, cassandra can still communicate with mail/web/physical A directly via lo0 so we need to stop that as well.
Step 2: Route cassandra traffic over the lo0
This rule looks exactly like the one above, but instead directly specifies the loopback interface. It seems like lo0 should be included in !vlan2 but it’s not, get over it.
pass out on lo0 route-to (vlan2 10.0.2.1) from 10.0.2.0/24 to any keep state
Now if cassandra wants to talk to mail/web or communicate with Physical A, it’s going to have to do it through the firewall which will implement all of our desired security policies.
Step 3: Respond to inbound traffic correctly
When another host wants to communicate with cassandra we need to make sure it responds on the correct interface.
pass in on vlan2 reply-to (vlan2 10.0.2.1) from any to 10.0.2.0/24 keep state
Step 4: Some intra-host security
We need to make sure that the only things that can communicate with cassandra jails on this machine are other cassandra jails.
block in on lo0 from !10.0.2.0/24 to 10.0.2.0/24
Note that this rule can be left out for public vlans since anything is allowed to connect to them.
Conclusion
You’ll need a set of rules like this for every vlan or subnet that you have, but the nice thing about this is that if you have a cluster of 50 or 100 machines, the same pf.conf file can be used for each one similar to the distributed virtual switch feature of VMWare. Another nice thing about keeping it all in pf is that you can also add queuing or prioritization for each subnet or host and using the firewall lets you perform aggregate queueing for any subnets that use it .. not a bad setup.
