Monday, August 25, 2014

Setting proxy ARP for bonded interface

One of the interesting challenges is about setting proxy ARP on a FW  bond interface to facilitate manual NAT rules.

There is a very good SK article about proxy ARP configuration that covers both physical firewalls and VSX. There is only one problem with that: it is not applicable to bond interfaces. HA or LS bond type, does not matter. The main issue is that one cannot use either MAC address of the NICs in bond, as frames may sometimes go through another physical link.

So is it absolutely impossible to use a combination of proxy ARP, manual NAT rules and bond interfaces?

Not exactly. Here are several steps that you need to do.

1. Set up VSX and not a physical FW. Even if you do not have VSX licenses, physical FW license will allow you to run a single Virtual System. That is all you need.
2. Instead of connecting your bond interface to the Virtual System (VS), define a Virtual Switch (license will also allow that) and connect it to the bond.
3. Create a virtual link (warp) between VS and the Virtual Switch.
4. Go to CLI, check MAC address of the warp link and use it in local.arp file. Do not forget, you have to set unique ARP entries for each cluster member.

Problem solved.


12 comments:

  1. something worth sharing I think:
    https://www.kernel.org/doc/Documentation/networking/bonding.txt

    ReplyDelete
    Replies
    1. Martjin, very nice doc. The only question is, what is the default behaviour on Gaia and if it can be changed.

      Delete
    2. true, but the doc gave me a very good understanding how things work under the hood and also explains in details why "striping" doesn't work. As for the default behavior I am totally with you...
      Haven't had issues so far with bonding actually. Let's find out in the future ;)

      Delete
  2. Valeri,

    I'm a bit surprised by what you're saying here.
    First of all, bond interfaces do have MACs, and actually the MAC for the bond is chosen from the first slave added, and is then set to also all the other slaves.

    Second of all, the SK does have the option to perform manual proxy ARP using just IPs and interface names, without MACs. That doesn't work?

    ReplyDelete
    Replies
    1. Thanks, David.

      For the first point, which interface is the first added slave after reboot? Can it be set for a specific physical interface? What about HA pair, when the first interface in the list is down?

      For the second point, I have not done that. The end customer has required local.arp to be set and not clish type of setup. I will try on the next project.

      The proposed solution may not be the best one. But it just works.

      Delete
    2. The MAC might change after the first reboot, depending on the exact implementation details. Will probably stay the same for consequent reboots. However, it is probably possible to manually set the MAC of the bond, and so eliminate this inconsistency.

      I didn't say that the proposed solution won't work - it just seems like an overkill, and also there's some nasty pitfalls that new-comers to VSX will probably fall into if they'll blindly follow your advice...

      As a side note - why was there a need for a manual NAT rule? What did the customer want to achieve that was not possible with automatic rules?

      Delete
    3. DBar, the second question first. It was all about migrating a physical FW to VSX, one to one, so the customer insisted on manual NAT rules and static ARP entries in local.arp.

      People below propose VMAC approach, and it seems to be much more interesting than your proposal to rely on bond MAC assignment. That one I did not consider, but again, there is more than one way to skin a cat.

      Delete
  3. Hi Valeri. When I configured bond and ARP Proxy with physical MACs, I had a problem that made the FW responses to many ARP requests that it shouldn't. It wasn't solved by now but I will have to configure VMAC with Cluster XL instead of VRRP. This is GAIA R77.10 and I experimented many unresolved or undocumented bugs which I think are VRRP related.

    ReplyDelete
    Replies
    1. As John mentiones the Virtual MAC functionality is a much better way to solve this in clusterxl enviroments. I have used it a couple of times and has been working without issues.
      This way configuration can also be moved to new hardware without confuguration changes.

      I used the R77.10 approach with proxy arp configuration via clish instead of local.arp and it works as long as you remember the "real-ipv4-address " part of the command. Clish accepts the command without but the gateways won't respond to requests. I learned this the hard way :-)

      Delete
    2. VMAC may be also a fitting solution, I could agree. There is one thing here though. It complicates bond connectivity troubleshooting a bit.

      Delete
    3. Right John, that was the concern. As already said, VMAC might be an alternative

      Delete
  4. All of my layer 3 interfaces are VLAN interfaces on bonds in my VSX cluster. We need to use manual NAT when doing port translation. The MAC of the bond interfaces never change. We just put the proxy-arp IP and the MAC of the bond in local.arp for each cluster member. There is not a third value on each line.

    ReplyDelete