The ZFS storage box from our last post has two 25G ports, and a single port saturates fast when a lot of machines pull images at once. We wanted to bond the two ports so the switch sees one logical link — more aggregate throughput, plus automatic failover if a port or a cable dies. On FreeBSD the kernel module for this is lagg (elsewhere it goes by bonding, aggregate, port-channel, port-group — same thing, many names).
This post is the full config we ran against a Huawei switch with LACP, plus the one thing people reliably get wrong: bonding two 25G ports does not give a single connection 50G.
Environment
os: FreeBSD 12.3-RELEASE
NIC: Mellanox Technologies ConnectX-4 Lx
switch: Huawei ce6820-48S6CQ-B
The requirement is simple: FreeBSD connects to the switch through both ConnectX-4 ports, and we want link aggregation so the two ports share the load.
FreeBSD side
Bring it up by hand first to confirm the link comes up, then persist it to a config file.
kldload if_lagg # load the lagg kernel module (bonding, in other words)
ifconfig lagg0 create # create the lagg interface
ifconfig mce0 up # bring up physical port mce0
ifconfig mce1 up # bring up physical port mce1
ifconfig lagg0 up laggproto lacp laggport mce0 laggport mce1 # start the aggregate, protocol lacp
Besides lacp, laggproto can be loadbalance, fec, failover, roundrobin, or none — the differences are below. For a Huawei switch, use lacp.
Switch side
On the Huawei, add the matching ports to an Eth-Trunk in LACP dynamic mode. Both ends must agree on the protocol or LACP won’t negotiate:
int eth 111
mode lacp-dynamic
Persisting to /etc/rc.conf
Once the manual run checks out, write it into /etc/rc.conf so it comes back after a reboot:
ifconfig_mce0="up"
ifconfig_mce1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="up laggproto lacp laggport mce0 laggport mce1"
ipv4_addrs_lagg0="x.x.x.x/x"
defaultrouter="x.x.x.x"
Note the IP goes on lagg0, not on the physical ports — the physical ports just carry traffic; the layer-3 address belongs to the aggregate interface.
Set your expectations about LACP hashing first
The classic mistake is assuming that bonding two 25G ports lets one download run at 50G. It won’t.
LACP picks a member port per flow by hashing the packet headers. FreeBSD’s lagg hash uses the Ethernet source/destination addresses, the VLAN tag if present, and the IPv4/IPv6 source/destination addresses. A single connection has fixed source and destination addresses, so its hash is fixed, so that connection always rides exactly one port — a single TCP connection still tops out at 25G. The win is load balancing across flows: when many clients pull images at once, those connections hash across both ports and aggregate throughput approaches 50G.
So aggregation buys you concurrent bandwidth and redundancy, not single-flow speed. If your bottleneck is one big flow, LACP can’t help and you need a different lever (more connections, multipath, or a faster single port). This is exactly why we turned on SMB3 multichannel on the storage side — so a single client splits itself into multiple flows and can actually use the bonded bandwidth.
The lagg protocols
The modes laggproto accepts, and why we picked LACP here:
| Protocol | Notes |
|---|---|
| failover | Active/standby. The first port is primary, the rest are standby. Traffic uses the primary and switches to a standby when the primary fails. Use it when you want redundancy but not stacked bandwidth. |
| LACP | Supports the IEEE 802.3ad Link Aggregation Control Protocol and the Marker Protocol. It negotiates one or more link aggregation groups (LAGs) with the peer; each LAG is made of same-speed, full-duplex ports, and traffic balances across the ports in the highest-total-speed LAG — usually a single LAG holds every port. It reconverges quickly when the physical topology changes. Outbound traffic is balanced by hashing the packet headers (Ethernet source/dest, VLAN tag, IPv4/IPv6 source/dest); inbound is accepted from any active port. This is the usual choice against a Huawei switch. |
| FEC | Supports Cisco EtherChannel. A static setup that neither negotiates with the peer nor exchanges frames to monitor the link. If the switch supports LACP, use LACP instead. Same header-hash balancing. |
| loadbalance | An alias for FEC. |
| roundrobin | Distributes outbound traffic round-robin across all active ports and accepts inbound from any active port. This reorders Ethernet frames and should be used with care, and it very likely won’t balance evenly. |
| none | Does nothing: it disables all traffic without disabling the lagg interface itself. |
We chose LACP because it negotiates bidirectionally with the switch, detects member-port failures, and reconverges fast — things the static modes (FEC/loadbalance) can’t give you. roundrobin could in theory let a single flow use both ports, but it reorders frames, which is a disaster for TCP, so we stay away from it.
Takeaway
The config itself is a dozen lines. The thing to actually internalize is the hashing rule: LACP aggregation raises total throughput and availability, not the speed of any one connection. After you’ve configured it, verify with concurrent clients that traffic really spreads across both ports — netstat -I or systat -ifstat shows live per-port throughput, so don’t draw conclusions from the aggregate counter alone.