Jasmine Moore's Personal Blog

Background

I purchased a mini-PC to use as a firewall / replacement router for my ISP. Unfortunately, its SFP+ modules were not detected under opnsense, even when connected via DAC to its on-“rack” switch. Let’s get to the bottom of this!

The router in question is a Gowin R86S. The switch is a Mokerlink 8-port ethernet switch, which has 8 2.5GBe ports plus an additional SFP port at 10GB. They are connected with a SFP DAC cable, which is simply a SFP port-to-port connection and is very cheap (cheaper than cat6!). The second SFP port can be used to add a connection to the downstairs rack through in-home cable (or the reverse!) fairly cheaply, but that will be a project for another day. Success here will be defined as the SFP interface recognized by opnsense and added to the LAN interface as a downstream switch.

User Setup

First step, why aren’t these recognized? There’s nothing usable in the GUI, so we’re going to need SSH access to opnsense. I’ve never added myself, as a user, so…

After pasting in the public key from puttygen, we’re ready to go!

Only this didn’t work. The key should allow access, but, no dice. Switched to password authentication, still no dice, and I got myself locked out of opnsense due to password retries… Some time later, I ticked the following two cursed boxes, and made sure it was only exposed on LAN:

And I’m in.

What Is This Port?

The machine has two SFP ports. That’s a form factor, not a specific card! Let’s see exactly what they used here:

root@OPNSense:~ # pciconf -l
hostb0@pci0:0:0:0:      class=0x060000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x4e24 subvendor=0x8086 subdevice=0x7270
vgapci0@pci0:0:2:0:     class=0x030000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4e61 subvendor=0x8086 subdevice=0x2212
xhci0@pci0:0:20:0:      class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4ded subvendor=0x8086 subdevice=0x7270
none0@pci0:0:20:2:      class=0x050000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4def subvendor=0x8086 subdevice=0x7270
none1@pci0:0:22:0:      class=0x078000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4de0 subvendor=0x8086 subdevice=0x7270
sdhci_pci0@pci0:0:26:0: class=0x080501 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4dc4 subvendor=0x8086 subdevice=0x7270
pcib1@pci0:0:28:0:      class=0x060400 rev=0x01 hdr=0x01 vendor=0x8086 device=0x4db8 subvendor=0x8086 subdevice=0x7270
pcib2@pci0:0:28:1:      class=0x060400 rev=0x01 hdr=0x01 vendor=0x8086 device=0x4db9 subvendor=0x8086 subdevice=0x7270
pcib3@pci0:0:28:2:      class=0x060400 rev=0x01 hdr=0x01 vendor=0x8086 device=0x4dba subvendor=0x8086 subdevice=0x7270
pcib4@pci0:0:28:4:      class=0x060400 rev=0x01 hdr=0x01 vendor=0x8086 device=0x4dbc subvendor=0x8086 subdevice=0x7270
isab0@pci0:0:31:0:      class=0x060100 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4d87 subvendor=0x8086 subdevice=0x7270
hdac0@pci0:0:31:3:      class=0x040300 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4dc8 subvendor=0x8086 subdevice=0x7270
none2@pci0:0:31:4:      class=0x0c0500 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4da3 subvendor=0x8086 subdevice=0x7270
none3@pci0:0:31:5:      class=0x0c8000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x4da4 subvendor=0x8086 subdevice=0x7270
igc0@pci0:1:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
igc1@pci0:2:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
igc2@pci0:3:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
mlx4_core0@pci0:4:0:0:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1003 subvendor=0x15b3 subdevice=0x0113

One bit of googling later, mlx4_core appears to be an Nvidia (formerly Mellanox) ConnectX-3. Sneaky sneaky! This too-good-to-be-true machine appears to be cobbled together from various used server parts, which, fair. I could also have read the article that I linked to in the first paragraph to obtain this information, but, linear writing is overrated. Let’s figure out why opnsense-aka-freebsd isn’t recognizing the Nvidia-Mellanox ConnectX-3.

Making OPNsense recognize 6-year-old networking hardware

What if we just need firmware? This thread on the FreeBSD forums seems to say that we can add it to /boot/loader.conf. However, in shell, we see that this has a big ol’ disclaimer to change it in System > Settings > Tuneables from the OPNsense GUI, so back to clicky-land we go.

In that menu, it was simple to add a ““Tuneable”” named mlx4en_load, a description of whatever, and the value YES (no quotes!). Rebooting the system and checking…

root@OPNsense:~ # ifconfig
igc0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: LAN (lan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 00:f0:da:ef:01:9c
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 00:f0:da:ef:01:9d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: WAN (wan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 00:f0:da:ef:01:9e
        inet 192.168.2.254 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        syncpeer: 0.0.0.0 maxupd: 128 defer: off
        syncok: 1
        groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
        groups: pflog
mlxen0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether f4:52:14:be:a8:8c
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
mlxen1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether f4:52:14:be:a8:8d
        media: Ethernet autoselect (autoselect <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

We have success!

Setting up a LAN Bridge

So now that the hard part is done, we just have to do the easy bit of… Oof. Swapping the interface on OPNSense.

We do not want to hook up a display to this thing… so let’s see if we can use the failover from our switch. While the opnsense bridge docs say to only set it up in certain cases, having a bridge instead of a raw interface gives us a lot of flexibility in the future!

It is important to follow that document exactly, which means that you should always have one port free (or free-able!) for management on your switch. We’re going to use one of the free ethernet ports I have for this.

After Step 3 of the linked guide, it works!! Step 4 was also done, so now all ports on the device (including mlx4) are now bridged. We have success! That I’m able to shell at all into the OPNsense server is partial success, and the cable was disconnected, giving the following from the OPNsense ifconfig:

root@OPNsense:~ # ifconfig
igc0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: igc0 (opt3)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 00:f0:da:ef:01:9c
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: Diagnostic (opt2)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 00:f0:da:ef:01:9d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: WAN (wan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 00:f0:da:ef:01:9e
        inet 192.168.2.254 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
...
mlxen1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: mlxen1 (opt1)
        options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether f4:52:14:be:a8:8d
        inet 192.168.1.254 netmask 0xffffffff broadcast 192.168.1.254
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: LAN (lan)
        ether 58:9c:fc:10:ff:ce
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: mlxen1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 55
        member: igc1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>