Mellanox 40G card not working pfsense/FreeBSD

Mellanox 40G card not working pfsense/FreeBSD

Installing pfsense or FreeBSD and finding that OS doesn’t see interface based on this card looked a little bit tricky, but finally it wasn’t that hot. First you need to check if the driver is loaded and if you see in pci devices previously mentioned card.

pciconf -BbceVv | grep Mellanox

If you see „none@pci…“ before this device you have problem that driver is not loaded. If the driver was loaded correctly you have to see „mlx4_core0@pci…“. To list loaded drivers in kernel type kldstat.

kldstat | grep mlx

If you have no output, navigate to „/boot/kernel“ and check there for 3 files (linuxkpi.ko, mlx4.ko, mlx4en.ko). Probably they will be missing so you have two possibilities now. Compile them, or boot another FreeBSD OS with underlying core version you are running (uname -a) and copy it from that system. Once you have copied them to „/boot/kernel“ you can try to load them manually. Don’t forget to chmod 555 them.

kldload mlx4en

If no errors reported, using „kldstat“ you now have to see loaded drivers mlx4, mlx4en, linuxkpi. So go to the „/boot/loader.conf“ and add mlx4en_load=“YES“ there. Reboot the system and now you should see mellanox cards with proper speed.

If you are still not able to see the cards, because of an error like this during booting you have to continue to another step (otherwise not):

Unable to determine PCI device chain minimum BW

You need to patch the driver for the newer version and replace the files, or force the creation of devices manually.

sysctl  sys.device.mlx4_core0.mlx4_port2="eth"
sysctl  sys.device.mlx4_core0.mlx4_port1="eth

After the commands are executed without errors you will definately see network cards working without problems. Next thing you need to figure out is to make the system execute this commands while booting. So in FreeBSD you can place this two lines (except sysctl part) into „/etc/sysctl.conf“ and they will be read and executed after startup. In pfsense you need to place them into „/cf/conf/config.xml“ after <system> part.

/sbin/sysctl  sys.device.mlx4_core0.mlx4_port1="eth" 
/sbin/sysctl  sys.device.mlx4_core0.mlx4_port2="eth"

Reboot and things will be fine from now.