NFS over InfiniBand

From Mark Furneaux's Wiki
Jump to: navigation, search

This guide assumes that you already have a functioning NFS installation and IPoIB network. This guide only covers configuration of NFS to use RDMA, using IPoIB for network addressing. IB kernel modules are present in modern Ubuntu builds, so this guide will not cover building them into the kernel. This guide assumes we are using Mellanox IB cards.

Configuration

Add the following to /etc/modules on the server:

svcrdma

Add the following to /etc/modules on the client:

xprtrdma

Add the following to /etc/rc.local on the server:

echo rdma 20049 > /proc/fs/nfsd/portlist

Add the "insecure" option to each export in /etc/exports on the server that will use the IB link:

/tank/storage 192.168.10.101(rw,async,insecure,no_subtree_check)
/tank/backup 192.168.10.101(rw,async,insecure,no_subtree_check)

and be sure to change the IPs to the IB subnet or add new rules altogether such as:

/tank/storage 192.168.30.101(rw,async,no_subtree_check) 192.168.10.101(rw,async,insecure,no_subtree_check)
/tank/backup 192.168.30.101(rw,async,no_subtree_check) 192.168.10.101(rw,async,insecure,no_subtree_check)

Add the "proto=rdma" and "port=20049" options on the client in /etc/fstab:

192.168.10.100:/tank/storage	/media/mark/storage	nfs4	_netdev,auto,proto=rdma,port=20049,hard,intr,rsize=65536,wsize=65536,noatime	0	0
192.168.10.100:/tank/backup	/media/mark/backup	nfs4	_netdev,auto,proto=rdma,port=20049,hard,intr,rsize=65536,wsize=65536,noatime	0	0

again ensuring to update the IP addresses.

Restart the server and then the client.

Confirm NFS is using RDMA by running from the client:
$ cat /proc/mounts
and verifying the "proto" for the NFS mounts is RDMA.

Known Issues

NFS using RDMA has a documented issue where reading blocks of over 812 bytes will fail starting with around kernel 3.5. A fix is present in 3.16. If using a kernel older than 3.16, the only option is to simply use TCP with IPoIB directly.

See Also