Cisco SFS-HCA-320-A1

From Mark Furneaux's Wiki
Jump to: navigation, search

The SFS-HCA-320-A1 is an InfiniBand host channel adapter (HCA) by Cisco. It uses the Mellanox MT25204 chipset and supports up to a single x4 DDR link running at 20Gbps and connects using PCI Express x8.

Cisco SFS-HCA-320-A1.jpg

Overview

  • Link Types: x1 SDR, x4 SDR, x1 DDR, x4 DDR, full duplex
  • Link Speeds: 2.5Gbps, 5Gbps, 10Gbps, 20Gbps
  • Connector Type: CX4 Latch Style
  • Connection: PCIe x8
  • Power Consumption: 4W
  • Onboard Memory: None
  • OS Support: Linux, Windows, BSD

Fixes

Power Off

On some PC hardware running Linux, the card can prevent an ACPI power off event and will cause the machine to hang on shutdown.
The simplest workaround is to unload the kernel module just before the kernel sends the ACPI power off command, which will shut down the card and prevent it from interfering with the shutdown.

Create the file /etc/rc0.d/S89ibunload containing the following:

#! /bin/sh

rmmod ib_mthca

Driver Lockups

The driver for this card does not support GFP_NOIO, and as such it has the possibility of encountering a race condition when allocating memory in the kernel. This will result in a deadlock. There is no fix, and if this is encountered, the only solution is to replace the card for a newer model not using the ib_mthca driver.

Firmware

The card ships with a custom Cisco firmware which is proprietary and is most likely out of date since Cisco stopped manufacturing the card around version 1.1.0 and version 1.2.0 is the latest. Since the card runs a standard Mellanox chipset, it can be safely flashed with a stock Mellanox firmware. The SFS-HCA-320-A1 is actually a Mellanox Cheetah MHGS18-XSC rev A1.

Flashing The Firmware

The MST flint utility is required to burn the firmware. Due to the age of this card, the current version of the utility from Mellanox is not compatible. The legacy version of the utility that Mellanox offers does not come in a Debian package and therefore cannot be used on Ubuntu. One solution is to use the mstflint utility available in the official repositories.

Install mstflint

Get mstflint from the Ubuntu repositories with:
# apt-get install mstflint

Get The Firmware Binary

  1. Download the firmware from Mellanox's website for the MT_03D0110002 PSID.
  2. Extract the .bin file from the archive.

Locate The HCA

Find the location of the card on the PCI bus by running:
$ lspci
The address is of the form X:XX.X

Burn The Firmware

Run mstflint as shown below:
# mstflint -d 02:00.0 -i ~/Documents/fw-25204-1_2_000-MHGS18-XSC_A1.bin -skip_is burn
02:00.0 - replace with the location of the card as determined by lspci.
-skip_is - Cisco uses an abnormal Invariant Sector on the card to deter this type of firmware flashing. You can ignore the warning generated by mstflint and burn anyways with this flag.

Clean Up

To ensure that everything went okay and the card is now running the new firmware, turn off the computer and remove power from the card to hard reset it.

The firmware version can then be checked using:
# ibstat

See Also