Friday, 1 August 2014

IBM x3950 x6 and shared mode IMM

For a project, we've recently installed an x3950 x6 system.
x3950 x6, image IBM Redbook

Now these systems are very cool in their own right, its an 8U chassis which is basically made up of a pair of x3850 x6 systems. The x3950 takes 8 compute modules with E7v2 cpus and supports up to 12Tb of ram. (not that we have that much!)

Being system x it supports IMM2 for out of band management (IPMI with more stuff). On the rest of system x kit, we run the IMM in shared mode with VLAN tagging where the system attaches the IMM port to the primary Ethernet port and reduces the number of cables we have to run to the system.

Normally you can configure this via the UEFI interface, you go in, switch it to shared from dedicated, set the IP and VLAN tag and that is it.

On the x6 shares mode isn't listed as an option.

To interject here, we don't actually normally configure these by hand as we use xcat to discover and "genesis" the systems. I wont go into xcat here, but for this part it works well for us. The genesis uses ipmitool internally to configure the IMM to run in shared mode, set the IP and VLAN. Except it didn't work on the x6, the IPMI status was listed as "set in progress". From experience, this means its sent something to the IMM that it doesn't understand.

OK, these are new systems and we're running version 1.0 of the UEFI (though why was there a newer 1.0 released without the version number changing??!)

It was at this point I went to check the IMM interface and found that only dedicated mode is available.

A call to our integrator and a call was logged with IBM support.

The engineer I spoke to didn't know what shared mode was, but to be fair after I explained he went off to read up about it.

He did come back with a workaround, and we did find our own workaround as well.

We got it escalated to L2 support who have confirmed it as a bug witha new firmware to fix it expected later this year.

So the workaround...

The first is to set the IMM up in dedicated mode so that you can ssh to the IMM interface, then run
ifconfig eth0 -nic shared_option_1

(You can do all VLAN config etc from this interface as well, see the IMM docs but in short:
ifconfig eth0 -vlan enabled -vlanid 3002 -nic shared_option_1)

The second was to tweak the xcat config. In the ipmi table you can define parameters to determine shared mode. The xcat docs list using 0 in the bmcid field to use shared on the first LAN port. This was what we had set but doesn't work on the x6. In the xcat docs, it lists other options, one is to use "2 0", which means use shared mode on the first port of the mezzanine card. When I set this, I found that genesis completed correctly automatically.

The row we now have in the ipmi table in xcat is:
"x3950x6","|\A|bmc|","2 0","3002",,,,,

And then we add the machine to the x3950x6 xcat group, this auto provisions the IMM on the first shared mezzanine port and assigns it to VLAN 3002.

So now I have an autoconfig solution and the promise of an updated firmware. Yes I could have settled with doing it with work around 1, but then I wouldn't have a solution for any future x6 systems ...

No comments:

Post a comment