At my day job, we have recently started rolling out NexentaStor 3 for our VM image storage as a trial. If all goes well, our long-term plans are to eventually migrate all storage from Netapp to NexentaStor. As we started rolling out our NexentaStor trial, one missing feature we quickly ran across is the lack of IPMP (IP Multipathing) support. The network configuration interface that they provide can currently configure aggregated interfaces with the LACP protocol, but they do not have any mechanism to configure IPMP to aggregate interfaces from multiple switches. We were able to work out an approach to configure IPMP manually, and received Nexenta’s blessing to use it in our environment. (Important note: if you are going to try this on a licensed copy of NexentaStor, please check with your support team to ensure that they are ok with you making these changes.)

[ad name=”Google Adsense 728×90″]

Updated 2010-Sep-16 — Added information on how to add static routes to configure ipmp ping targets

Server hardware configuration

First of all, I should detail what we are trying to configure. Our production machines are quite similar to the SuperMicro build I documented earlier, with a few varying specs. Here’s what is in it:

• 2x Intel E5620 CPUs
• 48G memory (6x 8G modules)
• 10x 1TB Seagate Constellation ES SAS drives
• 6x Intel 32gb X25-E drives
• 2x Crucial 128gb RealSSD drives
• 2x Intel Gigabit ET quad-port NICs

The machine has an 8TB license, with 2 of the disks configured as hot spares. The Intel SSDs are configured as 3 log-mirrors, and the RealSSDs are configured as cache devices.

Caveats

The only major caveat that I’ve hit with this configuration is that the ipmp interfaces will not be viewable via the Nexenta utilities. You can still see all the underlying interfaces; just not the ipmp ones. It’s mostly cosmetic, but is distracting and annoying.

Of course, YMMV – this worked for me, but no guarantees that it will work for you! :)

Network configuration

Desired configuration

Here is the network configuration we desire:

• LACP Aggregate #1 – 4 gigabit links to Switch-1
• LACP Aggregate #2 – 4 gigabit links to Switch-2
• IPMP Interface #1 (balancing LACP1 and LACP2) – Native VLAN on NICs; access to our management/backend network
• IPMP Interface #2 (balancing LACP1 and LACP2) – VLAN ID 100; VM storage network
• IPMP Interface #3 (balancing LACP1 and LACP2) – VLAN ID 200; NFS storage network

General configuration steps

As far as I know, you cannot do VLAN tagging on top of an IPMP trunk on Solaris, which means that we need to create the VLAN interfaces on each of the aggregate interfaces, and then create three separate IPMP interfaces – one per vlan. Here are the basic configuration steps:

1. Via NMC: Create individual LACP aggregates (aggr1 and aggr2), with igbX interfaces as members.
2. Via NMC: Create VLAN interfaces ‘100’ and ‘200’ on top of both ‘aggr1’ and ‘aggr2’. This will create the interfaces ‘aggr100001’, ‘aggr100002’, ‘aggr200001’, and ‘aggr200002’.
3. Via NMC: Configure an IP address from within each VLAN on each of these six interfaces. This will allow IPMP to use ICMP probes in addition to link detection to try to find failed links.
4. Via the console: Configure the three IPMP interfaces, and add the six aggr interfaces to the proper IPMP groups.

NMC – Create LACP aggregates

This assumes that whatever interface you configured during installation is *not* one of the interfaces you desire to be part of the aggregate. If that is not true, you will need to be on the system console (via IPMI hopefully!), and destroy that interface first. Here is an example of how to create the aggregates (output from NMC; so this is how it ends up being configured):

nmc@nexenta:/$ setup network aggregation create
Links to aggregate  : igb2,igb3,igb4,igb5
LACP mode           : active
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L3,L4    auto                 active        short       -----
nmc@nexenta:/$ setup network aggregation create
Links to aggregate  : igb6,igb7,igb8,igb9
LACP mode           : active
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr2           L3,L4    auto                 active        short       -----

NMC – Create vlan interfaces on each aggregate interface

nmc@nexenta:/$ setup network interface aggr1 vlan create
VLAN Id : 100
aggr100001: flags=201000842 mtu 9000 index 39
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3c:de

nmc@nexenta:/$ setup network interface aggr2 vlan create
VLAN Id : 100
aggr100002: flags=201000842 mtu 9000 index 40
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3d:de

nmc@nexenta:/$ setup network interface aggr1 vlan create
VLAN Id : 200
aggr200001: flags=201000842 mtu 9000 index 41
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3e:de 

nmc@nexenta:/$ setup network interface aggr2 vlan create
VLAN Id : 200
aggr200002: flags=201000842 mtu 9000 index 42
        inet 0.0.0.0 netmask 0
        ether 0:1b:21:6c:3f:de

NMC – Assign IP addresses

This assumes the following IP ranges:
Native VLAN: 10.100.0.0/24
VLAN 100: 10.100.100.0/24
VLAN 200: 10.100.200.0/24

It also assumes that aggregate 1 should be assigned .2 within each /24, and aggregate 2 should be assigned .3. The ipmp shared interface will be assigned .1.

nmc@nexenta:/$ setup network interface vlan aggr1 static
aggr1 IP address: 10.100.0.2
aggr1 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr1 as 10.100.0.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr2 static
aggr2 IP address: 10.100.0.3
aggr2 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr2 as 10.100.0.3/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr100001 static
aggr100001 IP address: 10.100.100.2
aggr100001 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr100001 as 10.100.100.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr100002 static
aggr100002 IP address: 10.100.100.3
aggr100002 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr100002 as 10.100.100.3/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr200001 static
aggr200001 IP address: 10.100.200.2
aggr200001 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr200001 as 10.100.200.2/255.255.255.0 ... OK.

nmc@nexenta:/$ setup network interface vlan aggr200002 static
aggr200002 IP address: 10.100.200.3
aggr200002 netmask  : 255.255.255.0
Name Server #1      : 10.0.0.101
Name Server #2      : 10.0.0.102
Name Server #3      :
Gateway IP address  : 172.16.0.254
Enabling aggr200002 as 10.100.200.3/255.255.255.0 ... OK.

Console – Set up IPMP

First, we need to get into expert mode.

nmc@nexenta:/$ options expert_mode=1                                                                          

nmc@nexenta:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). CAUTION: NexentaStor
appliance is not a general purpose operating system: managing the appliance via Unix shell is NOT
recommended. This management console (NMC) is the command-line interface (CLI) of the appliance,
specifically designed for all command-line interactions. Using Unix shell without authorization of your
support provider may not be supported and MAY VOID your license agreement. To display the agreement,
please use 'show appliance license agreement'.
Proceed anyway? (type No to return to the management console)  Yes

root@nexenta:/volumes#

[ad name=”Google Adsense 728×90″]

Next step is to set up the hostname files for each of the IPMP interfaces. I will name the interfaces as follows:

ipmp0 – ipmp interface for aggr1 and aggr2
ipmp100000 – ipmp interface for aggr100001 and aggr100002
ipmp200000 – ipmp interface for aggr200001 and aggr200002

These files also set the IP address that we would like the system to apply to these.

root@nexenta:/etc# cat hostname.ipmp0
ipmp group ipmp0 10.100.0.1 netmask 255.255.255.0 up
root@nexenta:/etc# cat hostname.ipmp100000
ipmp group ipmp100000 10.100.100.1 netmask 255.255.255.0 up
root@nexenta:/etc# cat hostname.ipmp200000
ipmp group ipmp200000 10.100.200.1 netmask 255.255.255.0 up

Next, the groups need to be configured in the hostname. files. We need to add ‘group -failover up’ to each of these files, before the ‘up’ at the end. Here are the files after the changes are made:

root@nexenta:/etc# for i in /etc/hostname.aggr* ; do echo $i ; cat $i ; done
/etc/hostname.aggr1
10.100.0.2 netmask 255.255.255.0 mtu 9000  broadcast +  group ipmp0 -failover up
/etc/hostname.aggr2
10.100.0.3 netmask 255.255.255.0 mtu 9000  broadcast + group ipmp0 -failover up
/etc/hostname.aggr100001
10.100.100.2 netmask 255.255.255.0  broadcast +  group ipmp100000 -failover up
/etc/hostname.aggr100002
10.100.100.3 netmask 255.255.255.0  broadcast + group ipmp100000 -failover up
/etc/hostname.aggr200001
10.100.200.2 netmask 255.255.255.0  broadcast +  group ipmp200000 -failover up
/etc/hostname.aggr200002
10.100.200.3 netmask 255.255.255.0  broadcast +  group ipmp200000 -failover up

Now that all the interface configs are in place, we can apply it.. here’s the easiest way I figured out; if anyone knows a better way I’d love to hear it!

# svcadm disable svc:/network/physical:default
# for i in aggr1 aggr2 aggr100001 aggr100002 aggr200001 aggr200002 ipmp0 ipmp100000 ipmp200000 ; ifconfig $i unplumb ; done
# svcadm enable svc:/network/physical:default

At this point, all of your interfaces should be up, and all the IP addresses should be pingable. Make sure that you can ping the individual interface IPs, and the IPMP IPs. You should be able to use the ‘ipmpstat’ command to see information about your groups; IE:

root@nexenta:/volumes# ipmpstat -a
ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND
nexenta-vl100        up     ipmp100000    aggr100001    aggr100002 aggr100001
nexenta-vl200        up     ipmp200000    aggr200001    aggr200002 aggr200001
nexenta                up     ipmp0       aggr1       aggr2 aggr1

Note that this configuration provides failover and outbound load balancing, but it does not provide inbound load balancing. If you would like inbound load balancing, you need to configure an IP alias on each of the ipmp interfaces, and then mix up which IP you use from the host that is connecting to your Nexenta machine (or use multipathing if it’s iSCSI!)

One last thing – once everything is configured, you will probably want to define your own ping targets. You can view the ones that ipmp picked automatically by running ‘ipmpstat -t’. On our configuration, on one VLAN, two Nexenta nodes picked each other.. so when we took machine two down (intentionally), machine one marked that interface down, and then when we booted machine two back up, it could not reach machine one’s interface, and marked its interface on that vlan down. Nice race condition. Oddly, mpathd (the daemon that does the checking) does not use a configuration file for ping targets, but instead relies on host routes. What we’ve done is added routes to the individual IP addresses that we would like to have it monitor by using the NMC command ‘setup network routes add’, and specifying the IP address to monitor as both the ‘Destination’ and the ‘Gateway’. We picked four to five IPs on each VLAN that were stable hosts (routers, Xen domain-0’s and the like), and added them on both hosts.. this will give more consistent results, as multiple core machines would have to go down before the interface would be disabled incorrectly.

I hope this helps! Please feel free to leave a comment if you run into any trouble getting it working.