IPMP on Solaris 11

I have recently been testing the pre-release of Oracle Solaris 11, snv_167 to be precise. While Oracle has done well to update documentation surrounding their recent updates to the Solaris 11 networking and virtualization stack, it will still come as a shock to many experienced Solaris administrators that the old methods of configuring IPMP (IP multipathing) using configuration files is no more! Yes, I’m referring to the /etc/hostname.<interface> files that were so easily understood (and misunderstood in the case of IPMP).

Solaris 11 brings some well overdue changes to how we administer the range of networking. In particular, there are two new commands that effectively end the need for ever using /usr/sbin/ifconfig again. The first of these is /usr/sbin/ipadm which is now delivered by the system/network package. The ipadm command is very straight-forward and easy to use. It is intended by the developers to eliminate ifconfig and the hostname.<interface> files. I must commend them on their efforts since it not only accomplishes their goal, but also simplifies our administration of the network layer.

While there are many ways to configure IPMP, the full-breadth of options will not be discussed here. Instead, I must refer you to the full documentation, Oracle Solaris Administration: Network Interfaces and Network Virtualization guide. This post only covers IPMP in Active-Passive mode. However, in regards to documentation, be certain to read the man page for ipadm as the man page contains better examples than the guide. It seems the guide lacks updates and still refers to ifconfig over ipadm.

As background information for my testing, I have created an LDom and assigned it two interfaces (vnet0 and vnet1) which are bound to separate virtual switches (vsw0 and vsw1) on aggregates (aggr1 and aggr2 respectively) composed of separate physical network cards, each of which connect to a separate physical switch. The diagram below will help 😉

IPMP Configuration Diagram in an LDom using Link Aggregation

Now, to configure IPMP in Active-Passive mode using vnet0 and vnet1 I simply used the following example.

Active-Passive IPMP Example:
# ipadm create-ip vnet0
# ipadm create-ip vnet1
# ipadm set-ifprop –p standby=on –m ip vnet1
# ipadm create-ipmp –i vnet0 –i vnet1 ipmp0
# ipadm create-addr –T static –a local= ipmp0/v4

Now what actually occurs is quite interesting. The first two commands using the create-ip subcommand actually perform two tasks. Each command both “plumbs” or enables the network interface and creates an IP interface on that network interface. Next, the ipadm set-ifprop command marks the vnet1 IP interface as the “STANDBY” interface. The fourth command using ipadm create-ipmp, performs 3 separate actions. It creates the IPMP group named “ipmp0” and adds both IP interfaces (vnet0 and vnet1) to the IPMP group with the IPMP interface name of “ipmp0”. Finally, the ipadm create-addr command sets a static IPv4 address to the IPMP interface “ipmp0”. Note the “/v4” syntax appended to the IPMP interface name. This is required for IPv4. Refer to the ipadm man page if configuring IPMP on an IPv6 network. This results in the following active working configuration.

# ifconfig -a4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet netmask ff000000
ipmp0: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 1500 index 14
        inet netmask ffffc000 broadcast
        groupname ipmp0
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 17
        inet netmask ff000000
        groupname ipmp0
        ether 0:14:4f:f9:ed:f2
vnet1: flags=61000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,STANDBY,INACTIVE> mtu 1500 index 16
        inet netmask ff000000
        groupname ipmp0
        ether 0:14:4f:f8:66:6d

There are also two new methods of verifying the new IPMP configuration. The first still uses the ipadm command and is shown below.

# ipadm show-addr ipmp0/v4
ADDROBJ           TYPE     STATE        ADDR
ipmp0/v4          static   ok 
# ipadm show-if -o all
lo0        loopback ok       yes    -m-v------46  --46       --
ipmp0      ipmp     ok       yes    bm--------46  --46       vnet0 vnet1
vnet0      ip       ok       yes    bm---l----46  -l46       --
vnet1      ip       ok       no     bm--sli---46  sl46       --

The first command above, using the show-addr subcommand, displays the IP address previously assigned to the IPMP interface “ipmp0”. The second command using the show-if subcommand displays all of the interfaces with their assigned “CLASS” and “CURRENT” associated flags. These are explained in detail in the ipadm man page. However, a little more explanation is warranted here on the “CURRENT” flags. IPMP interface “ipmp0” has “bm——–46” which stands for broadcast, multicast, IPv4, and IPv6 enabled, respectively. IP interfaces “vnet0” and “vnet1” both have flags enabled for broadcast, multicast, link-based probing, IPv4 and IPv6. IP interface “vnet1” has the additional flags of “s” and “i” for standby and inactive, respectively. These last two flags were enabled by the ipadm set-ifprop –p standby=on –m ip vnet1 command.

Finally, the second new command I mentioned toward the beginning of this post, /usr/sbin/ipmpstat. The ipmpstat command is much more simplified and aimed solely at IPMP configurations. There are several additional options besides the ones shown below, but those are referenced in the man page for ipmpstat.

# ipmpstat -g
ipmp0       ipmp0       ok        10.00s    vnet0 (vnet1)
# ipmpstat -i
vnet0       yes     ipmp0       --mbM--   up        ok        ok
vnet1       no      ipmp0       is-----   up        ok        ok

The first command above displays quite simply the contents or IPMP interfaces for the IPMP group named “ipmp0”. The parentheses’ surrounding the IP interface named “vnet1” indicates that as a “STANDBY” interface. The second command, ipmpstat –i, displays roughly the same information, but with more detail in the “FLAGS” section. Note how “vnet1” shows the same inactive and standby flags as previously displayed using the ipadm show-if –o all command. However, “vnet0” displays a new flag of “M” for IPv6 multicast. This shows that all multicast communication (both IPv4 and IPv6) is sent via the “ACTIVE” IPMP interface.

There you have it, IPMP without configuration files! However, I must throw out the disclaimer for the geeky Solaris Admins out there. A file does exist at /etc/ipadm/ipadm.conf that contains any and all information for the active networking configuration created by ipadm. Also, as usual, Oracle advises not to edit the file directly.

Happy networking.


Brad Hudson is an Established Leader in IT Infrastructure, Engineering, Operations, and Customer Service with extensive experience developing and delivering infrastructure solutions, building and growing customer service organizations, leading change initiatives, providing data center management, and managing implementation activities.

Tagged with: , , , , , ,
Posted in LDoms, Oracle, Solaris 11
About the Author

Brad Hudson is an Established Leader in IT Infrastructure, Engineering, Operations, and Customer Service with extensive experience developing and delivering infrastructure solutions, building and growing customer service organizations, leading change initiatives, providing data center management, and managing implementation activities.[Read More...]

%d bloggers like this: