I have recently been testing the pre-release of Oracle Solaris 11, snv_167 to be precise. While Oracle has done well to update documentation surrounding their recent updates to the Solaris 11 networking and virtualization stack, it will still come as a shock to many experienced Solaris administrators that the old methods of configuring IPMP (IP multipathing) using configuration files is no more! Yes, I’m referring to the
/etc/hostname.<interface> files that were so easily understood (and misunderstood in the case of IPMP).
Solaris 11 brings some well overdue changes to how we administer the range of networking. In particular, there are two new commands that effectively end the need for ever using
/usr/sbin/ifconfig again. The first of these is
/usr/sbin/ipadm which is now delivered by the
system/network package. The
ipadm command is very straight-forward and easy to use. It is intended by the developers to eliminate
ifconfig and the
hostname.<interface> files. I must commend them on their efforts since it not only accomplishes their goal, but also simplifies our administration of the network layer.
While there are many ways to configure IPMP, the full-breadth of options will not be discussed here. Instead, I must refer you to the full documentation, Oracle Solaris Administration: Network Interfaces and Network Virtualization guide. This post only covers IPMP in Active-Passive mode. However, in regards to documentation, be certain to read the man page for
ipadm as the man page contains better examples than the guide. It seems the guide lacks updates and still refers to
As background information for my testing, I have created an LDom and assigned it two interfaces (vnet0 and vnet1) which are bound to separate virtual switches (vsw0 and vsw1) on aggregates (aggr1 and aggr2 respectively) composed of separate physical network cards, each of which connect to a separate physical switch. The diagram below will help 😉
Now, to configure IPMP in Active-Passive mode using vnet0 and vnet1 I simply used the following example.
Active-Passive IPMP Example:
# ipadm create-ip vnet0
# ipadm create-ip vnet1
# ipadm set-ifprop –p standby=on –m ip vnet1
# ipadm create-ipmp –i vnet0 –i vnet1 ipmp0
# ipadm create-addr –T static –a local=10.10.10.1/18 ipmp0/v4
Now what actually occurs is quite interesting. The first two commands using the
create-ip subcommand actually perform two tasks. Each command both “plumbs” or enables the network interface and creates an IP interface on that network interface. Next, the
ipadm set-ifprop command marks the vnet1 IP interface as the “STANDBY” interface. The fourth command using
ipadm create-ipmp, performs 3 separate actions. It creates the IPMP group named “ipmp0” and adds both IP interfaces (vnet0 and vnet1) to the IPMP group with the IPMP interface name of “ipmp0”. Finally, the
ipadm create-addr command sets a static IPv4 address to the IPMP interface “ipmp0”. Note the “/v4” syntax appended to the IPMP interface name. This is required for IPv4. Refer to the
ipadm man page if configuring IPMP on an IPv6 network. This results in the following active working configuration.
# ifconfig -a4 lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 ipmp0: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 1500 index 14 inet 10.10.10.1 netmask ffffc000 broadcast 10.10.191.255 groupname ipmp0 vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 17 inet 0.0.0.0 netmask ff000000 groupname ipmp0 ether 0:14:4f:f9:ed:f2 vnet1: flags=61000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,STANDBY,INACTIVE> mtu 1500 index 16 inet 0.0.0.0 netmask ff000000 groupname ipmp0 ether 0:14:4f:f8:66:6d #
There are also two new methods of verifying the new IPMP configuration. The first still uses the
ipadm command and is shown below.
# ipadm show-addr ipmp0/v4
ADDROBJ TYPE STATE ADDR
ipmp0/v4 static ok 10.10.10.1/18
# ipadm show-if -o all
IFNAME CLASS STATE ACTIVE CURRENT PERSISTENT OVER
lo0 loopback ok yes -m-v------46 --46 --
ipmp0 ipmp ok yes bm--------46 --46 vnet0 vnet1
vnet0 ip ok yes bm---l----46 -l46 --
vnet1 ip ok no bm--sli---46 sl46 --
The first command above, using the
show-addr subcommand, displays the IP address previously assigned to the IPMP interface “ipmp0”. The second command using the
show-if subcommand displays all of the interfaces with their assigned “CLASS” and “CURRENT” associated flags. These are explained in detail in the
ipadm man page. However, a little more explanation is warranted here on the “CURRENT” flags. IPMP interface “ipmp0” has “bm——–46” which stands for broadcast, multicast, IPv4, and IPv6 enabled, respectively. IP interfaces “vnet0” and “vnet1” both have flags enabled for broadcast, multicast, link-based probing, IPv4 and IPv6. IP interface “vnet1” has the additional flags of “s” and “i” for standby and inactive, respectively. These last two flags were enabled by the
ipadm set-ifprop –p standby=on –m ip vnet1 command.
Finally, the second new command I mentioned toward the beginning of this post,
ipmpstat command is much more simplified and aimed solely at IPMP configurations. There are several additional options besides the ones shown below, but those are referenced in the man page for
# ipmpstat -g
GROUP GROUPNAME STATE FDT INTERFACES
ipmp0 ipmp0 ok 10.00s vnet0 (vnet1)
# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
vnet0 yes ipmp0 --mbM-- up ok ok
vnet1 no ipmp0 is----- up ok ok
The first command above displays quite simply the contents or IPMP interfaces for the IPMP group named “ipmp0”. The parentheses’ surrounding the IP interface named “vnet1” indicates that as a “STANDBY” interface. The second command,
ipmpstat –i, displays roughly the same information, but with more detail in the “FLAGS” section. Note how “vnet1” shows the same inactive and standby flags as previously displayed using the
ipadm show-if –o all command. However, “vnet0” displays a new flag of “M” for IPv6 multicast. This shows that all multicast communication (both IPv4 and IPv6) is sent via the “ACTIVE” IPMP interface.
There you have it, IPMP without configuration files! However, I must throw out the disclaimer for the geeky Solaris Admins out there. A file does exist at
/etc/ipadm/ipadm.conf that contains any and all information for the active networking configuration created by
ipadm. Also, as usual, Oracle advises not to edit the file directly.