VSAN 10G Switch Configuration Cisco 3548 VMWare Distributed Switch Part #2

If you haven't seen my earlier post, the starting point here is 3 VMWare Esxi 5.5u3a VSAN clustered hosts, each with 4 10G nic's. The goal is to get all 3 working with distributed switches for VSAN and VMotion. The first step (in post 1) was to remove the vcenter standard switch from one host, add a distributed switch and configure it for vmotion for that host, and leave the portgroup in passive mode. Thus, we move on to

Cisco Switch Configuration

The cisco nexus 3548 switch runs nx-os, which is pretty easy to learn. It is well documented online, and there are tons of examples. Having said that, I will still show the exact commands and their output.Also, these switches have already been configured as peer-links.

The first thing I did was create a mapping of every port on every host, to each switch port. Some people may be great at documentation and have this on hand from the install, but I used a slower and less efficient method. I shut down each port using

1. conf t
2. interface ethernet 1/5 
3. shutdown

Then I would flip over to vcenter and see which link was disconnected. Then run no shutdown for the same port to turn it back on. Here are my results.
Host/vmnic/switch/switchport

On this host I have 4 NICs. I am currently using vmnic0/5 for VSAN connection. So I'll be working with the orange highlighted.

Well, I just realized something is amiss with this configuration. Just so there aren't inconsistencies with my first post, I'm going to go ahead and address this now before moving on. I have the wrong vmnics grouped together for VSAN and VMotion. The other hosts in my VSAN cluster have vmnic1 and vmnic4 on different switches. This was built in for triple redundancy (cable failure, card failure, and switch failure). However, it looks like I should be grouping together vmnic0/4 and vmnic 1/5 to achieve this. Let me update my vcenter switch and I'll come back. So, in case you are wondering why part #2 of this post has a slightly different configuration than part #1, now you know. (And knowing is half the battle)

Ok, so I changed my adapters, moving forward with vmnic0/4 for Vmotion on vsphere distributed switch.

I like to bring up two ssh sessions, put cmcdrswitch1 on the left and cmcdrswitch2 on the right. It is easy to get confused.

Here is a screenshot of the command show interface brief for the ports we'll be changing

cmcdrswitch 1 show interface brief

On this switch we will be working with interface ethernet 1/2. This is the pretty much the default config. I think the only thing I changed was the VLAN id when initially setting this up.

To see the running config for a port run

cmcdrswitch 1 show running-config interface ethernet 1/2

I will run the following commands to setup the port-channel

1. conf t
2. interface ethernet 1/2
3. switchport mode trunk
4. switchport access vlan 15
5. channel-group 22 mode active
6. no shutdown

Now if you run show interface brief again, you can see the changes to the port. (will show Mode as trunk and Port CH# as 22).

And if you run show port-channel summary, you should see port-channel 22.

So, what have we done? Nothing yet. We need to do the exact same thing on switch2 for interface ethernet 1/1. 

I'll save the verbosity of repeating the steps above. Here is the result on switch2.

Surely this is an accomplishment we can boast of? Nope. We've just setup port-channeling on two different switches. Now is when the magic happens. on both switches, run the commands

1. conf t
2. interface port-channel 22 
3. vpc 22

 Now run: show vpc brief on each switch to verify that it is up and running.

Now we've really done something. The vsphere distributed switch (with lacp enabled) will utilize both of these links for load balancing and fail over. This is quite different and way better than"teaming" in a vsphere standard switch. If you want the technical answer you can Google it, or you can just take my word for it.


Lastly, I need to run a few vmotions to see if this is even working right. Don't forget to change the portgroup mode to Active. And be sure all of the MTU's match between vmotion switches, etc. I have everything set to Jumbo Frames (9000) even though I don't think it makes much of a difference for VSAN.

Next up, Testing and Validation.