VSAN 10G Switch Configuration Cisco 3548 VMWare Distributed Switch Part #1

Best practices for VMWare VSAN call for using the vcenter distributed switch. It's a funny thing about licensing, because I actually didn't know that I was able to use the distributed switch until I read around a bit on the internet. I knew that our version of vcenter (enterprise, NOT enterprise plus) wasn't licensed for the Distributed Switch, but when we purchased our VSAN licenses, the Distributed switch was included with it. So, there you have it.

I have to admit, I was a bit anxious about setting this up properly. First of all, I'm not a networking expert. I had done some remedial switch config to get the VSAN up, but Etherchannels, port-channels, VPCs and the like were not exactly my strong suite. However, I did manage to setup my two cisco nexus 3548 switches as peer links, so I had a working knowledge of the basics.

Just as a quick review, the starting point for this endeavor is 3 VMWare Esxi 5.5u3a hosts, with 4 10Gig connections each. Currently, they are all connected using the VMWare Standard Switch. I hope to nail down the correct configuration to have all connections setup through the distributed switch.

I'm sure VMWare mentions it somewhere, but migrating from standard switch to distributed switch seems like a lot of work. Do I configure the switch first, then the hosts? How do I test it? How do I know if it is working right?

I'm starting off with two Vcenter standard switches on each host. One is setup for VSAN traffic, and one is setup for VMotion traffic. I had a heck of a time configuring the VSAN initially, so to mitigate any issues with the VSAN, I'm going to experiment with the vmotion switch. If I mess this up, at least my VSAN config will still be working.

So my plan of attack is as follows:

1. Setup the distributed switch on a single host for vmotion
2. Configure the cisco switch for LACP/port channeling for the two connections 
3. Verify storage vmotion capability for distributed switch host
4. Move all Hosts vmotion networking to distributed switch
5. Move all Hosts VSAN networking to distributed switch* 

*If I'm lucky, I'll get through steps 1-4 in this post and save #5 for later.

Vcenter Configuration (step 1)

First I removed the vcenter standard switch with vmnic0 and vmnic4 from my hosts network configuration section.

Next I created a VCenter Distributed Switch via the web client.

On the left hand nav bar select VCenter->Distributed Switches. On the right pane click the + button for Create New Distributed Switch.

I chose a really lame lame, and specified Distributed Switch version 5.5. I want the best LACP has to offer!

On the number of uplinks I changed this to 6 (2 per host), however, I think this is configurable later, so I'm not sure it really matters. I also selected Default port group and named it vmotionportgroup2 (how unoriginal!). And last but not least, click Finish.

Here is what it looks like in web client. Nothing groundshaking here. 

Next. Go to the switch and select Manage->Settings. Here, I selected the LACP tab and added a new Link Aggregation Group. For lack of a creative name, I called mine vmotionLag, 6 ports, defaults for the rest, and chose Passive for the mode. (This means it is off, once I configure the switchports, I can come back and turn this to Active.

There are a number of ways of adding a host to a vcenter distributed switch (or is it vice versa?), but I am going to use the easiest way I can think of, the wizard. From Vcenter web client, select the distributed switch and click the Actions Tab on the right pane. From there select Add and manage Hosts. 

1. Add hosts
2. New hosts...
3. <choose one host>, remember, I want to get distributed switch working on just one host for now. KISS!
4. Next..
5. Check the first two boxes for Manage physical adapters & Manage VMKernel adapters
6. Next.

Shown above, vmnic1 and vmnic4 are free. VMWare makes it pretty easy by showing which vmnic's are free. On this step, DON'T click next until you have setup each uplink. 

7. Highlight one of the free vmnic's. (vmnic1)
8. Click the Assign Uplink button

 

9. Since we enabled LACP we have the vmotionLag2 ports, don't use the regular uplinks.
10. Do the same thing for vmnic4.

Here is the end result. 

11. Click Next. This takes you to the Manage VMKernel Network Adapters page. As you can see I have my original vmk0 and vmk1 (my other 10G connections I'm keeping on standard switch for now). I want to create vmk2 which will use the distributed switch. 

12. Click New adapter
13. In step#1, browse for adpater type, choose vmotionportgroup2
14. In step#2, enable vMotion traffic checkbox
15. In step#3, assign a statpic IPv4 address/Mask
16. Finish

17. Next
18. Next again
19. Finish (Finally!)

Ok, just want to hop back to my host view / networking, and make sure everything is there. Everything Looks ok, if you notice my vmotionportgroup2's virtual Link Light is not lit up...this is still in passive mode. Will have to come back and change this to Active once the cisco switches are setup.

Part #2, Switch Configuration