VSAN 10G Switch Configuration Cisco 3548 SFP+

VSAN Switch Configuration. 

Part One: Physical Switch

3 Hosts, 4 10GB connections per host
2 Cisco 3548 Nexus Switches configured for failover (VPC)
Hosts using VCenter Standard Switch

Initially I got all my switches setup and plugged everything in, only to find that my datastore was segregated into three separate "Network Partition Groups". I dug around a little and found out that this had to do with the IGMP settings on my switch. VMWare uses some multicast  addresses for this

224.1.2.3 
224.2.3.4

If igmp snooping is not setup correctly on your switches, you will see this error

"Network status: Misconfiguration detected" and will most likely see Group 1, Group 2, Group 3 in the Network Partition Groups column of VSAN Disk Management. They should all be in the same group.















The switches we are using run NX-OS, here is the result from query

show running-config | grep igmp

In addition to changing the IGMP settings, I also had to enable Virtual Port Channeling for switch failover. Here is shown the output of

show interface brief

 

 All of the host network interfaces are connected to Eth1/1-Eth1/6. Eth1/15 and Eth1/16 are connected from switchA to swtichB. 

Running the command show vpc brief  on each switch, will let you know if you've configured it properly. Here is the output from both switches

Part Two: Virtual Switch (VCenter Standard Switch)

All the documentation out there points to the VCenter Distributed Switch as the recommended implementation for VSAN. To keep things simple I decided to start first with a VCenter Standard Switch, and move to the distributed switch once I was up and running.

Each Host has 4 10GB Nics. 2 cards with 2 ports each. In order they are vmnic0, 1 and vmnic 4,5.  

First, I decided to pair vmnic0 and vmnic5 into a standard switch for VSAN traffic, and vmnic1 and vmnic4 into a separate switch dedicated to VMotion traffic. For redundancy, each pair of 10G connections are routed to a separate switch. Thus, the failure of a single link, network card, or physical switch will not undermine the integrity of my two 10G networks. 

So far the performance is quite good. I can't imagine the Distributed Switch being much better, but in time I will get there and have some benchmarks ready for comparison. 

Thanks for reading.