Hello vComunnity,
First and foremost do not panic, let’s identify if our vSAN cluster is partitioned, one of the easiest ways to do this is by going to our vCenter and checking the Skyline Health.
A non-partitioned cluster should look like this:
A partitioned cluster looks like this:
If your vCenter was inside the cluster and is no longer reachable, you can perform a health check within any of the ESXi nodes by running.
[[email protected]:~] esxcli vsan health cluster list Health Test Name Status -------------------------------------------------- ---------- Overall health green (OK) Cluster green ESXi vSAN Health service installation green vSAN Health Service up-to-date green Advanced vSAN configuration in sync green vSAN daemon liveness green vSAN Disk Balance green Resync operations throttling green Software version compatibility green Disk format version green Network green Hosts with connectivity issues green vSAN cluster partition green All hosts have a vSAN vmknic configured green vSAN: Basic (unicast) connectivity check green vSAN: MTU check (ping with large packet size) green vMotion: Basic (unicast) connectivity check green vMotion: MTU check (ping with large packet size) green Network latency check green Data green vSAN object health green Capacity utilization green Disk space green Read cache reservations green Component green What if the most consumed host fails green Physical disk green Operation health green Disk capacity green Congestion green Component limit health green Component metadata health green Memory pools (heaps) green Memory pools (slabs) green Performance service green Stats DB object green Stats master election green Performance data collection green All hosts contributing stats green Stats DB object conflicts green
more specifically:
[[email protected]:~] esxcli vsan health cluster get -t "vSAN cluster partition" vSAN cluster partition green Checks if the vSAN cluster is partitioned due to a network issue. Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.clusterpartition Partition list Host Partition Host UUID ------------------------------------------------------------------------- 172.168.11.53 1 611d0e3d-d1d8-de43-7c12-00505683393a 172.168.11.51 1 611d0e3d-d3dd-b16c-2107-005056831525 172.168.11.52 1 611d0edc-aeee-6efd-7011-00505683e982 172.168.11.54 1 611d0edd-e952-d9b8-4951-00505683749b
Running the command from the partitioned node will provide only its partition info:
[[email protected]:~] esxcli vsan health cluster get -t "vSAN cluster partition" vSAN cluster partition red Checks if the vSAN cluster is partitioned due to a network issue. Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.clusterpartition Partition list Host Partition Host UUID ------------------------------------------------------------------------------------ 172.168.11.51 1 611d0e3d-d3dd-b16c-2107-005056831525 172.168.11.54 Partition is unknown 611d0edd-e952-d9b8-4951-00505683749b 172.168.11.53 Partition is unknown 611d0e3d-d1d8-de43-7c12-00505683393a 172.168.11.52 Partition is unknown 611d0edc-aeee-6efd-7011-00505683e982
The rest of the hosts that are in a single partition will show the partitioned node, in this case esx01
[[email protected]:~] esxcli vsan health cluster get -t "vSAN cluster partition" vSAN cluster partition red Checks if the vSAN cluster is partitioned due to a network issue. Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.clusterpartition Partition list Host Partition Host UUID ------------------------------------------------------------------------------------ 172.168.11.53 1 611d0e3d-d1d8-de43-7c12-00505683393a 172.168.11.52 1 611d0edc-aeee-6efd-7011-00505683e982 172.168.11.54 1 611d0edd-e952-d9b8-4951-00505683749b 172.168.11.51 Partition is unknown 611d0e3d-d3dd-b16c-2107-005056831525
If you confirm that the vSAN partitioned is occurring, now it is time to review some possible causes and how to fix them.
- Networking: Even though it is not my favorite topic it is important to know enough networking to determine if it is the root cause of our problem, keep in mind that vSAN uses networking to operate properly if our networking is not good and will affect vsan operations.
1 – Make sure that your physical networking is not affecting the active-standby uplink used for vsan: I have particularly seen vlan changes made by networking engineers and they do not notify the vSphere administrator about it or faulty or disconnected uplinks.
Validate which uplinks/vmnics vSAN uses and check if they are up, in this example vSAN uses uplink3 (vmnic2) and uplink4 (vmnic3) as active/standby,
Go to the partitioned node and check if these uplinks are up (here they were down causing the partitioning even from a networking standpoint.
2 – VLAN changes: If you are planning to change your vSAN’s VLAN, make sure that your new VLAN exists in your switches first so you avoid taking down your vSAN operations during the change of this value.
My recommendation for this is to double-check if your uplinks are passing vSAN traffic through the VLAN
Let’s enable the VLAN stats to the interface we want to monitor and then confirm if packets are being sent and received
[[email protected]:~] esxcli network nic vlan stats set -e true -n vmnic1 [[email protected]:~] esxcli network nic vlan stats get -n vmnic1 VLAN 0 Packets received: 410 Packets sent: 840 VLAN 200 <----- vSAN VLAN Packets received: 64539 Packets sent: 31553 VLAN 201 Packets received: 7708 Packets sent: 3760
When a non working VLAN is used it looks like this:
[[email protected]:~] esxcli network nic vlan stats get -n vmnic1 VLAN 0 Packets received: 9013 Packets sent: 14389 VLAN 200 <------ A very low amount of packets transmitted, this is not normal for vSAN Packets received: 4 Packets sent: 102 VLAN 201 Packets received: 4 Packets sent: 2
Another personal favorite is the tcpdump-uw test, which this I can validate if the vSAN communication is occurring, use vSAN’s vmk for the test
[[email protected]:~] tcpdump-uw -i vmk3 port 12321 tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode listening on vmk3, link-type EN10MB (Ethernet), capture size 262144 bytes 18:29:35.235210 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200 18:29:57.163498 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472 18:30:06.410854 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200 18:30:06.410937 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200 18:30:06.410963 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472 18:30:06.410970 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200 18:30:06.410977 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200 18:30:06.411006 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200 18:30:06.411087 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472 18:30:06.411093 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200 18:30:06.411099 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200 18:30:06.934927 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472 18:30:06.944259 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200 18:30:06.944317 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200 18:30:07.237912 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200 18:30:07.935050 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472 18:30:07.944428 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200 18:30:07.944498 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200 18:30:08.238002 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200 19 packets captured 19 packets received by filter 0 packets dropped by kernel
Here you can see that esxi2 talks to esxi3, esxi4 and vice versa, but we don’t see esxi1
172.168.11.54 > esxi4
172.168.11.53 > esxi3
172.168.11.52 > esxi2
If the communication is good, you’ll see 172.168.11.51 > esxi1
[[email protected]:~] tcpdump-uw -i vmk3 port 12321 tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode listening on vmk3, link-type EN10MB (Ethernet), capture size 262144 bytes 18:33:54.952684 IP 172.168.11.53.12321 > 172.168.11.51.12321: UDP, length 504 18:34:17.133393 IP 172.168.11.54.12321 > 172.168.11.51.12321: UDP, length 200 18:34:27.135761 IP 172.168.11.51.12321 > 172.168.11.53.12321: UDP, length 200 18:34:27.135844 IP 172.168.11.51.12321 > 172.168.11.52.12321: UDP, length 200 <---- esxi1 talking to esxi2 18:34:27.135851 IP 172.168.11.53.12321 > 172.168.11.51.12321: UDP, length 504 18:34:27.135859 IP 172.168.11.54.12321 > 172.168.11.51.12321: UDP, length 200 18:34:27.135865 IP 172.168.11.52.12321 > 172.168.11.51.12321: UDP, length 200 <---- esxi1 talking to esxi2 18:34:27.135872 IP 172.168.11.51.12321 > 172.168.11.54.12321: UDP, length 200 18:34:27.135883 IP 172.168.11.53.12321 > 172.168.11.51.12321: UDP, length 504 18:34:27.135889 IP 172.168.11.54.12321 > 172.168.11.51.12321: UDP, length 200
- Incorrect unicast agent entry: This was a thing when people used to have mismatched vCenter/ESXi versions, like older vCenters than the ESXi,
[[email protected]:~] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint ------------------------------------ --------- ---------------- ------------- ----- ---------- ----------------------------------------------------------- 611d0e3d-d3dd-b16c-2107-005057942525 0 true 172.168.11.51 12321 84:D2:E9:67:49:9E:B9:5C:C8:40:37:4F:F3:EA:80:F2:6C:FB:04:50 611d0e3d-d1d8-de43-7c12-00505683393a 0 true 172.168.11.53 12321 69:BC:06:4F:21:6A:45:30:EC:D2:AB:43:AE:89:66:81:94:F8:F1:DB 611d0edd-e952-d9b8-4951-00505683749b 0 true 172.168.11.54 12321 A9:83:84:F0:07:AB:7F:28:20:59:3E:85:DC:55:1F:EF:F4:C5:B3:C3
By running the command cmmds-tool whoami you’ll be able to get the actual node UUID used for the unicastagent list, compare the Node UUIDs with the list.
[[email protected]:~] cmmds-tool whoami 611d0e3d-d3dd-b16c-2107-005056831525 [[email protected]:~] cmmds-tool whoami 611d0edc-aeee-6efd-7011-00505683e982 [[email protected]:~] cmmds-tool whoami 611d0e3d-d1d8-de43-7c12-00505683393a [[email protected]:~] cmmds-tool whoami 611d0edd-e952-d9b8-4951-00505683749b
You have to be very careful and review the entry, if you did you’ll see that the UUID from esxi1 is not correct on the esx02‘s unicastagent list output, this can be easily corrected by following VMware’s KB Configuring vSAN Unicast networking from the command line (2150303)
- Incorrect shutdown: A loss of electrical power can occur and your UPS won’t last longer to allow you to properly shut down your cluster, let me tell you that if you don’t have enough time to properly shutdown your vSAN cluster, then, only take care of the production you have running there (gracefully shutdown all VMs / Preferably) and let the host go off, I know it might sound terrible, but starting a vSAN shutdown that might not be able to finish could leave you in a worse situation.
1 – vSAN DecomState: You decided to take the risk and will start putting hosts in maintenance mode (MM) no data migration of course and you ran out of time, when the power energy comes back your VMs are shown with an incorrect number or might not be even there, and of course, your node does not seem to be in MM anyways but guess what, it is likely that your node is in vSAN MM and while in that state, it won’t provide any vSAN storage until is taken out.
If your vCenter is online you can easily identify if a host is in DecomState by checking the Skyline Health
If your vCenter is down because it was inside the cluster, you can run the following command esxcli vsan cluster get and look for Maintenance Mode State: ON
[[email protected]:~] esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2021-10-23T18:18:38Z Local Node UUID: 611d0e3d-d3dd-b16c-2107-005056831525 Local Node Type: NORMAL Local Node State: AGENT Local Node Health State: HEALTHY Sub-Cluster Master UUID: 611d0e3d-d1d8-de43-7c12-00505683393a Sub-Cluster Backup UUID: 611d0edd-e952-d9b8-4951-00505683749b Sub-Cluster UUID: 526e0e30-c21d-f2a5-cd83-be3aeb61f99b Sub-Cluster Membership Entry Revision: 5 Sub-Cluster Member Count: 4 Sub-Cluster Member UUIDs: 611d0e3d-d1d8-de43-7c12-00505683393a, 611d0edd-e952-d9b8-4951-00505683749b, 611d0edc-aeee-6efd-7011-00505683e982, 611d0e3d-d3dd-b16c-2107-005056831525 Sub-Cluster Member HostNames: esx03.local, esx04.local, esx02.local, esx01.local Sub-Cluster Membership UUID: c4757261-6a14-e410-8842-00505683393a Unicast Mode Enabled: true Maintenance Mode State: ON <------ DecomState Config Generation: e99835c2-8bfd-41bc-98b7-d965e5f33a97 8 2021-10-23T17:45:30.600
To exit this state you can run localcli vsan maintenancemode cancel so you don’t have to put the host in maintenance mode and exit
[[email protected]:~] localcli vsan maintenancemode cancel VsanUtil: AffObj count: 0 [[email protected]:~] esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2021-10-23T18:25:22Z Local Node UUID: 611d0e3d-d3dd-b16c-2107-005056831525 Local Node Type: NORMAL Local Node State: AGENT Local Node Health State: HEALTHY Sub-Cluster Master UUID: 611d0e3d-d1d8-de43-7c12-00505683393a Sub-Cluster Backup UUID: 611d0edd-e952-d9b8-4951-00505683749b Sub-Cluster UUID: 526e0e30-c21d-f2a5-cd83-be3aeb61f99b Sub-Cluster Membership Entry Revision: 5 Sub-Cluster Member Count: 4 Sub-Cluster Member UUIDs: 611d0e3d-d1d8-de43-7c12-00505683393a, 611d0edd-e952-d9b8-4951-00505683749b, 611d0edc-aeee-6efd-7011-00505683e982, 611d0e3d-d3dd-b16c-2107-005056831525 Sub-Cluster Member HostNames: esx03.local, esx04.local, esx02.local, esx01.local Sub-Cluster Membership UUID: c4757261-6a14-e410-8842-00505683393a Unicast Mode Enabled: true Maintenance Mode State: OFF <------ No longer in DecomState Config Generation: e99835c2-8bfd-41bc-98b7-d965e5f33a97 8 2021-10-23T17:45:30.600
- Errors during troubleshooting: Yep, we are all humans, and maybe during the heat of the situation they might make some mistakes, those mistakes can be related to HW/Networking or logical troubleshooting, so you can always review what we already explained to see if somebody made a mistake.
Hope this was a cool post for you guys to know,
Do now hesitate in contacting me if you have any comments,
Jorluis
