Hello vComunnity,

First and foremost do not panic, let’s identify if our vSAN cluster is partitioned, one of the easiest ways to do this is by going to our vCenter and checking the Skyline Health.

A non-partitioned cluster should look like this:

A partitioned cluster looks like this:

If your vCenter was inside the cluster and is no longer reachable, you can perform a health check within any of the ESXi nodes by running.

[[email protected]:~] esxcli vsan health cluster list
Health Test Name                                    Status
--------------------------------------------------  ----------
Overall health                                      green (OK)
Cluster                                             green
  ESXi vSAN Health service installation             green
  vSAN Health Service up-to-date                    green
  Advanced vSAN configuration in sync               green
  vSAN daemon liveness                              green
  vSAN Disk Balance                                 green
  Resync operations throttling                      green
  Software version compatibility                    green
  Disk format version                               green
Network                                             green
  Hosts with connectivity issues                    green
  vSAN cluster partition                            green
  All hosts have a vSAN vmknic configured           green
  vSAN: Basic (unicast) connectivity check          green
  vSAN: MTU check (ping with large packet size)     green
  vMotion: Basic (unicast) connectivity check       green
  vMotion: MTU check (ping with large packet size)  green
  Network latency check                             green
Data                                                green
  vSAN object health                                green
Capacity utilization                                green
  Disk space                                        green
  Read cache reservations                           green
  Component                                         green
  What if the most consumed host fails              green
Physical disk                                       green
  Operation health                                  green
  Disk capacity                                     green
  Congestion                                        green
  Component limit health                            green
  Component metadata health                         green
  Memory pools (heaps)                              green
  Memory pools (slabs)                              green
Performance service                                 green
  Stats DB object                                   green
  Stats master election                             green
  Performance data collection                       green
  All hosts contributing stats                      green
  Stats DB object conflicts                         green

more specifically:

[[email protected]:~] esxcli vsan health cluster get -t "vSAN cluster partition"
vSAN cluster partition        green

Checks if the vSAN cluster is partitioned due to a network issue.
Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.clusterpartition

Partition list
Host              Partition     Host UUID
-------------------------------------------------------------------------
172.168.11.53     1             611d0e3d-d1d8-de43-7c12-00505683393a
172.168.11.51     1             611d0e3d-d3dd-b16c-2107-005056831525
172.168.11.52     1             611d0edc-aeee-6efd-7011-00505683e982
172.168.11.54     1             611d0edd-e952-d9b8-4951-00505683749b

Running the command from the partitioned node will provide only its partition info:

[[email protected]:~] esxcli vsan health cluster get -t "vSAN cluster partition"

vSAN cluster partition        red

Checks if the vSAN cluster is partitioned due to a network issue.
Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.clusterpartition

Partition list
Host              Partition                Host UUID
------------------------------------------------------------------------------------
172.168.11.51     1                        611d0e3d-d3dd-b16c-2107-005056831525
172.168.11.54     Partition is unknown     611d0edd-e952-d9b8-4951-00505683749b
172.168.11.53     Partition is unknown     611d0e3d-d1d8-de43-7c12-00505683393a
172.168.11.52     Partition is unknown     611d0edc-aeee-6efd-7011-00505683e982

The rest of the hosts that are in a single partition will show the partitioned node, in this case esx01

[[email protected]:~] esxcli vsan health cluster get -t "vSAN cluster partition"
vSAN cluster partition        red

Checks if the vSAN cluster is partitioned due to a network issue.
Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.clusterpartition

Partition list
Host              Partition                Host UUID
------------------------------------------------------------------------------------
172.168.11.53     1                        611d0e3d-d1d8-de43-7c12-00505683393a
172.168.11.52     1                        611d0edc-aeee-6efd-7011-00505683e982
172.168.11.54     1                        611d0edd-e952-d9b8-4951-00505683749b
172.168.11.51     Partition is unknown     611d0e3d-d3dd-b16c-2107-005056831525

If you confirm that the vSAN partitioned is occurring, now it is time to review some possible causes and how to fix them.

  • Networking: Even though it is not my favorite topic it is important to know enough networking to determine if it is the root cause of our problem, keep in mind that vSAN uses networking to operate properly if our networking is not good and will affect vsan operations.

1 – Make sure that your physical networking is not affecting the active-standby uplink used for vsan: I have particularly seen vlan changes made by networking engineers and they do not notify the vSphere administrator about it or faulty or disconnected uplinks.

Validate which uplinks/vmnics vSAN uses and check if they are up, in this example vSAN uses uplink3 (vmnic2) and uplink4 (vmnic3) as active/standby,

Go to the partitioned node and check if these uplinks are up (here they were down causing the partitioning even from a networking standpoint.

2 – VLAN changes: If you are planning to change your vSAN’s VLAN, make sure that your new VLAN exists in your switches first so you avoid taking down your vSAN operations during the change of this value.

My recommendation for this is to double-check if your uplinks are passing vSAN traffic through the VLAN

Let’s enable the VLAN stats to the interface we want to monitor and then confirm if packets are being sent and received

[[email protected]:~] esxcli network nic vlan stats set -e true -n vmnic1
[[email protected]:~] esxcli network nic vlan stats get -n vmnic1
VLAN 0
   Packets received: 410
   Packets sent: 840

VLAN 200 <----- vSAN VLAN
   Packets received: 64539
   Packets sent: 31553

VLAN 201
Packets received: 7708
Packets sent: 3760

When a non working VLAN is used it looks like this:

[[email protected]:~] esxcli network nic vlan stats get -n vmnic1
VLAN 0
   Packets received: 9013
   Packets sent: 14389

VLAN 200 <------ A very low amount of packets transmitted, this is not normal for vSAN
   Packets received:  4
   Packets sent: 102

VLAN 201
   Packets received: 4
   Packets sent: 2

Another personal favorite is the tcpdump-uw test, which this I can validate if the vSAN communication is occurring, use vSAN’s vmk for the test

[[email protected]:~] tcpdump-uw -i vmk3 port 12321
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk3, link-type EN10MB (Ethernet), capture size 262144 bytes
18:29:35.235210 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200
18:29:57.163498 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472
18:30:06.410854 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200
18:30:06.410937 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200
18:30:06.410963 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472
18:30:06.410970 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200
18:30:06.410977 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200
18:30:06.411006 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200
18:30:06.411087 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472
18:30:06.411093 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200
18:30:06.411099 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200
18:30:06.934927 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472
18:30:06.944259 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200
18:30:06.944317 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200
18:30:07.237912 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200
18:30:07.935050 IP 172.168.11.53.12321 > 172.168.11.52.12321: UDP, length 472
18:30:07.944428 IP 172.168.11.52.12321 > 172.168.11.53.12321: UDP, length 200
18:30:07.944498 IP 172.168.11.52.12321 > 172.168.11.54.12321: UDP, length 200
18:30:08.238002 IP 172.168.11.54.12321 > 172.168.11.52.12321: UDP, length 200

19 packets captured
19 packets received by filter
0 packets dropped by kernel

Here you can see that esxi2  talks to esxi3, esxi4 and vice versa, but we don’t see esxi1

172.168.11.54 > esxi4

172.168.11.53 > esxi3

172.168.11.52 > esxi2

If the communication is good, you’ll see 172.168.11.51 > esxi1

[[email protected]:~] tcpdump-uw -i vmk3 port 12321
tcpdump-uw: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmk3, link-type EN10MB (Ethernet), capture size 262144 bytes
18:33:54.952684 IP 172.168.11.53.12321 > 172.168.11.51.12321: UDP, length 504
18:34:17.133393 IP 172.168.11.54.12321 > 172.168.11.51.12321: UDP, length 200
18:34:27.135761 IP 172.168.11.51.12321 > 172.168.11.53.12321: UDP, length 200
18:34:27.135844 IP 172.168.11.51.12321 > 172.168.11.52.12321: UDP, length 200   <---- esxi1 talking to esxi2
18:34:27.135851 IP 172.168.11.53.12321 > 172.168.11.51.12321: UDP, length 504
18:34:27.135859 IP 172.168.11.54.12321 > 172.168.11.51.12321: UDP, length 200
18:34:27.135865 IP 172.168.11.52.12321 > 172.168.11.51.12321: UDP, length 200   <---- esxi1 talking to esxi2
18:34:27.135872 IP 172.168.11.51.12321 > 172.168.11.54.12321: UDP, length 200
18:34:27.135883 IP 172.168.11.53.12321 > 172.168.11.51.12321: UDP, length 504
18:34:27.135889 IP 172.168.11.54.12321 > 172.168.11.51.12321: UDP, length 200
  • Incorrect unicast agent entry: This was a thing when people used to have mismatched vCenter/ESXi versions, like older vCenters than the ESXi,
[[email protected]:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint
------------------------------------ --------- ---------------- ------------- ----- ---------- -----------------------------------------------------------
611d0e3d-d3dd-b16c-2107-005057942525 0 true 172.168.11.51 12321 84:D2:E9:67:49:9E:B9:5C:C8:40:37:4F:F3:EA:80:F2:6C:FB:04:50
611d0e3d-d1d8-de43-7c12-00505683393a 0 true 172.168.11.53 12321 69:BC:06:4F:21:6A:45:30:EC:D2:AB:43:AE:89:66:81:94:F8:F1:DB
611d0edd-e952-d9b8-4951-00505683749b 0 true 172.168.11.54 12321 A9:83:84:F0:07:AB:7F:28:20:59:3E:85:DC:55:1F:EF:F4:C5:B3:C3

By running the command cmmds-tool whoami you’ll be able to get the actual node UUID used for the unicastagent list, compare the Node UUIDs with the list.

[[email protected]:~] cmmds-tool whoami
611d0e3d-d3dd-b16c-2107-005056831525
[[email protected]:~] cmmds-tool whoami
611d0edc-aeee-6efd-7011-00505683e982
[[email protected]:~] cmmds-tool whoami
611d0e3d-d1d8-de43-7c12-00505683393a
[[email protected]:~] cmmds-tool whoami
611d0edd-e952-d9b8-4951-00505683749b

You have to be very careful and review the entry, if you did you’ll see that the UUID from esxi1 is not correct on the esx02‘s unicastagent list output, this can be easily corrected by following VMware’s KB Configuring vSAN Unicast networking from the command line (2150303)

  • Incorrect shutdown: A loss of electrical power can occur and your UPS won’t last longer to allow you to properly shut down your cluster, let me tell you that if you don’t have enough time to properly shutdown your vSAN cluster, then, only take care of the production you have running there (gracefully shutdown all VMs / Preferably) and let the host go off, I know it might sound terrible, but starting a vSAN shutdown that might not be able to finish could leave you in a worse situation.

1 – vSAN DecomState: You decided to take the risk and will start putting hosts in maintenance mode (MM) no data migration of course and you ran out of time, when the power energy comes back your VMs are shown with an incorrect number or might not be even there, and of course, your node does not seem to be in MM anyways but guess what, it is likely that your node is in vSAN MM and while in that state, it won’t provide any vSAN storage until is taken out.

If your vCenter is online you can easily identify if a host is in DecomState by checking the Skyline Health

If your vCenter is down because it was inside the cluster, you can run the following command esxcli vsan cluster get and look for Maintenance Mode State: ON

[[email protected]:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2021-10-23T18:18:38Z
   Local Node UUID: 611d0e3d-d3dd-b16c-2107-005056831525
   Local Node Type: NORMAL
   Local Node State: AGENT
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 611d0e3d-d1d8-de43-7c12-00505683393a
   Sub-Cluster Backup UUID: 611d0edd-e952-d9b8-4951-00505683749b
   Sub-Cluster UUID: 526e0e30-c21d-f2a5-cd83-be3aeb61f99b
   Sub-Cluster Membership Entry Revision: 5
   Sub-Cluster Member Count: 4
   Sub-Cluster Member UUIDs: 611d0e3d-d1d8-de43-7c12-00505683393a, 611d0edd-e952-d9b8-4951-00505683749b, 611d0edc-aeee-6efd-7011-00505683e982, 611d0e3d-d3dd-b16c-2107-005056831525
   Sub-Cluster Member HostNames: esx03.local, esx04.local, esx02.local, esx01.local
   Sub-Cluster Membership UUID: c4757261-6a14-e410-8842-00505683393a
   Unicast Mode Enabled: true
   Maintenance Mode State: ON <------ DecomState
   Config Generation: e99835c2-8bfd-41bc-98b7-d965e5f33a97 8 2021-10-23T17:45:30.600

To exit this state you can run localcli vsan maintenancemode cancel so you don’t have to put the host in maintenance mode and exit

[[email protected]:~] localcli vsan maintenancemode cancel
VsanUtil: AffObj count: 0
[[email protected]:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2021-10-23T18:25:22Z
   Local Node UUID: 611d0e3d-d3dd-b16c-2107-005056831525
   Local Node Type: NORMAL
   Local Node State: AGENT
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 611d0e3d-d1d8-de43-7c12-00505683393a
   Sub-Cluster Backup UUID: 611d0edd-e952-d9b8-4951-00505683749b
   Sub-Cluster UUID: 526e0e30-c21d-f2a5-cd83-be3aeb61f99b
   Sub-Cluster Membership Entry Revision: 5
   Sub-Cluster Member Count: 4
   Sub-Cluster Member UUIDs: 611d0e3d-d1d8-de43-7c12-00505683393a, 611d0edd-e952-d9b8-4951-00505683749b, 611d0edc-aeee-6efd-7011-00505683e982, 611d0e3d-d3dd-b16c-2107-005056831525
   Sub-Cluster Member HostNames: esx03.local, esx04.local, esx02.local, esx01.local   Sub-Cluster Membership UUID: c4757261-6a14-e410-8842-00505683393a
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF <------ No longer in DecomState
   Config Generation: e99835c2-8bfd-41bc-98b7-d965e5f33a97 8 2021-10-23T17:45:30.600
  • Errors during troubleshooting: Yep, we are all humans, and maybe during the heat of the situation they might make some mistakes, those mistakes can be related to HW/Networking or logical troubleshooting, so you can always review what we already explained to see if somebody made a mistake.

Hope this was a cool post for you guys to know,

Do now hesitate in contacting me if you have any comments,

Jorluis

Buy me a coffeeBuy me a coffee