Hello vComunnity,

The other day I was on a vSAN deployment where right after the node was added to the vCenter/Cluster it reported this:

Registration/unregistration of third-party IO filter storage providers fails on host.

From logs you might see something like this:

ESXi log: /var/log/iofiltervpd.log

2021-10-22T00:30:43Z iofiltervpd: iofiltervpd is running.
2021-10-22T00:30:43Z iofiltervpd[2099422]: IOFilterVPControlHandleMessage:239:received command: addconf /usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml VMW_vmwarevmcrypt_1.0.0
2021-10-22T00:30:43Z iofiltervpd[2099422]: ReadConfigFromFile:247:/usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml: successfully loaded filter configuration.
2021-10-22T00:30:43Z iofiltervpd[2099422]: ReadConfigFromFile:248:FilterName: vmwarevmcrypt
2021-10-22T00:30:43Z iofiltervpd[2099422]: ReadConfigFromFile:249:FilterClass: encryption
2021-10-22T00:30:43Z iofiltervpd[2099422]: ReadConfigFromFile:250:FilterVersion: 1.0.0
2021-10-22T00:30:43Z iofiltervpd[2099422]: ReadConfigFromFile:251:FilterType: disk
2021-10-22T00:30:43Z iofiltervpd[2099422]: LoadFilterCatalogs:191:Added 8 catalog(s) for filter vmwarevmcrypt
2021-10-22T00:30:43Z iofiltervpd[2099422]: LoadConfiguration:158:Added configuration file: /usr/lib/vmware/vmiof/disk/vmwarevmcrypt-config.xml successfully.
2021-10-22T00:30:43Z iofiltervpd[2099422]: AddConfiguration:258:Number of filters loaded: 2
2021-10-22T00:30:44Z iofiltervpd[2099422]: main:239:IOFilterVP successfully started.
2021-10-22T01:02:22Z iofiltervpd[2099422]: run:170:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
2021-10-22T01:03:02Z iofiltervpd[2099422]: run:170:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
2021-10-22T01:03:42Z iofiltervpd[2099422]: run:170:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
2021-10-22T01:04:32Z iofiltervpd[2099422]: run:170:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
2021-10-22T01:05:12Z iofiltervpd[2099422]: run:170:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
2021-10-22T01:05:44Z iofiltervpd[2099422]: IOFilterVPControlHandleMessage:239:received command: refreshcert
2021-10-22T01:05:54Z iofiltervpd[2099422]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
2021-10-22T01:06:03Z iofiltervpd[2099422]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
2021-10-22T01:06:04Z iofiltervpd[2099422]: run:170:SSL Connection error 30 : SSL_ERROR_SSL error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
2021-10-22T01:06:13Z iofiltervpd[2099422]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
2021-10-22T01:06:23Z iofiltervpd[2099422]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
2021-10-22T01:06:33Z iofiltervpd[2099422]: IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified
vCenter log: /var/log/vmware/vmware-sps/sps.log

2021-10-22T13:31:50.641Z [pool-27-thread-1] ERROR opId=sps-Main-909485-144 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://esxi-01:9080/version.xml is offline

The troubleshooting that works 90% of the time is this:

1. Put the host in Maintenance mode with Ensure Accessibility
2. Take a backup of the current cert file /etc/vmware/ssl/castore.pem ( by running: cp /etc/vmware/ssl/castore.pem /etc/vmware/ssl/castore.pem.backup )
3. Copy the file /etc/vmware/ssl/castore.pem from a working host to the affected hosts (Alternatively you can use scp or winscp to upload a copy of the file directly to /etc/vmware/ssl/)
4. Run the command to replace the older file with newer one : cp /tmp/castore.pem /etc/vmware/ssl/castore.pem
5. Reboot the hosts

BUT, here I’ll show you how to get it fixed if the steps above did not, Credits to VMware’s KB Certain IOFIlter Providers are showing as offline (76633)

Before you proceed, make sure to take a powered-off snapshot of the VCSA and ALL VCSA that are connected to this including PSCs. If you use VCHA then you will need full backups NOT snapshots.

Log in to the following link https://<VC-IP>/sms/mob

1) Click on QueryStorageManager link


2) Click on Invoke method, then on the “storageManager” link in the Method Invocation Result


3) Click on the QueryProvider link and Invoke method


4) Click through the vasaProvider-XXX links in the Method Invocation Results one by one


5) For each vasaProvider selected click on QueryProviderInfo link and Invoke Method


6) Cross reference the result for “uid” field and note it down for each “url” entry that matches from the url’s mentioned in the vCenter Storage Provider pane as showing offline

7) Go Back to the QueryStorageManager link (see step 1 and step 2)
8) Click on UnregisterProvider_Task link


9) Enter one of the the provider uid’s found in step 6 and click on the invoke method.

Step 9 needs to be repeated for each of the  uuid’s noted in step 6 above.

##############################
# Registering IOFilter VP’s: #
##############################

– Restart SPS service with these commands:

[email protected] [ ~ ]# service-control --stop vmware-sps && service-control --start vmware-sps
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service sps...
Successfully stopped service sps
Operation not cancellable. Please wait for it to finish...
Performing start operation on service sps...
Successfully started service sps
[email protected] [ ~ ]#

Then you go from this:

to this:

Hope this was a cool post for you guys to know,

Do now hesitate in contacting me if you have any comments,

Jorluis

Buy me a coffeeBuy me a coffee