web-ui doesn't show vdisks when a node goes down

[yongshengma]

I have 3-node cluster. When a node goes down, i can still access web-ui from other nodes. However, it's not easy for the UI to show vdisks and the content or list is often empty. GET operation doesn't return anything and it looks hanging up.

[JeffreyDevloo]

Hi
Could you give us an overview of the packages that were installed (dpkg -l | grep openstorage).

Best regards

[yongshengma]

dpkg -l | grep openvstorage

ii  blktap-openvstorage-utils            2.0.90-2ubuntu5                       amd64        utilities to work with VHD disk images files
ii  libblktapctl0-openvstorage           2.0.90-2ubuntu5                       amd64        Xen API blktapctl shared library (shared library)
ii  libvhd0-openvstorage                 2.0.90-2ubuntu5                       amd64        VHD file format access library
ii  libvhdio-2.0.90-openvstorage         2.0.90-2ubuntu5                       amd64        Xen API blktap shared library (shared library)
ii  openvstorage                         2.9.5-1                               amd64        openvStorage
ii  openvstorage-backend                 1.9.1-1                               amd64        openvStorage Backend plugin
ii  openvstorage-backend-core            1.9.1-1                               amd64        openvStorage Backend plugin core
ii  openvstorage-backend-webapps         1.9.1-1                               amd64        openvStorage Backend plugin Web Applications
ii  openvstorage-core                    2.9.5-1                               amd64        openvStorage core
ii  openvstorage-hc                      1.9.1-1                               amd64        openvStorage Backend plugin HyperConverged
ii  openvstorage-sdm                     1.9.0-1                               amd64        Open vStorage Backend ASD Manager
ii  openvstorage-webapps                 2.9.5-1                               amd64        openvStorage Web Applications
ii  tgt-openvstorage                     99:1.0.63-0ovs1.4-1ubuntu1.1          amd64        Linux SCSI target user-space daemon and tools - OpenvStorage support

[JeffreyDevloo]

Hi
Do you see any Arakoon related errors in the webapi logging?
journalctl -u ovs-webapp-api.service
It could be that the Arakoon cluster is temporarily without a master and he might be electing a new one.

Best regards

[yongshengma]

I have to wait some seconds, maybe 30s, but the vdisk list finally comes out. If I go to other pages and come back, I have to wait some time again before it shows up.

`
Aug 14 12:20:06 test-2 gunicorn[2509]: 2018-08-14 06:20:06 16900 +0200 - test-2 - 3108/140622612104688 - hybrids/storagedriver.py - _edge_clients - 285 - ERROR - Error loading edge clients from pool-001PXP0lu82BYCjnaQ1
Aug 14 12:20:06 test-2 gunicorn[2509]: Traceback (most recent call last):
Aug 14 12:20:06 test-2 gunicorn[2509]: File "/opt/OpenvStorage/ovs/dal/hybrids/storagedriver.py", line 81, in _edge_clients
Aug 14 12:20:06 test-2 gunicorn[2509]: for item in self.vpool.storagedriver_client.list_client_connections(str(self.storagedriver_id), req_timeout_secs=2):
Aug 14 12:20:06 test-2 gunicorn[2509]: RuntimeError: failed to send XMLRPC request ListClientConnections
Aug 14 12:20:06 test-2 gunicorn[2509]: 2018-08-14 06:20:06 34300 +0200 - test-2 - 3121/140622609734544 - hybrids/storagedriver.py - _edge_clients - 958 - ERROR - Error loading edge clients from pool-001PXP0lu82BYCjnaQ1
Aug 14 12:20:06 test-2 gunicorn[2509]: Traceback (most recent call last):
Aug 14 12:20:06 test-2 gunicorn[2509]: File "/opt/OpenvStorage/ovs/dal/hybrids/storagedriver.py", line 81, in _edge_clients
Aug 14 12:20:06 test-2 gunicorn[2509]: for item in self.vpool.storagedriver_client.list_client_connections(str(self.storagedriver_id), req_timeout_secs=2):
Aug 14 12:20:06 test-2 gunicorn[2509]: RuntimeError: failed to send XMLRPC request ListClientConnections
Aug 14 12:20:09 test-2 gunicorn[2509]: 2018-08-14 06:20:09 34900 +0200 - test-2 - 3108/140622612104688 - hybrids/vdisk.py - fetch_statistics - 286 - ERROR - Error loading statistics_volume from 86fd1161-0a7d-4049-85bf-5e9d81f0a8b3: failed to send XMLRPC request volumePerformanceCountersV3
Aug 14 12:20:09 test-2 gunicorn[2509]: 2018-08-14 06:20:09 34900 +0200 - test-2 - 3121/140622609734544 - hybrids/vdisk.py - fetch_statistics - 959 - ERROR - Error loading statistics_volume from 86fd1161-0a7d-4049-85bf-5e9d81f0a8b3: failed to send XMLRPC request volumePerformanceCountersV3
Aug 14 12:20:09 test-2 gunicorn[2509]: 2018-08-14 06:20:09 43100 +0200 - test-2 - 3108/140622612104688 - api/decorators.py - new_function - 287 - WARNING - API call vdisks.list took 5.32s

`

[JeffreyDevloo]

Hi yongshengma

Thank you for the feedback.
The GUI wants to display information that it needs from the volumedriver. It looks like certain management requests are taking too long to resolve because that volumedriver is down.
We will look into optimizing the GUI so the essential data can be retrieved first.

Best regards

[wimpers]

@JeffreyDevloo didn't #1850 fix this ?

[JeffreyDevloo]

No, it does not.
The GUI should go back to lazy loading certain fields to display the essential data faster.