Update our core dependencies

[JeffreyDevloo]

Problem description

While looking around for certain bugs/annoyances in our libraries, I discovered that we are 4.5 years behind on updates on our core dependencies:

Front-end

• Knockout:
  • Installed: 3.3.0
  • Candidate: 3.4.2
• Durandal:
  • Installed: 2.0.1
  • Candidate: 2.2.0
• JQuery:
  • Installed: 1.9.1
  • Candidate: 2.1

Back-end

• Celery:
  • Installed: 3.1.23-5
  • Candidate: 4.2
• RabbitMQ:
  • Installed: 3.5.7
  • Candidate: 3.6.12
• Django Rest Framework (API):
  • Installed: 2.3.12
  • Candidate: 3.7.1
• Python:
  • Installed: 2.7
  • Candidate: 3.6.2

Needless to say, being this behind is handicapping us in terms of feature and bug fixes

Update progress

• JQuery (Update JQuery #1840)
• Durandal (Update durandal #1341)
• Knockout (Update durandal #1341)
• Celery
• RabbitMQ (Update RabbitMQ #1497)
• DJango Rest Framework
• Python (https://github.com/openvstorage/home/issues/42)

[JeffreyDevloo]

Reworking certain GUI aspects for #1656

Picked up the following tickets:

• Update durandal #1341
• Update JQuery #1840

[wimpers]

@JeffreyDevloo can you add an estimate for the update of Celery, rabbitmq and Django . What would be the impact on the code? Maybe we can update to the latest minor version of the one we are currently on instead of the latest latest?

Update to Python 3 would be out of scope.

[JeffreyDevloo]

@wimpers
I'd estimate every update to be at least a weeks worth of testing/bugfixing
The impact of Django is rather high due to all decorators needing a visit
The impact of both Celery and Rabbitmq is currently unknown to me

[matthiasdeblock]

Gunicorn currently has a bug in deleting temporary files on the filesystem which are already gone from the directory metadata. These files are filling up the disk and can only be deleted when you restart the webapp-api.

PR Gunicorn: benoitc/gunicorn#1428

This PR got merged in December 2017 but the latest Gunicorn release is from March 2017 so it is not yet possible to implement a fixed release.

I've also noticed that the ubuntu 16.04 is also running behind. It is at version 19.4.5 while the latest release is 19.7.1 (implemented in ubuntu 18.04).

root@DEV-3N-199-181:~# lsof |grep -i deleted | grep gunicorn 
gunicorn   7130                   root    6u      REG              253,1         0     262242 /tmp/wgunicorn-WvrCzS (deleted)
gunicorn   7130                   root    7u      REG              253,1         0     279988 /tmp/wgunicorn-Lb2C_G (deleted)
gunicorn   7130                   root    8u      REG              253,1         0     281825 /tmp/wgunicorn-nZmI6u (deleted)
gunicorn   7130                   root    9u      REG              253,1         0     281826 /tmp/wgunicorn-OB4bdy (deleted)
gunicorn   7130                   root   10u      REG              253,1         0     281842 /tmp/wgunicorn-KlODhS (deleted)
gunicorn   7156                   root    6u      REG              253,1         0     262242 /tmp/wgunicorn-WvrCzS (deleted)
gunicorn   7156 20618             root    6u      REG              253,1         0     262242 /tmp/wgunicorn-WvrCzS (deleted)
gunicorn   7156 20648             root    6u      REG              253,1         0     262242 /tmp/wgunicorn-WvrCzS (deleted)
gunicorn   7156 27429             root    6u      REG              253,1         0     262242 /tmp/wgunicorn-WvrCzS (deleted)
gunicorn   7158                   root    6u      REG              253,1         0     262242 /tmp/wgunicorn-WvrCzS (deleted)

root@DEV-3N-199-181:~# lsof |grep -i deleted | grep gunicorn -c
73

[jeroenmaelbrancke]

Rabbitmq has a bug rabbitmq/rabbitmq-server#714 between master and slaves (deadlocks). This has been fixed in 3.6.x.
Today GIG ended up with this scenario. See ticket http://support.openvstorage.com/browse/OVSSUP-429

[matthiasdeblock]

Digital Energy had the same rabbitmq issue as described in the comment above.

[wimpers]

DE had another instance of this http://tickets.openvstorage.com/browse/OVSSUP-453

[wimpers]

Note that 18.04 comes with RabbitMQ 3.6.10 so maybe better to with that version instead of 3.612 or 3.7.X.

[jeroenmaelbrancke]

I'm running RabbitMQ 3.7.4-1 on my test environment.

root@LC1SRV0008:~# apt-cache policy rabbitmq-server 
rabbitmq-server:
  Installed: 3.7.4-1
  Candidate: 3.7.4-1
  Version table:
 *** 3.7.4-1 500

See ansible role how to setup: https://github.com/openvstorage/setups/tree/qa/openvstorage/qa/VIR-18N-50.10/roles/externalRabbitMQ