'st2ctl register' fails to register rules on a fresh DB

[arm4b]

Previous PR #429 removed all sleep occurencies from the installer and fixed some deep issues with the st2 services startup, handling better st2api, st2auth, st2stream to report correct status only when respective service is listening on the network port.

But not all bugs were caught. Under Ubuntu Xenial after initial st2 install running

sudo st2ctl start && st2ctl reload --register-all

fails to register the rules.

Reproduce

Run xenial installer (with removed sleep):

curl -sSL https://raw.githubusercontent.com/StackStorm/st2-packages/27ec5a09d87c3bc2bcfd0e01adf3ff4daa294551/scripts/st2bootstrap-deb.sh | bash -s -- --user=st2admin --password=st2admin --unstable --staging

Happens only during the first boot.

Error

Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
2017-03-14 17:58:48,626 INFO [-] Connecting to database "st2" @ "0.0.0.0:27017" as user "stackstorm".
2017-03-14 17:58:50,440 INFO [-] =========================================================
2017-03-14 17:58:50,440 INFO [-] ############## Registering triggers #####################
2017-03-14 17:58:50,440 INFO [-] =========================================================
2017-03-14 17:58:50,550 INFO [-] Registered 0 triggers.
2017-03-14 17:58:50,550 INFO [-] =========================================================
2017-03-14 17:58:50,550 INFO [-] ############## Registering sensors ######################
2017-03-14 17:58:50,551 INFO [-] =========================================================
2017-03-14 17:58:50,618 INFO [-] Registered 1 sensors.
2017-03-14 17:58:50,619 INFO [-] =========================================================
2017-03-14 17:58:50,619 INFO [-] ############## Registering runners ######################
2017-03-14 17:58:50,619 INFO [-] =========================================================
2017-03-14 17:58:51,204 INFO [-] Registered 13 runners.
2017-03-14 17:58:51,204 INFO [-] =========================================================
2017-03-14 17:58:51,204 INFO [-] ############## Registering actions ######################
2017-03-14 17:58:51,204 INFO [-] =========================================================
2017-03-14 17:58:53,184 INFO [-] Registered 46 actions.
2017-03-14 17:58:53,185 INFO [-] =========================================================
2017-03-14 17:58:53,185 INFO [-] ############## Registering rules ########################
2017-03-14 17:58:53,185 INFO [-] =========================================================
2017-03-14 17:58:53,286 WARNING [-] Failed to register rules: Failed to register rule "/opt/stackstorm/packs/chatops/rules/notify_hubot.yaml" from pack "chatops": A simple trigger should have been created when registering triggertype. Cannot create trigger: {'type': 'core.st2.generic.notifytrigger', 'parameters': {}, 'pack': 'core'}.
Traceback (most recent call last):
  File "/usr/bin/st2-register-content", line 22, in <module>
    sys.exit(content_loader.main(sys.argv[1:]))
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/content/bootstrap.py", line 387, in main
    register_content()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/content/bootstrap.py", line 339, in register_content
    register_rules()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/content/bootstrap.py", line 248, in register_rules
    raise e
ValueError: Failed to register rule "/opt/stackstorm/packs/chatops/rules/notify_hubot.yaml" from pack "chatops": A simple trigger should have been created when registering triggertype. Cannot create trigger: {'type': 'core.st2.generic.notifytrigger', 'parameters': {}, 'pack': 'core'}.

---

This time the problem could be related to st2 core itself. Needs more investigation and proper fix.

[arm4b]

After further debugging, here is the script to easily reproduce an error on Ubuntu16 (works on Vagrant too):

#!/bin/bash

set -o xtrace

# disable mongo auth to run dropDatabase() from cli
if grep -q 'authorization: enabled' /etc/mongod.conf; then
  sudo sed -i 's/authorization: enabled/authorization: disabled/' /etc/mongod.conf
  sudo service mongod restart
fi

# emulate fresh & clean start
sudo st2ctl stop
mongo st2 --eval 'db.dropDatabase();'
sudo st2ctl start

# Failed to register rules: Failed to register rule "/opt/stackstorm/packs/chatops/rules/notify_hubot.yaml"
st2ctl reload --register-all

Seems at early point st2 is busy bootstrapping DB schema in mongo (?) and for some reason can't register a rule (some db resource is not yet available ?) when saving it for the first time.

---

So from the packaging point of view, I can't do anything here.

@Kami @lakshmi-kannan @m4dcoder Can you help with debugging this case in StackStorm itself?

[estee-tew]

https://stackstorm.slack.com/archives/community/p1492461999780716

[estee-tew]

https://stackstorm.slack.com/archives/community/p1492633077162687

[csillab]

I have the same issue with newly installed vagrant ubuntu 16.06.

[Mierdin]

I looked into this a bit tonight, and haven't figured out why this is happening, but it does seem to be a delay in what's being written to MongoDB, and the availability of that resource to be read, since I have been able to fix this by simply inserting a 5 second sleep before the query. Still looking.