Packs that have actions in sub-directories importing modules in same sub-directory fail

[amanda11]

SUMMARY

I have observed in a couple of packs when using StackStorm 3.4dev that they now get ModuleNotFoundError when the pack has a module actions/lib/moduleA.py and that moduleA.py has an import moduleB where moduleB is in the same library.
These packs were working in previous versions.
Observed in both ActiveDirectory and Excel packs.

STACKSTORM VERSION

st2 3.4dev (c422f00), on Python 3.6.8

OS, environment, install method

CentOS 8.2, single line bash installer

Steps to reproduce the problem

Observed with following actions:

• activedirectory.get_ad_user
• excel.get_keys_for_rows

Expected Results

Actions to work as before and not throw ModuleNotFoundError

Actual Results

Excel Stack Trace:

{
  "stdout": "",
  "stderr": "Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 238, in _get_action_instance
    actions_cls = action_loader.register_plugin(Action, self._file_path)
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/loader.py\", line 166, in register_plugin
    module = imp.load_source(module_name, plugin_abs_file_path)
  File \"/opt/stackstorm/virtualenvs/excel/lib64/python3.6/imp.py\", line 172, in load_source
    module = _load(spec)
  File \"<frozen importlib._bootstrap>\", line 684, in _load
  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked
  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module
  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed
  File \"/opt/stackstorm/packs/excel/actions/get_keys_for_rows.py\", line 14, in <module>
    from lib import excel_action, excel_reader
  File \"/opt/stackstorm/packs/excel/actions/lib/excel_reader.py\", line 16, in <module>
    import string_converter
ModuleNotFoundError: No module named 'string_converter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 334, in <module>
    obj.run()
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 192, in run
    action = self._get_action_instance()
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 245, in _get_action_instance
    raise exc_cls(msg)
ModuleNotFoundError: Failed to load action class from file \"/opt/stackstorm/packs/excel/actions/get_keys_for_rows.py\" (action file most likely doesn't exist or contains invalid syntax): No module named 'string_converter'

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 238, in _get_action_instance
    actions_cls = action_loader.register_plugin(Action, self._file_path)
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/loader.py\", line 166, in register_plugin
    module = imp.load_source(module_name, plugin_abs_file_path)
  File \"/opt/stackstorm/virtualenvs/excel/lib64/python3.6/imp.py\", line 172, in load_source
    module = _load(spec)
  File \"<frozen importlib._bootstrap>\", line 684, in _load
  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked
  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module
  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed
  File \"/opt/stackstorm/packs/excel/actions/get_keys_for_rows.py\", line 14, in <module>
    from lib import excel_action, excel_reader
  File \"/opt/stackstorm/packs/excel/actions/lib/excel_reader.py\", line 16, in <module>
    import string_converter
ModuleNotFoundError: No module named 'string_converter'

",
  "exit_code": 1,
  "result": "None"
}

Active Directory StackTrace:

{
  "stdout": "",
  "stderr": "Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 238, in _get_action_instance
    actions_cls = action_loader.register_plugin(Action, self._file_path)
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/loader.py\", line 166, in register_plugin
    module = imp.load_source(module_name, plugin_abs_file_path)
  File \"/opt/stackstorm/virtualenvs/activedirectory/lib64/python3.6/imp.py\", line 172, in load_source
    module = _load(spec)
  File \"<frozen importlib._bootstrap>\", line 684, in _load
  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked
  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module
  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed
  File \"/opt/stackstorm/packs/activedirectory/actions/run_cmdlet.py\", line 1, in <module>
    from lib import action
  File \"/opt/stackstorm/packs/activedirectory/actions/lib/action.py\", line 1, in <module>
    from winrm_connection import WinRmConnection
ModuleNotFoundError: No module named 'winrm_connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 334, in <module>
    obj.run()
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 192, in run
    action = self._get_action_instance()
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 245, in _get_action_instance
    raise exc_cls(msg)
ModuleNotFoundError: Failed to load action class from file \"/opt/stackstorm/packs/activedirectory/actions/run_cmdlet.py\" (action file most likely doesn't exist or contains invalid syntax): No module named 'winrm_connection'

Traceback (most recent call last):
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py\", line 238, in _get_action_instance
    actions_cls = action_loader.register_plugin(Action, self._file_path)
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/loader.py\", line 166, in register_plugin
    module = imp.load_source(module_name, plugin_abs_file_path)
  File \"/opt/stackstorm/virtualenvs/activedirectory/lib64/python3.6/imp.py\", line 172, in load_source
    module = _load(spec)
  File \"<frozen importlib._bootstrap>\", line 684, in _load
  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked
  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module
  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed
  File \"/opt/stackstorm/packs/activedirectory/actions/run_cmdlet.py\", line 1, in <module>
    from lib import action
  File \"/opt/stackstorm/packs/activedirectory/actions/lib/action.py\", line 1, in <module>
    from winrm_connection import WinRmConnection
ModuleNotFoundError: No module named 'winrm_connection'

",
  "exit_code": 1,
  "result": "None"
}

Workarounds were possible to get this to work, e.g. change the ActiveDirectory to use "from .winrm_connection import WinRm"

[amanda11]

1. Create fresh CentOS 8.2 VM and install STackStorm 3.3. Install excel pack. Put in place blank spreadsheet. Run excel.get_keys_for_rows -> all ok
2. Create fresh CentOS 8.2 VM and install StackStorm unstable. Repeat above -> get the import error. (So reproduced it on 2 different CentOS 8.2 VMs now - just in case I was going mad!)

e.g. just run:
st2 run excel.get_keys_for_rows sheet=<sheetname> excel_file=<spreadsheet>

In fact don't even need to have a valid spreadsheet e.g.

$ st2 run excel.get_keys_for_rows sheet=rubbish excel_file=/opt/stackstorm/rubbish.xlsx
.
id: 601055e7509b8ab7a3c1b0d7
action.ref: excel.get_keys_for_rows
context.user: st2admin
parameters: 
  excel_file: /opt/stackstorm/rubbish.xlsx
  sheet: rubbish
status: failed
start_timestamp: Tue, 26 Jan 2021 17:48:23 UTC
end_timestamp: Tue, 26 Jan 2021 17:48:24 UTC
result: 
  exit_code: 1
  result: None
  stderr: "Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 238, in _get_action_instance
    actions_cls = action_loader.register_plugin(Action, self._file_path)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/loader.py", line 166, in register_plugin
    module = imp.load_source(module_name, plugin_abs_file_path)
  File "/opt/stackstorm/virtualenvs/excel/lib64/python3.6/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/stackstorm/packs/excel/actions/get_keys_for_rows.py", line 14, in <module>
    from lib import excel_action, excel_reader
  File "/opt/stackstorm/packs/excel/actions/lib/excel_reader.py", line 16, in <module>
    import string_converter
ModuleNotFoundError: No module named 'string_converter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 334, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 192, in run
    action = self._get_action_instance()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 245, in _get_action_instance
    raise exc_cls(msg)
ModuleNotFoundError: Failed to load action class from file "/opt/stackstorm/packs/excel/actions/get_keys_for_rows.py" (action file most likely doesn't exist or contains invalid syntax): No module named 'string_converter'

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 238, in _get_action_instance
    actions_cls = action_loader.register_plugin(Action, self._file_path)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/loader.py", line 166, in register_plugin
    module = imp.load_source(module_name, plugin_abs_file_path)
  File "/opt/stackstorm/virtualenvs/excel/lib64/python3.6/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/stackstorm/packs/excel/actions/get_keys_for_rows.py", line 14, in <module>
    from lib import excel_action, excel_reader
  File "/opt/stackstorm/packs/excel/actions/lib/excel_reader.py", line 16, in <module>
    import string_converter
ModuleNotFoundError: No module named 'string_converter'

"
  stdout: ''

[blag]

I believe that this is related to the PYTHONPATH changes in #5100. I'm working on a fix for this now.