Run `ci/do_ci.sh fix_format` cmd got error `ModuleNotFoundError: No module named 'pip'`

[qqustc]

ci/do_ci.sh fix_format cmd returned error:

Traceback (most recent call last):
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'

@mum4k helped me go around this problem by adding --system-site-packages to source_venv() in nighthawk/tools/shell_utils.sh :

source_venv() {
  VENV_DIR=$1
  if [[ "${VIRTUAL_ENV}" == "" ]]; then
    if [[ ! -d "${VENV_DIR}"/venv ]]; then
      virtualenv "${VENV_DIR}"/venv --python=python3 --system-site-packages
    fi
    source "${VENV_DIR}"/venv/bin/activate
  else
    echo "Found existing virtualenv"
  fi
}

However now ci/do_ci.sh fix_format cmd returns new error:

Traceback (most recent call last):
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/flake8/checker.py", line 451, in run_check
    return plugin["plugin"](**arguments)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/flake8/plugins/pyflakes.py", line 99, in __init__
    super(FlakesChecker, self).__init__(
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 497, in __init__
    self.handleChildren(tree)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 824, in handleChildren
    self.handleNode(node, tree)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 871, in handleNode
    handler(node)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 824, in handleChildren
    self.handleNode(node, tree)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 871, in handleNode
    handler(node)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 824, in handleChildren
    self.handleNode(node, tree)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 870, in handleNode
    handler = self.getNodeHandler(node.__class__)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/pyflakes/checker.py", line 700, in getNodeHandler
    self._nodeHandlers[node_class] = handler = getattr(self, nodeType)
AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/flake8/checker.py", line 682, in _run_checks
    return checker.run_checks()
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/flake8/checker.py", line 612, in run_checks
    self.run_ast_checks()
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/flake8/checker.py", line 513, in run_ast_checks
    checker = self.run_check(plugin, tree=ast)
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/flake8/checker.py", line 456, in run_check
    raise exceptions.PluginExecutionFailed(
  File "/usr/local/google/home/qqin/github/nighthawk/tools/pyformat/venv/lib/python3.8/site-packages/flake8/exceptions.py", line 86, in __init__
    self.original_exception = kwargs.pop("exception")
KeyError: 'exception'

Is there a better way to modify nighthawk/tools/shell_utils.sh so that ci/do_ci.sh fix_format can work? Thanks!

\cc @oschaaf

[oschaaf]

Searching for a bit, this looks similar to PyCQA/pyflakes#367
It seems pyflakes versions older than 2.1.0 are incompatible with Python 3.8
We could attempt to require a later revision of it?

[oschaaf]

I pushed a change in #395 that should resolve this.

[qqustc]

thanks @oschaaf ! I tried the change in #395 and it does fix the issue

[oschaaf]

@qqustc thanks for confirming