Fix no such container error with docker-compose

[sanders41]

Purpose

Fix the error when running run_infrastructure.py

Changes

• Set IMAGE_NAME to fidesops_webserver
• Add container_name to docker-compose files

Checklist

• Update CHANGELOG.md file
  • Merge in main so the most recent CHANGELOG.md file is being appended to
  • Add description within the Unreleased section in an appropriate category. Add a new category from the list at the top of the file if the needed one isn't already there.
  • Add a link to this PR at the end of the description with the PR number as the text. example: #1
• Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.
• If docs updated (select one):
  • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
  • documentation issue created (tag docs-team to complete issue separately)
• Good unit test/integration test coverage
• This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
• The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Ticket

Fixes #756

[sanders41]

@pattisdr some of the run_infrastructure.py functions work, make user for example, and some don't, make pytest-integration for example, but they all use the same IMAGE_NAME. Any ideas what could be causing it?

[pattisdr]

interesting, I still couldn't get make user to work on this branch:

ERROR: No such service: fidesops_webserver
no custom setup logic found for postgres, skipping
Running create superuser...
WARNING: The FIDESOPS__HOT_RELOAD variable is not set. Defaulting to a blank string.
WARNING: Found orphan containers (fidesops, fidesops_celery_1, fidesops_mssql_example_1, fidesops_mongodb_example_1, fidesops_mysql_example_1, fidesops_mariadb_example_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
fidesops_db_1 is up-to-date
fidesops_redis_1 is up-to-date
fidesops_postgres_example_1 is up-to-date
fidesops_docs_1 is up-to-date
fidesops_worker is up-to-date
fidesops_webserver is up-to-date
Startup configuration: reloading = False, dev_mode = True
Startup configuration: pii logging = True
Traceback (most recent call last):
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 309, in <module>
    run_infrastructure(
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 96, in run_infrastructure
    return _run_create_superuser(path, IMAGE_NAME)
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 169, in _run_create_superuser
    _run_cmd_or_err(f"docker exec -it {image_name} python create_superuser.py")
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 145, in _run_cmd_or_err
    raise Exception(f"Error executing command: {cmd}")
Exception: Error executing command: docker exec -it fidesops_webserver python create_superuser.py
make: *** [Makefile:219: user] Error 1

I can switch back to this branch and experiment some more tomorrow. I don't know if it's my M1 Mac but docker has always been slow and lately it's become painfully slow. Difficult to iterate quickly on testing things.

[sanders41]

Very interesting, make user and make quickstart both worked for me.

Docker is painfully slow for me also. I think it is our image and not the M1 because I had the same issue before I officially started working and was using my Linux computer.

[sanders41]

I got it to work in all the places I have seen it fail by removing the fidesops_ prefix everywhere, and keeping a container_name in the docker-compose files.

[pattisdr]

I think what is happening on my end is that the webserver container is failing, so I can't exec into it and run the quickstart or user script:

webserver    | Startup configuration: reloading = False, dev_mode = True
webserver    | Startup configuration: pii logging = True
webserver    | Traceback (most recent call last):
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 568, in _build_master
webserver    |     ws.require(__requires__)
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
webserver    |     needed = self.resolve(parse_requirements(requirements))
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 777, in resolve
webserver    |     raise VersionConflict(dist, req).with_context(dependent_req)
webserver    | pkg_resources.ContextualVersionConflict: (fideslib 2.1.1 (/usr/local/lib/python3.9/site-packages), Requirement.parse('fideslib==2.1.0'), {'fidesops'})
webserver    | 
webserver    | During handling of the above exception, another exception occurred:
webserver    | 
webserver    | Traceback (most recent call last):
webserver    |   File "/usr/local/bin/fidesops", line 33, in <module>
webserver    |     sys.exit(load_entry_point('fidesops', 'console_scripts', 'fidesops')())
webserver    |   File "/usr/local/bin/fidesops", line 25, in importlib_load_entry_point
webserver    |     return next(matches).load()
webserver    |   File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
webserver    |     module = import_module(match.group('module'))
webserver    |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
webserver    |     return _bootstrap._gcd_import(name[level:], package, level)
webserver    |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
webserver    |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
webserver    |   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
webserver    |   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
webserver    |   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
webserver    |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
webserver    |   File "/fidesops/src/fidesops/cli.py", line 6, in <module>
webserver    |     from fidesops.main import start_webserver
webserver    |   File "/fidesops/src/fidesops/main.py", line 19, in <module>
webserver    |     from fidesops.api.v1.api import api_router
webserver    |   File "/fidesops/src/fidesops/api/v1/api.py", line 3, in <module>
webserver    |     from fidesops.api.v1.endpoints import (
webserver    |   File "/fidesops/src/fidesops/api/v1/endpoints/connection_endpoints.py", line 58, in <module>
webserver    |     from fidesops.service.connectors import get_connector
webserver    |   File "/fidesops/src/fidesops/service/connectors/__init__.py", line 9, in <module>
webserver    |     from fidesops.service.connectors.sql_connector import (
webserver    |   File "/fidesops/src/fidesops/service/connectors/sql_connector.py", line 5, in <module>
webserver    |     from snowflake.sqlalchemy import URL as Snowflake_URL
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/sqlalchemy/__init__.py", line 25, in <module>
webserver    |     from . import base, snowdialect
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/sqlalchemy/snowdialect.py", line 13, in <module>
webserver    |     from snowflake.connector import errors as sf_errors
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/connector/__init__.py", line 16, in <module>
webserver    |     from .connection import SnowflakeConnection
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/connector/connection.py", line 24, in <module>
webserver    |     from . import errors, proxy
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/connector/errors.py", line 15, in <module>
webserver    |     from .compat import BASE_EXCEPTION_CLASS
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/connector/compat.py", line 19, in <module>
webserver    |     from . import constants
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/connector/constants.py", line 11, in <module>
webserver    |     from .options import installed_pandas
webserver    |   File "/usr/local/lib/python3.9/site-packages/snowflake/connector/options.py", line 13, in <module>
webserver    |     import pkg_resources
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3243, in <module>
webserver    |     def _initialize_master_working_set():
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
webserver    |     f(*args, **kwargs)
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
webserver    |     working_set = WorkingSet._build_master()
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 570, in _build_master
webserver    |     return cls._build_from_requirements(__requires__)
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 583, in _build_from_requirements
webserver    |     dists = ws.resolve(reqs, Environment())
webserver    |   File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
webserver    |     raise DistributionNotFound(req, requirers)
webserver    | pkg_resources.DistributionNotFound: The 'fideslib==2.1.0' distribution was not found and is required by fidesops

[sanders41]

Some of the dependencies listed there are older than the current ones in requirements.txt which makes me think it might be using an old container. If you run docker system prune --all -f --volumes first does it work after that?

[pattisdr]

trying again! I did run make clean before trying to run the quickstart

[pattisdr]

I did just get the quickstart running after your command, instead of clearing out with make clean, maybe we could change make clean to include the -f too. Would that have been the difference?

[pattisdr]

OK! make user and make quickstart are now both running. This PR looks good to me, per Paul's suggestion, I needed to run docker system prune --all -f --volumes, I was picking up an older container.

Unrelated to this PR, I can't complete the quickstart in full, because it looks for results of a privacy request and this now seems to be timing out. Possibly related to Celery.

Please wait...
INFO:__main__:Created fidesops privacy request for email=jane@example.com via /api/v1/privacy-request
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
INFO:__main__:Waiting for privacy request results...
Traceback (most recent call last):
  File "/fidesops/quickstart.py", line 705, in <module>
    print_results(request_id=privacy_request_id)
  File "/fidesops/quickstart.py", line 470, in print_results
    raise RuntimeError(
RuntimeError: fidesops privacy request results not found at results_path=fides_uploads/pri_45013dc2-c534-46bf-8c78-1aca823ff74d.json
Traceback (most recent call last):
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 309, in <module>
    run_infrastructure(
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 84, in run_infrastructure
    return _run_quickstart(path, IMAGE_NAME)
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 157, in _run_quickstart
    _run_cmd_or_err(f"docker exec -it {image_name} python quickstart.py")
  File "/Users/Dawn/fidesops/run_infrastructure.py", line 145, in _run_cmd_or_err
    raise Exception(f"Error executing command: {cmd}") 

EDIT: Turns out it was related!

[pattisdr]

thanks for tracking down that last issue with the quickstart @seanpreston and good find with the container_name @sanders41