From bc41608554b9d39aa059e97b07bb06f4bf0bbb95 Mon Sep 17 00:00:00 2001 From: Kyle McCormick Date: Fri, 30 Jul 2021 11:22:07 -0400 Subject: [PATCH 1/6] docs: minor corrections to devserver-restarting docs 1. -restart doesn't work for every backend; -restart-devserver is more reliable. 2. -restart-devserver doesn't work on frontends; recommend attach and Ctrl+C instead. --- docs/devstack_interface.rst | 4 ++++ docs/workflow.rst | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/devstack_interface.rst b/docs/devstack_interface.rst index f5553faa68..5f12ce42c3 100644 --- a/docs/devstack_interface.rst +++ b/docs/devstack_interface.rst @@ -112,6 +112,10 @@ Useful Commands and Summary When to use: When automatic code reloading is not working and you need to manually restart a particular application server. + Tip: This only works for backend services. To restart frontends, use ``dev.attach.frontend-app-`` and ``Ctrl+C``. + + Variation: ``make -restart-devserver``. + - ``dev.restart-container.`` restarts service container. This is essentially a stronger version of ``dev.restrart-devserver`` Note: this will only restart and not its dependencies diff --git a/docs/workflow.rst b/docs/workflow.rst index 6e258bbe99..a1aa9de79f 100644 --- a/docs/workflow.rst +++ b/docs/workflow.rst @@ -27,11 +27,11 @@ These instructions are written using the LMS as an example. Replace ``lms`` with - If the logs show warning messages about missing tables or needed migrations, run ``make lms-migrate`` and then continue - - If there are complaints about import failures, Python package requirements may have changed since the last disk image. Run ``make lms-shell`` and then ``make requirements`` from inside the shell, then restart the service with ``make lms-restart``. + - If there are complaints about import failures, Python package requirements may have changed since the last disk image. Run ``make lms-shell`` and then ``make requirements`` from inside the shell, then restart the service with ``make lms-restart-devserver``. #. Your service should now be up and accessible, and you can develop in your IDA's repo. When you make changes on disk, a file watcher will restart the service in devstack. It may take a moment for the service to come back up with your changes. - - For some changes, this auto-restarting is insufficient, and you'll need to make a change from inside ``make lms-shell`` (such as ``make requirements`` or a migrations or other management command) and then run ``make lms-restart`` from the outside. + - For some changes, this auto-restarting is insufficient, and you'll need to make a change from inside ``make lms-shell`` (such as ``make requirements`` or a migrations or other management command) and then run ``make lms-restart-devserver`` from the outside. #. When you're done, you can either run ``make lms-stop`` to shut down the service but leave the container intact (with requirements installations and other file changes preserved) or ``make lms-down`` to destroy the containers as well. From 945f72e1cffa31f7d78d3c656142ef07adcd7ab2 Mon Sep 17 00:00:00 2001 From: Kyle McCormick Date: Thu, 17 Jun 2021 02:10:51 -0400 Subject: [PATCH 2/6] feat!: make frontend-app service and folder names consistent All micro-frontends in Devstack follow the frontend-app-* naming convention except course-authoring, gradebook, and program-console. This was presumably done to reduce typing, but adds confusion via its inconsistency. Furthermore, the upcoming change (in which backend services will depend on MFEs) will make it so these frontend names will be referenced much less often in the typical dev workflow. For example, `make dev.up.lms` will be sufficient to start Gradebook, so typing `make dev.up.frontend-app-gradebook` soon won't be necessary. This is a breaking interface change in that it renames the commands needed to start/stop/manage these frontends; however, the change is not expected to be breaking in any other way. --- README.rst | 12 ++++++------ docker-compose-host-nfs.yml | 22 +++++++++++----------- docker-compose-host.yml | 26 +++++++++++--------------- docker-compose-sync.yml | 18 +++++++++--------- docker-compose.yml | 22 +++++++++++----------- docker-sync.yml | 8 ++++---- docs/devstack_interface.rst | 4 ++-- options.mk | 4 ++-- 8 files changed, 56 insertions(+), 60 deletions(-) diff --git a/README.rst b/README.rst index f01fbaf1b2..e8a7e96fa9 100644 --- a/README.rst +++ b/README.rst @@ -300,15 +300,15 @@ Instead of a service name or list, you can also run commands like ``make dev.pro +------------------------------------+-------------------------------------+----------------+--------------+ | `frontend-app-publisher`_ | http://localhost:18400/ | MFE (React.js) | Default | +------------------------------------+-------------------------------------+----------------+--------------+ -| `gradebook`_ | http://localhost:1994/ | MFE (React.js) | Default | +| `frontend-app-gradebook`_ | http://localhost:1994/ | MFE (React.js) | Default | +------------------------------------+-------------------------------------+----------------+--------------+ | `registrar`_ | http://localhost:18734/api-docs/ | Python/Django | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ -| `program-console`_ | http://localhost:1976/ | MFE (React.js) | Extra | +| `frontend-app-program-console`_ | http://localhost:1976/ | MFE (React.js) | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ | `frontend-app-library-authoring`_ | http://localhost:3001/ | MFE (React.js) | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ -| `course-authoring`_ | http://localhost:2001/ | MFE (React.js) | Extra | +| `frontend-app-course-authoring`_ | http://localhost:2001/ | MFE (React.js) | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ | `xqueue`_ | http://localhost:18040/api/v1/ | Python/Django | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ @@ -326,15 +326,15 @@ Some common service combinations include: .. _forum: https://github.com/edx/cs_comments_service .. _frontend-app-payment: https://github.com/edx/frontend-app-payment .. _frontend-app-publisher: https://github.com/edx/frontend-app-publisher -.. _gradebook: https://github.com/edx/frontend-app-gradebook +.. _frontend-app-gradebook: https://github.com/edx/frontend-app-gradebook .. _lms: https://github.com/edx/edx-platform -.. _program-console: https://github.com/edx/frontend-app-program-console +.. _frontend-app-program-console: https://github.com/edx/frontend-app-program-console .. _registrar: https://github.com/edx/registrar .. _studio: https://github.com/edx/edx-platform .. _lms: https://github.com/edx/edx-platform .. _frontend-app-learning: https://github.com/edx/frontend-app-learning .. _frontend-app-library-authoring: https://github.com/edx/frontend-app-library-authoring -.. _course-authoring: https://github.com/edx/frontend-app-course-authoring +.. _frontend-app-course-authoring: https://github.com/edx/frontend-app-course-authoring .. _xqueue: https://github.com/edx/xqueue diff --git a/docker-compose-host-nfs.yml b/docker-compose-host-nfs.yml index 3932616d8f..2c1a4768bd 100644 --- a/docker-compose-host-nfs.yml +++ b/docker-compose-host-nfs.yml @@ -45,22 +45,22 @@ services: registrar-worker: volumes: - ${DEVSTACK_WORKSPACE}/registrar:/edx/app/registrar/registrar - gradebook: + frontend-app-gradebook: volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-gradebook:/edx/app/gradebook:cached - - gradebook_node_modules:/edx/app/gradebook/node_modules - program-console: + - ${DEVSTACK_WORKSPACE}/frontend-app-gradebook:/edx/app/frontend-app-gradebook:cached + - frontend_app_gradebook_node_modules:/edx/app/frontend-app-gradebook/node_modules + frontend-app-program-console: volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/program-console:cached - - program_console_node_modules:/edx/app/program-console/node_modules + - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/frontend-app-program-console:cached + - frontend_app_program_console_node_modules:/edx/app/frontend-app-program-console/node_modules frontend-app-learning: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-learning:/edx/app/frontend-app-learning:cached - frontend_app_learning_node_modules:/edx/app/frontend-app-learning/node_modules - course-authoring: + frontend-app-course-authoring: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-course-authoring:/edx/app/frontend-app-course-authoring:cached - - course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules + - frontend_app_course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules frontend-app-payment: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-payment:/edx/app/frontend-app-payment:cached @@ -81,12 +81,12 @@ volumes: edxapp_media: edxapp_node_modules: edxapp_uploads: - gradebook_node_modules: - program_console_node_modules: + frontend_app_gradebook_node_modules: + frontend_app_program_console_node_modules: frontend_app_learning_node_modules: frontend_app_payment_node_modules: frontend_app_publisher_node_modules: - course_authoring_node_modules: + frontend_app_course_authoring_node_modules: frontend_app_library_authoring_node_modules: edx-nfs: driver: local diff --git a/docker-compose-host.yml b/docker-compose-host.yml index 229caefbe3..208713b7ae 100644 --- a/docker-compose-host.yml +++ b/docker-compose-host.yml @@ -50,16 +50,14 @@ services: registrar-worker: volumes: - ${DEVSTACK_WORKSPACE}/registrar:/edx/app/registrar/registrar - gradebook: + frontend-app-gradebook: volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-gradebook:/edx/app/gradebook:cached - - gradebook_node_modules:/edx/app/gradebook/node_modules - - gradebook_tox:/edx/app/gradebook/.tox - program-console: + - ${DEVSTACK_WORKSPACE}/frontend-app-gradebook:/edx/app/frontend-app-gradebook:cached + - frontend_app_gradebook_node_modules:/edx/app/frontend-app-gradebook/node_modules + frontend-app-program-console: volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/program-console:cached - - program_console_node_modules:/edx/app/program-console/node_modules - - program_console_tox:/edx/app/program-console/.tox + - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/frontend-app-program-console:cached + - frontend_app_program_console_node_modules:/edx/app/frontend-app-program-console/node_modules frontend-app-payment: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-payment:/edx/app/frontend-app-payment:cached @@ -72,10 +70,10 @@ services: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-learning:/edx/app/frontend-app-learning:cached - frontend_app_learning_node_modules:/edx/app/frontend-app-learning/node_modules - course-authoring: + frontend-app-course-authoring: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-course-authoring:/edx/app/frontend-app-course-authoring:cached - - course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules + - frontend_app_course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules frontend-app-library-authoring: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-library-authoring:/edx/app/frontend-app-library-authoring:cached @@ -88,16 +86,14 @@ volumes: edxapp_media: edxapp_node_modules: edxapp_uploads: - gradebook_node_modules: - program_console_node_modules: + frontend_app_gradebook_node_modules: + frontend_app_program_console_node_modules: frontend_app_payment_node_modules: frontend_app_publisher_node_modules: frontend_app_learning_node_modules: - course_authoring_node_modules: + frontend_app_course_authoring_node_modules: frontend_app_library_authoring_node_modules: credentials_tox: discovery_tox: ecommerce_tox: edxapp_tox: - gradebook_tox: - program_console_tox: diff --git a/docker-compose-sync.yml b/docker-compose-sync.yml index 2a1118b609..4a7f058edb 100644 --- a/docker-compose-sync.yml +++ b/docker-compose-sync.yml @@ -29,17 +29,17 @@ services: volumes: - registrar-sync:/edx/app/registrar/registrar:nocopy - source-sync:/edx/src:nocopy - gradebook: + frontend-app-gradebook: volumes: - - gradebook-sync:/edx/app/gradebook/gradebook:nocopy + - frontend-app-gradebook-sync:/edx/app/frontend-app-gradebook:nocopy - source-sync:/edx/src:nocopy - program-console: + frontend-app-program-console: volumes: - - program-console-sync:/edx/app/program-console:nocopy + - frontend-app-program-console-sync:/edx/app/frontend-app-program-console:nocopy - source-sync:/edx/src:nocopy - course-authoring: + frontend-app-course-authoring: volumes: - - course-authoring-sync:/edx/app/frontend-app-course-authoring:nocopy + - frontend-app-course-authoring-sync:/edx/app/frontend-app-course-authoring:nocopy - source-sync:/edx/src:nocopy frontend-app-library-authoring: volumes: @@ -59,11 +59,11 @@ volumes: external: true registrar-sync: external: true - gradebook-sync: + frontend-app-gradebook-sync: external: true - program-console-sync: + frontend-app-program-console-sync: external: true - course-authoring-sync: + frontend-app-course-authoring-sync: external: true frontend-app-library-authoring-sync: external: true diff --git a/docker-compose.yml b/docker-compose.yml index e2a53874af..23f073a7eb 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -509,47 +509,47 @@ services: depends_on: - lms - gradebook: + frontend-app-gradebook: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/gradebook' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.gradebook" + working_dir: '/edx/app/frontend-app-gradebook' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-gradebook" networks: default: aliases: - - edx.devstack.gradebook + - edx.devstack.frontend-app-gradebook ports: - "1994:1994" depends_on: - lms - program-console: + frontend-app-program-console: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/program-console' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.program-console" + working_dir: '/edx/app/frontend-app-program-console' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-program-console" networks: default: aliases: - - edx.devstack.program-console + - edx.devstack.frontend-app-program-console ports: - "1976:1976" depends_on: - lms - registrar - course-authoring: + frontend-app-course-authoring: extends: file: microfrontend.yml service: microfrontend working_dir: '/edx/app/frontend-app-course-authoring' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.course-authoring" + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-course-authoring" networks: default: aliases: - - edx.devstack.course-authoring + - edx.devstack.frontend-app-course-authoring ports: - "2001:2001" depends_on: diff --git a/docker-sync.yml b/docker-sync.yml index 790bba7cdc..3e31a5c8ae 100644 --- a/docker-sync.yml +++ b/docker-sync.yml @@ -35,12 +35,12 @@ syncs: src: '../registrar/' sync_excludes: [ '.git', '.idea' ] - gradebook-sync: + frontend-app-gradebook-sync: host_disk_mount_mode: 'cached' - src: '../gradebook/' + src: '../frontend-app-gradebook/' sync_excludes: [ '.git', '.idea' ] - program-console-sync: + frontend-app-program-console-sync: host_disk_mount_mode: 'cached' src: '../frontend-app-program-console/' sync_excludes: [ '.git', '.idea' ] @@ -49,7 +49,7 @@ syncs: host_disk_mount_mode: 'cached' src: '../frontend-app-learning/' - course-authoring-sync: + frontend-app-course-authoring-sync: host_disk_mount_mode: 'cached' src: '../frontend-app-course-authoring/' sync_excludes: [ '.git', '.idea' ] diff --git a/docs/devstack_interface.rst b/docs/devstack_interface.rst index 5f12ce42c3..2e27417a50 100644 --- a/docs/devstack_interface.rst +++ b/docs/devstack_interface.rst @@ -15,8 +15,8 @@ Examples: make dev.shell.lms make lms-shell - make dev.logs.gradebook - make gradebook-logs + make dev.logs.frontend-app-gradebook + make frontend-app-gradebook-logs The user interface for devstack often also gives you both big hammers(``make dev.pull.large-and-slow``) and small hammers(``make dev.pull.``) to do things. It is recommend you opt for the small hammer commands, because they often tend to be alot faster. diff --git a/options.mk b/options.mk index 955d3a9fff..aca3eaa521 100644 --- a/options.mk +++ b/options.mk @@ -68,13 +68,13 @@ FS_SYNC_STRATEGY ?= local-mounts # The current value was chosen such that it would not change the existing # Devstack behavior. DEFAULT_SERVICES ?= \ -credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-payment+frontend-app-publisher+frontend-app-learning+gradebook+lms+studio +credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-payment+frontend-app-publisher+frontend-app-learning+frontend-app-gradebook+lms+studio # All edX services, whether or not they are run by default. # Separated by plus signs. # Separated by plus signs. Listed in alphabetical order for clarity. EDX_SERVICES ?= \ -course-authoring+credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-learning+frontend-app-library-authoring+frontend-app-payment+frontend-app-publisher+gradebook+lms+lms_watcher+program-console+registrar+registrar-worker+studio+studio_watcher+xqueue+xqueue_consumer +frontend-app-course-authoring+credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-learning+frontend-app-library-authoring+frontend-app-payment+frontend-app-publisher+frontend-app-gradebook+lms+lms_watcher+frontend-app-program-console+registrar+registrar-worker+studio+studio_watcher+xqueue+xqueue_consumer # Services with database migrations. # Should be a subset of $(EDX_SERVICES). From d1844be23cdf4acc81c198a6896d34c9b5c94f3f Mon Sep 17 00:00:00 2001 From: Kyle McCormick Date: Thu, 17 Jun 2021 02:25:53 -0400 Subject: [PATCH 3/6] style: alphabetize service definitions and dependencies Diffing / spotting issues in repetative config can be difficult. Alphabetization makes it easier. --- docker-compose-host-nfs.yml | 28 ++++----- docker-compose-host.yml | 54 ++++++++--------- docker-compose-sync.yml | 24 ++++---- docker-compose.yml | 112 ++++++++++++++++++------------------ docker-sync.yml | 20 +++---- options.mk | 4 +- 6 files changed, 122 insertions(+), 120 deletions(-) diff --git a/docker-compose-host-nfs.yml b/docker-compose-host-nfs.yml index 2c1a4768bd..596aab294f 100644 --- a/docker-compose-host-nfs.yml +++ b/docker-compose-host-nfs.yml @@ -45,34 +45,34 @@ services: registrar-worker: volumes: - ${DEVSTACK_WORKSPACE}/registrar:/edx/app/registrar/registrar + frontend-app-course-authoring: + volumes: + - ${DEVSTACK_WORKSPACE}/frontend-app-course-authoring:/edx/app/frontend-app-course-authoring:cached + - frontend_app_course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules frontend-app-gradebook: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-gradebook:/edx/app/frontend-app-gradebook:cached - frontend_app_gradebook_node_modules:/edx/app/frontend-app-gradebook/node_modules - frontend-app-program-console: - volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/frontend-app-program-console:cached - - frontend_app_program_console_node_modules:/edx/app/frontend-app-program-console/node_modules frontend-app-learning: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-learning:/edx/app/frontend-app-learning:cached - frontend_app_learning_node_modules:/edx/app/frontend-app-learning/node_modules - frontend-app-course-authoring: + frontend-app-library-authoring: volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-course-authoring:/edx/app/frontend-app-course-authoring:cached - - frontend_app_course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules + - ${DEVSTACK_WORKSPACE}/frontend-app-library-authoring:/edx/app/frontend-app-library-authoring:cached + - frontend_app_library_authoring_node_modules:/edx/app/frontend-app-library-authoring/node_modules frontend-app-payment: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-payment:/edx/app/frontend-app-payment:cached - frontend_app_payment_node_modules:/edx/app/frontend-app-payment/node_modules + frontend-app-program-console: + volumes: + - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/frontend-app-program-console:cached + - frontend_app_program_console_node_modules:/edx/app/frontend-app-program-console/node_modules frontend-app-publisher: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-publisher:/edx/app/frontend-app-publisher:cached - frontend_app_publisher_node_modules:/edx/app/frontend-app-publisher/node_modules - frontend-app-library-authoring: - volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-library-authoring:/edx/app/frontend-app-library-authoring:cached - - frontend_app_library_authoring_node_modules:/edx/app/frontend-app-library-authoring/node_modules volumes: credentials_node_modules: @@ -81,13 +81,13 @@ volumes: edxapp_media: edxapp_node_modules: edxapp_uploads: + frontend_app_course_authoring_node_modules: frontend_app_gradebook_node_modules: - frontend_app_program_console_node_modules: frontend_app_learning_node_modules: + frontend_app_library_authoring_node_modules: frontend_app_payment_node_modules: + frontend_app_program_console_node_modules: frontend_app_publisher_node_modules: - frontend_app_course_authoring_node_modules: - frontend_app_library_authoring_node_modules: edx-nfs: driver: local driver_opts: diff --git a/docker-compose-host.yml b/docker-compose-host.yml index 208713b7ae..833b6bfbdb 100644 --- a/docker-compose-host.yml +++ b/docker-compose-host.yml @@ -21,6 +21,9 @@ services: - ecommerce_node_modules:/edx/app/ecommerce/ecommerce/node_modules - ecommerce_tox:/edx/app/ecommerce/ecommerce/.tox - ${DEVSTACK_WORKSPACE}/src:/edx/src:cached + forum: + volumes: + - ${DEVSTACK_WORKSPACE}/cs_comments_service:/edx/app/forum/cs_comments_service:cached lms: volumes: - ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform:cached @@ -33,6 +36,12 @@ services: volumes: - ${DEVSTACK_WORKSPACE}/edx-notes-api:/edx/app/edx_notes_api/edx_notes_api:cached - ${DEVSTACK_WORKSPACE}/src:/edx/src:cached + registrar: + volumes: + - ${DEVSTACK_WORKSPACE}/registrar:/edx/app/registrar/registrar + registrar-worker: + volumes: + - ${DEVSTACK_WORKSPACE}/registrar:/edx/app/registrar/registrar studio: volumes: - ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform:cached @@ -41,43 +50,34 @@ services: - edxapp_tox:/edx/app/edxapp/edx-platform/.tox - edxapp_uploads:/edx/var/edxapp/uploads - ${DEVSTACK_WORKSPACE}/src:/edx/src:cached - forum: - volumes: - - ${DEVSTACK_WORKSPACE}/cs_comments_service:/edx/app/forum/cs_comments_service:cached - registrar: - volumes: - - ${DEVSTACK_WORKSPACE}/registrar:/edx/app/registrar/registrar - registrar-worker: + frontend-app-course-authoring: volumes: - - ${DEVSTACK_WORKSPACE}/registrar:/edx/app/registrar/registrar + - ${DEVSTACK_WORKSPACE}/frontend-app-course-authoring:/edx/app/frontend-app-course-authoring:cached + - frontend_app_course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules frontend-app-gradebook: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-gradebook:/edx/app/frontend-app-gradebook:cached - frontend_app_gradebook_node_modules:/edx/app/frontend-app-gradebook/node_modules - frontend-app-program-console: + frontend-app-learning: volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/frontend-app-program-console:cached - - frontend_app_program_console_node_modules:/edx/app/frontend-app-program-console/node_modules + - ${DEVSTACK_WORKSPACE}/frontend-app-learning:/edx/app/frontend-app-learning:cached + - frontend_app_learning_node_modules:/edx/app/frontend-app-learning/node_modules + frontend-app-library-authoring: + volumes: + - ${DEVSTACK_WORKSPACE}/frontend-app-library-authoring:/edx/app/frontend-app-library-authoring:cached + - frontend_app_library_authoring_node_modules:/edx/app/frontend-app-library-authoring/node_modules frontend-app-payment: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-payment:/edx/app/frontend-app-payment:cached - frontend_app_payment_node_modules:/edx/app/frontend-app-payment/node_modules + frontend-app-program-console: + volumes: + - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/frontend-app-program-console:cached + - frontend_app_program_console_node_modules:/edx/app/frontend-app-program-console/node_modules frontend-app-publisher: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-publisher:/edx/app/frontend-app-publisher:cached - frontend_app_publisher_node_modules:/edx/app/frontend-app-publisher/node_modules - frontend-app-learning: - volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-learning:/edx/app/frontend-app-learning:cached - - frontend_app_learning_node_modules:/edx/app/frontend-app-learning/node_modules - frontend-app-course-authoring: - volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-course-authoring:/edx/app/frontend-app-course-authoring:cached - - frontend_app_course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules - frontend-app-library-authoring: - volumes: - - ${DEVSTACK_WORKSPACE}/frontend-app-library-authoring:/edx/app/frontend-app-library-authoring:cached - - frontend_app_library_authoring_node_modules:/edx/app/frontend-app-library-authoring/node_modules volumes: credentials_node_modules: @@ -86,13 +86,13 @@ volumes: edxapp_media: edxapp_node_modules: edxapp_uploads: + frontend_app_course_authoring_node_modules: frontend_app_gradebook_node_modules: - frontend_app_program_console_node_modules: - frontend_app_payment_node_modules: - frontend_app_publisher_node_modules: frontend_app_learning_node_modules: - frontend_app_course_authoring_node_modules: frontend_app_library_authoring_node_modules: + frontend_app_payment_node_modules: + frontend_app_program_console_node_modules: + frontend_app_publisher_node_modules: credentials_tox: discovery_tox: ecommerce_tox: diff --git a/docker-compose-sync.yml b/docker-compose-sync.yml index 4a7f058edb..fb24c58815 100644 --- a/docker-compose-sync.yml +++ b/docker-compose-sync.yml @@ -29,22 +29,22 @@ services: volumes: - registrar-sync:/edx/app/registrar/registrar:nocopy - source-sync:/edx/src:nocopy - frontend-app-gradebook: - volumes: - - frontend-app-gradebook-sync:/edx/app/frontend-app-gradebook:nocopy - - source-sync:/edx/src:nocopy - frontend-app-program-console: - volumes: - - frontend-app-program-console-sync:/edx/app/frontend-app-program-console:nocopy - - source-sync:/edx/src:nocopy frontend-app-course-authoring: volumes: - frontend-app-course-authoring-sync:/edx/app/frontend-app-course-authoring:nocopy - source-sync:/edx/src:nocopy + frontend-app-gradebook: + volumes: + - frontend-app-gradebook-sync:/edx/app/frontend-app-gradebook:nocopy + - source-sync:/edx/src:nocopy frontend-app-library-authoring: volumes: - frontend-app-library-authoring-sync:/edx/app/frontend-app-library-authoring:nocopy - source-sync:/edx/src:nocopy + frontend-app-program-console: + volumes: + - frontend-app-program-console-sync:/edx/app/frontend-app-program-console:nocopy + - source-sync:/edx/src:nocopy volumes: credentials-sync: @@ -59,13 +59,13 @@ volumes: external: true registrar-sync: external: true - frontend-app-gradebook-sync: - external: true - frontend-app-program-console-sync: - external: true frontend-app-course-authoring-sync: external: true + frontend-app-gradebook-sync: + external: true frontend-app-library-authoring-sync: external: true + frontend-app-program-console-sync: + external: true source-sync: external: true diff --git a/docker-compose.yml b/docker-compose.yml index 23f073a7eb..de19c60321 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -194,9 +194,9 @@ services: container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.credentials" hostname: credentials.devstack.edx depends_on: - - mysql57 - - memcached - lms + - memcached + - mysql57 # Allows attachment to the credentials service using 'docker attach '. stdin_open: true tty: true @@ -219,9 +219,9 @@ services: container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.discovery" hostname: discovery.devstack.edx depends_on: - - mysql57 - elasticsearch7 - memcached + - mysql57 # Allows attachment to the discovery service using 'docker attach '. stdin_open: true tty: true @@ -247,10 +247,10 @@ services: container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.ecommerce" hostname: ecommerce.devstack.edx depends_on: - - mysql57 - - memcached - - lms - discovery + - lms + - memcached + - mysql57 # Allows attachment to the ecommerce service using 'docker attach '. stdin_open: true tty: true @@ -299,9 +299,9 @@ services: container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.forum" hostname: forum.devstack.edx depends_on: - - mongo - - memcached - elasticsearch7 + - memcached + - mongo image: edxops/forum:${OPENEDX_RELEASE:-latest} stdin_open: true tty: true @@ -318,12 +318,12 @@ services: hostname: lms.devstack.edx depends_on: - devpi - - mysql57 - - memcached - - mongo - discovery - - forum - elasticsearch7 + - forum + - memcached + - mongo + - mysql57 # Allows attachment to the LMS service using 'docker attach '. stdin_open: true tty: true @@ -427,11 +427,13 @@ services: hostname: studio.devstack.edx depends_on: - devpi - - mysql57 - elasticsearch7 - memcached - mongo - lms + - memcached + - mongo + - mysql57 # Allows attachment to the Studio service using 'docker attach '. stdin_open: true tty: true @@ -464,115 +466,116 @@ services: # for micro-frontends in devtack. # ========================================================================== - frontend-app-learning: + frontend-app-course-authoring: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/frontend-app-learning' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-learning" + working_dir: '/edx/app/frontend-app-course-authoring' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-course-authoring" networks: default: aliases: - - edx.devstack.frontend-app-learning + - edx.devstack.frontend-app-course-authoring ports: - - "2000:2000" + - "2001:2001" depends_on: - - lms + - studio - frontend-app-payment: + frontend-app-gradebook: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/frontend-app-payment' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-payment" + working_dir: '/edx/app/frontend-app-gradebook' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-gradebook" networks: default: aliases: - - edx.devstack.frontend-app-payment + - edx.devstack.frontend-app-gradebook ports: - - "1998:1998" + - "1994:1994" depends_on: - - ecommerce + - lms - frontend-app-publisher: + frontend-app-learning: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/frontend-app-publisher' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-publisher" + working_dir: '/edx/app/frontend-app-learning' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-learning" networks: default: aliases: - - edx.devstack.frontend-app-publisher + - edx.devstack.frontend-app-learning ports: - - "18400:18400" + - "2000:2000" depends_on: - lms - frontend-app-gradebook: + frontend-app-library-authoring: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/frontend-app-gradebook' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-gradebook" + working_dir: '/edx/app/frontend-app-library-authoring' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-library-authoring" networks: default: aliases: - - edx.devstack.frontend-app-gradebook + - edx.devstack.frontend-app-library-authoring ports: - - "1994:1994" + - "3001:3001" depends_on: - lms + - studio - frontend-app-program-console: + frontend-app-payment: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/frontend-app-program-console' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-program-console" + working_dir: '/edx/app/frontend-app-payment' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-payment" networks: default: aliases: - - edx.devstack.frontend-app-program-console + - edx.devstack.frontend-app-payment ports: - - "1976:1976" + - "1998:1998" depends_on: - - lms - - registrar + - ecommerce - frontend-app-course-authoring: + frontend-app-program-console: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/frontend-app-course-authoring' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-course-authoring" + working_dir: '/edx/app/frontend-app-program-console' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-program-console" networks: default: aliases: - - edx.devstack.frontend-app-course-authoring + - edx.devstack.frontend-app-program-console ports: - - "2001:2001" + - "1976:1976" depends_on: - - studio + - lms + - registrar - frontend-app-library-authoring: + frontend-app-publisher: extends: file: microfrontend.yml service: microfrontend - working_dir: '/edx/app/frontend-app-library-authoring' - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-library-authoring" + working_dir: '/edx/app/frontend-app-publisher' + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.frontend-app-publisher" networks: default: aliases: - - edx.devstack.frontend-app-library-authoring + - edx.devstack.frontend-app-publisher ports: - - "3001:3001" + - "18400:18400" depends_on: - lms - - studio volumes: discovery_assets: + devpi_data: edxapp_lms_assets: edxapp_studio_assets: elasticsearch_data: @@ -581,4 +584,3 @@ volumes: mongo_data: mysql_data: mysql57_data: - devpi_data: diff --git a/docker-sync.yml b/docker-sync.yml index 3e31a5c8ae..7f4a32d924 100644 --- a/docker-sync.yml +++ b/docker-sync.yml @@ -35,31 +35,31 @@ syncs: src: '../registrar/' sync_excludes: [ '.git', '.idea' ] - frontend-app-gradebook-sync: + frontend-app-course-authoring-sync: host_disk_mount_mode: 'cached' - src: '../frontend-app-gradebook/' + src: '../frontend-app-course-authoring/' sync_excludes: [ '.git', '.idea' ] - frontend-app-program-console-sync: + frontend-app-gradebook-sync: host_disk_mount_mode: 'cached' - src: '../frontend-app-program-console/' + src: '../frontend-app-gradebook/' sync_excludes: [ '.git', '.idea' ] frontend-app-learning-sync: host_disk_mount_mode: 'cached' src: '../frontend-app-learning/' - frontend-app-course-authoring-sync: + frontend-app-library-authoring-sync: host_disk_mount_mode: 'cached' - src: '../frontend-app-course-authoring/' + src: '../frontend-app-library-authoring/' sync_excludes: [ '.git', '.idea' ] - source-sync: + frontend-app-program-console-sync: host_disk_mount_mode: 'cached' - src: '../src/' + src: '../frontend-app-program-console/' sync_excludes: [ '.git', '.idea' ] - frontend-app-library-authoring-sync: + source-sync: host_disk_mount_mode: 'cached' - src: '../frontend-app-library-authoring/' + src: '../src/' sync_excludes: [ '.git', '.idea' ] diff --git a/options.mk b/options.mk index aca3eaa521..296d8b0bf9 100644 --- a/options.mk +++ b/options.mk @@ -68,13 +68,13 @@ FS_SYNC_STRATEGY ?= local-mounts # The current value was chosen such that it would not change the existing # Devstack behavior. DEFAULT_SERVICES ?= \ -credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-payment+frontend-app-publisher+frontend-app-learning+frontend-app-gradebook+lms+studio +credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-gradebook+frontend-app-payment+frontend-app-publisher+frontend-app-learning+lms+studio # All edX services, whether or not they are run by default. # Separated by plus signs. # Separated by plus signs. Listed in alphabetical order for clarity. EDX_SERVICES ?= \ -frontend-app-course-authoring+credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-learning+frontend-app-library-authoring+frontend-app-payment+frontend-app-publisher+frontend-app-gradebook+lms+lms_watcher+frontend-app-program-console+registrar+registrar-worker+studio+studio_watcher+xqueue+xqueue_consumer +credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-course-authoring+frontend-app-gradebook+frontend-app-learning+frontend-app-library-authoring+frontend-app-payment+frontend-app-program-console+frontend-app-publisher+lms+lms_watcher+registrar+registrar-worker+studio+studio_watcher+xqueue+xqueue_consumer # Services with database migrations. # Should be a subset of $(EDX_SERVICES). From 3e03cc4880d673436e3b662d18f0ff85c6fefa7f Mon Sep 17 00:00:00 2001 From: Kyle McCormick Date: Thu, 17 Jun 2021 02:29:39 -0400 Subject: [PATCH 4/6] refactor: pull xqueue service def into main docker-compose.yml It was once separated out because the idea of DEFAULT_SERVICES and a generalized `dev.up.*` command hadn't been developed, so everything in docker-compose.yml was always started. This is no longer the case (the Registrar extra service exists in docker-compose.yml yet isn't started by default). So, the separation of docker-compose-xqueue.yml is just confusing. Furthermore, by pulling xqueue and xqueue_consumer into docker-compose.yml, we can properly define mysql57 as its dependency. --- Makefile | 1 - docker-compose-xqueue.yml | 28 ---------------------------- docker-compose.yml | 28 ++++++++++++++++++++++++++++ 3 files changed, 28 insertions(+), 29 deletions(-) delete mode 100644 docker-compose-xqueue.yml diff --git a/Makefile b/Makefile index f1c0d8cfdb..fd83eb4e1c 100644 --- a/Makefile +++ b/Makefile @@ -81,7 +81,6 @@ ifeq ($(FS_SYNC_STRATEGY),local-mounts) COMPOSE_FILE := docker-compose-host.yml COMPOSE_FILE := $(COMPOSE_FILE):docker-compose-themes.yml COMPOSE_FILE := $(COMPOSE_FILE):docker-compose-watchers.yml -COMPOSE_FILE := $(COMPOSE_FILE):docker-compose-xqueue.yml endif # Files for use with Network File System -based synchronization. diff --git a/docker-compose-xqueue.yml b/docker-compose-xqueue.yml deleted file mode 100644 index 968221d858..0000000000 --- a/docker-compose-xqueue.yml +++ /dev/null @@ -1,28 +0,0 @@ -version: "2.1" - -services: - xqueue: - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.xqueue" - image: edxops/xqueue:${OPENEDX_RELEASE:-latest} - command: bash -c 'source /edx/app/xqueue/xqueue_env && while true; do python /edx/app/xqueue/xqueue/manage.py runserver 0.0.0.0:18040 ; sleep 2; done' - volumes: - - ${DEVSTACK_WORKSPACE}/xqueue:/edx/app/xqueue/xqueue:cached - # depends_on: even though we need mysql, we can't refer to it because it's started in the other compose file - networks: - default: - aliases: - - edx.devstack.xqueue - ports: - - 18040:18040 - - xqueue_consumer: - container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.xqueue_consumer" - image: edxops/xqueue:${OPENEDX_RELEASE:-latest} - command: bash -c 'source /edx/app/xqueue/xqueue_env && while true; do python /edx/app/xqueue/xqueue/manage.py run_consumer ; sleep 2; done' - volumes: - - ${DEVSTACK_WORKSPACE}/xqueue:/edx/app/xqueue/xqueue:cached - networks: - default: - aliases: - - edx.devstack.xqueue_consumer - # depends_on: even though we need mysql, we can't refer to it because it's started in the other compose file diff --git a/docker-compose.yml b/docker-compose.yml index de19c60321..7da1501eac 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -458,6 +458,34 @@ services: volumes: - edxapp_studio_assets:/edx/var/edxapp/staticfiles/ + xqueue: + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.xqueue" + image: edxops/xqueue:${OPENEDX_RELEASE:-latest} + command: bash -c 'source /edx/app/xqueue/xqueue_env && while true; do python /edx/app/xqueue/xqueue/manage.py runserver 0.0.0.0:18040 ; sleep 2; done' + volumes: + - ${DEVSTACK_WORKSPACE}/xqueue:/edx/app/xqueue/xqueue:cached + depends_on: + - mysql57 + networks: + default: + aliases: + - edx.devstack.xqueue + ports: + - 18040:18040 + + xqueue_consumer: + container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.xqueue_consumer" + image: edxops/xqueue:${OPENEDX_RELEASE:-latest} + command: bash -c 'source /edx/app/xqueue/xqueue_env && while true; do python /edx/app/xqueue/xqueue/manage.py run_consumer ; sleep 2; done' + volumes: + - ${DEVSTACK_WORKSPACE}/xqueue:/edx/app/xqueue/xqueue:cached + depends_on: + - mysql57 + networks: + default: + aliases: + - edx.devstack.xqueue_consumer + # ========================================================================== # edX Microfrontends # From 0b5f7769a071c07350ee7438c9afa53bb0f51b52 Mon Sep 17 00:00:00 2001 From: Kyle McCormick Date: Thu, 17 Jun 2021 02:39:43 -0400 Subject: [PATCH 5/6] feat!: invert dependencies so that backends now depend on frontends lms now depends on frontend-app-learning; previously, the converse was true. Running `make dev.up.lms` will start the frontend-app-learning container. This change has been made for all Devstack frontends/backends. This is breaking in that `make dev.up.frontend-app-*` is no longer the best way to start a frontend from a clean state, since it will not start related backend service(s). Instead, the backend service should be started, which will cause all related frontend apps to be started as dependencies. See included ADR (#4) for more context. Also includes some doc updates related to this change. TNL-8407 --- README.rst | 37 +++++------ docker-compose.yml | 26 +++----- .../0004-backends-depend-on-frontends.rst | 63 +++++++++++++++++++ docs/workflow.rst | 15 +++++ options.mk | 4 +- 5 files changed, 105 insertions(+), 40 deletions(-) create mode 100644 docs/decisions/0004-backends-depend-on-frontends.rst diff --git a/README.rst b/README.rst index e8a7e96fa9..84b5aa51da 100644 --- a/README.rst +++ b/README.rst @@ -13,27 +13,22 @@ The Devstack runs as multiple containers with `Docker Compose`_ at its core. A Devstack installation includes the following Open edX components by default: -* The Learning Management System (LMS) -* The Learning micro-frontend (A.K.A the new Courseware experience) +* The Learning Management System (LMS). +* LMS micro-frontends, including Gradebook and Learning (a.k.a. the "new courseware experience"). * Open Response Assessments (ORA2), among other LMS plug-ins. -* Open edX Studio -* Discussion Forums -* E-Commerce -* Credentials -* Notes -* Course Discovery -* Open edX Search -* A demonstration Open edX course -* The Publisher and Gradebook micro-frontends +* Discussion Forums. +* Open edX Studio, including the Library- and Course-Authoring micro-frontends. +* E-Commerce, including the Payment micro-frontend. +* Course Discovery, including the Publisher micro-frontend. +* Credentials. +* Notes. +* Open edX Search. +* A demonstration Open edX course. It also includes the following extra components: -* XQueue -* The Program Console micro-frontend -* The Library Authoring micro-frontend -* edX Registrar service. -* The course-authoring micro-frontend - +* XQueue and an example XQueue consumer. +* Registrar, including the Program Console micro-frontend. .. contents:: **Table of Contents:** @@ -302,13 +297,13 @@ Instead of a service name or list, you can also run commands like ``make dev.pro +------------------------------------+-------------------------------------+----------------+--------------+ | `frontend-app-gradebook`_ | http://localhost:1994/ | MFE (React.js) | Default | +------------------------------------+-------------------------------------+----------------+--------------+ -| `registrar`_ | http://localhost:18734/api-docs/ | Python/Django | Extra | +| `frontend-app-library-authoring`_ | http://localhost:3001/ | MFE (React.js) | Default | +------------------------------------+-------------------------------------+----------------+--------------+ -| `frontend-app-program-console`_ | http://localhost:1976/ | MFE (React.js) | Extra | +| `frontend-app-course-authoring`_ | http://localhost:2001/ | MFE (React.js) | Default | +------------------------------------+-------------------------------------+----------------+--------------+ -| `frontend-app-library-authoring`_ | http://localhost:3001/ | MFE (React.js) | Extra | +| `frontend-app-program-console`_ | http://localhost:1976/ | MFE (React.js) | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ -| `frontend-app-course-authoring`_ | http://localhost:2001/ | MFE (React.js) | Extra | +| `registrar`_ | http://localhost:18734/api-docs/ | Python/Django | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ | `xqueue`_ | http://localhost:18040/api/v1/ | Python/Django | Extra | +------------------------------------+-------------------------------------+----------------+--------------+ diff --git a/docker-compose.yml b/docker-compose.yml index 7da1501eac..557c633260 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -220,6 +220,7 @@ services: hostname: discovery.devstack.edx depends_on: - elasticsearch7 + - frontend-app-publisher - memcached - mysql57 # Allows attachment to the discovery service using 'docker attach '. @@ -248,6 +249,7 @@ services: hostname: ecommerce.devstack.edx depends_on: - discovery + - frontend-app-payment - lms - memcached - mysql57 @@ -321,6 +323,8 @@ services: - discovery - elasticsearch7 - forum + - frontend-app-gradebook + - frontend-app-learning - memcached - mongo - mysql57 @@ -354,6 +358,7 @@ services: hostname: registrar.devstack.edx depends_on: - discovery + - frontend-app-program-console - lms - mysql57 - memcached @@ -428,8 +433,9 @@ services: depends_on: - devpi - elasticsearch7 - - memcached - - mongo + - frontend-app-course-authoring + - frontend-app-library-authoring + - frontend-app-publisher - lms - memcached - mongo @@ -506,8 +512,6 @@ services: - edx.devstack.frontend-app-course-authoring ports: - "2001:2001" - depends_on: - - studio frontend-app-gradebook: extends: @@ -521,8 +525,6 @@ services: - edx.devstack.frontend-app-gradebook ports: - "1994:1994" - depends_on: - - lms frontend-app-learning: extends: @@ -536,8 +538,6 @@ services: - edx.devstack.frontend-app-learning ports: - "2000:2000" - depends_on: - - lms frontend-app-library-authoring: extends: @@ -551,9 +551,6 @@ services: - edx.devstack.frontend-app-library-authoring ports: - "3001:3001" - depends_on: - - lms - - studio frontend-app-payment: extends: @@ -567,8 +564,6 @@ services: - edx.devstack.frontend-app-payment ports: - "1998:1998" - depends_on: - - ecommerce frontend-app-program-console: extends: @@ -582,9 +577,6 @@ services: - edx.devstack.frontend-app-program-console ports: - "1976:1976" - depends_on: - - lms - - registrar frontend-app-publisher: extends: @@ -598,8 +590,6 @@ services: - edx.devstack.frontend-app-publisher ports: - "18400:18400" - depends_on: - - lms volumes: discovery_assets: diff --git a/docs/decisions/0004-backends-depend-on-frontends.rst b/docs/decisions/0004-backends-depend-on-frontends.rst new file mode 100644 index 0000000000..26cc3d2708 --- /dev/null +++ b/docs/decisions/0004-backends-depend-on-frontends.rst @@ -0,0 +1,63 @@ +4. Backend services now depend on frontend apps +----------------------------------------------- + +Status +====== + +Approved + + +Context +======= + +Micro-frontends as default experiences +************************************** + +As of mid June 2021 (between the Lilac and Maple releases), an Open edX instance with default configuration will now direct users to the Learning MFE (Micro-Frontend) for courseware, with a temporary opt-out flag existing to revert to legacy LMS-rendered frontend. Thus, to test a typical learner experience, Devstack users now require the frontend-app-learning container to be started alongside the LMS. This is in contrast to the previous state of affairs, in which MFE experiences were only available via an opt-IN flag, allowing reasonable Devstack usage without having to start any MFE containers. + +We anticipate that other learner, author, and administrator experiences will soon begin to use MFE features by default, requiring that more and more MFEs be started in order to simulate user experiences in Devstack. Thus, we anticipate an imminent developer experience issue, in which developers will need to type in convoluated commands like:: + + make dev.up.frontend-app-authn+frontend-app-discussions+frontend-app-gradebook+frontend-app-learning + + +in order to enable the feature set that was previously available using simply:: + + make dev.up.lms + + +Docker-compose service dependencies +*********************************** + +Devstack uses docker-compose to orchestrate containers by defining services in ``docker-compose.yml``. Note that "services" here encompasses backends, frontends, and generic resources like MySQL. + +Each service definition may indicate a list of depentent services using the ``depends_on`` key. Dependencies are transitive, and may not be cyclical. When a developer runs ``make dev.up.``, docker-compose is invoked in order to start both the service as well as its dependencies. For example, LMS depends on Mongo and Discovery, among other services. So, running ``make dev.up.lms`` will start not just LMS, but also Mongo, Discovery, all of Discovery's dependencies, and so on. + +Currently, micro-frontend services (those prefixed with ``frontend-app-``) are defined to depend on backends, but not vice versa. So, starting frontend-app-learning will automatically start LMS, but starting LMS will not automatically start frontend-app-learning. This makes sense under logic that "frontends depend on the APIs of backends in order to function". + +However, it can be argued that the opposite dependency relationship also makes sense. That is, one may assert that backends should depend on frontends in order to expose their APIs in a usable way. One could further assert that frontends shouldn't have hard dependencies on backend APIs, and should instead gracefully degrade when some or all of its APIs are unavailable. + + +Decision +======== + +Whichever dependency direction (frontends depend on backends, or vice versa) is more logically sound, we conclude that, for the purposes of Devstack, *asserting that backends depend on frontends is more useful to developers*. Specifically, it is beneficial to current and future developer workflows if ``make dev.up.lms`` automatically starts and learning-related frontends, ``make dev.up.studio`` automatically starts all authoring-related frontends, ``make dev.up.ecommerce`` starts all purchasing-related frontends, and so on. + +A necessary corollary to this decision is that *all micro-frontends required for default functionality must be included in devstack*. While it is encouraged that *all* new and existing micro-frontends are added to devstack using the pattern described above, it is absolutely necessary that MFEs which are required for out-of-the-box functionality be added to devstack. + + +Consequences +============ + +* ``docker-compose.yml`` will be updated to reflect that backend services depend on frontend-app services, not the other way around. Devstack documentation will be upated accordingly. +* ``docker-compose-host.yml`` will be updated to address an issue with local usage of JS packages, which currently forces some frontend development workflows to occur outside of devstack. The `documentation in frontend-build`_ will be updated accordingly. See `ADR 5`_ for details. +* An email and Slack message will be sent out to explain these changes and how we anticipate that they will impact developer workflows. The email will explain that if a micro-frontend is required to simulate common user story in the default configuration, then that frontend should be devstack, and should be automatically started by the relevant backend using ``depends_on``. + + +.. _documentation in frontend-build: https://github.com/edx/frontend-build#local-module-configuration-for-webpack +.. _ADR 5: ./0005-frontend-package-mounts.rst + +Rejected Alternatives +===================== + +* Keep the old dependency relationships, but add convenience targets (such as ``dev.up.domain.learning``) to start groups of related micro-frontends. We determine that this would increase the already-large congnitive overhead of the Devstack interface. +* Invert dependency relationships as described in this ADR, and also add targets such as ``make dev.up.lms-backend`` in order to start LMS without associated frontends. We determine that this would create a cascade of new inconsistencies in the Devstack interface: since only one of ``lms`` or ``lms-backend`` could exist as a docker-compose service, rules for the other would have to be hard-coded into the Makefile as special cases. diff --git a/docs/workflow.rst b/docs/workflow.rst index a1aa9de79f..f6ff466b15 100644 --- a/docs/workflow.rst +++ b/docs/workflow.rst @@ -54,3 +54,18 @@ Database backups ~~~~~~~~~~~~~~~~ You can routinely create backups of your local databases. To create a backup, use ``make dev.backup``. When you want to restore you database to the backup, run ``make dev.restore``. Warning, this will retore all your databases. You might have to cycle the database containers off and on using ``make dev.down.`` and ``make dev.up.``. + +Running micro-frontends outside of devstack +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Although several micro-frontends (MFEs) are built into devstack (the full list is in the `service table`_), some users prefer to run those MFEs directly on their host machine. You can achieve this by first removing the devstack MFE container, and then starting the host version. For example:: + + make dev.down.frontend-app-learning # Bring down the devstack version of the Learning MFE. + cd # Navigate to the Learning MFE's repository. + npm install && npm start # Install JS packages, and start the NPM devserver on your local host. + +Of course ``learning`` can be replaced with ``gradebook``, ``payment``, or another frontend-app name. + +If you forget to bring down the devstack version of the MFE, you will notice a port conflict when trying to start the host version. + +.. _service table: ../README.rst#service-list diff --git a/options.mk b/options.mk index 296d8b0bf9..087dfbcf02 100644 --- a/options.mk +++ b/options.mk @@ -58,6 +58,8 @@ FS_SYNC_STRATEGY ?= local-mounts # Services that are to be pulled, provisioned, run, and checked by default # when no services are specified manually. # Should be a subset of $(EDX_SERVICES). +# frontend-apps are not included here, but several of them are dependencies of default +# services. # Separated by plus signs. Listed in alphabetical order for clarity. # WARNING: You may remove services from this list in order to make Devstack lighter, # but beware that some services have implicit, undocumented dependencies on @@ -68,7 +70,7 @@ FS_SYNC_STRATEGY ?= local-mounts # The current value was chosen such that it would not change the existing # Devstack behavior. DEFAULT_SERVICES ?= \ -credentials+discovery+ecommerce+edx_notes_api+forum+frontend-app-gradebook+frontend-app-payment+frontend-app-publisher+frontend-app-learning+lms+studio +credentials+discovery+ecommerce+edx_notes_api+forum+lms+studio # All edX services, whether or not they are run by default. # Separated by plus signs. From 37a834e9d3d64179c0495d81877c3e4dec81670f Mon Sep 17 00:00:00 2001 From: Kyle McCormick Date: Thu, 29 Jul 2021 15:20:47 -0400 Subject: [PATCH 6/6] feat: mount src/ into /edx/app/src of frontend containers (#803) Allows installation of local versions of NPM packages via module.config.js, allowing frontend devs to test out frontend library changes within devstack. The mounts follow nearly the same pattern that micro-services do, which allows devs to develop local versions of Python packages alongside devstack micro-services. See included ADR #5 for details and rationale. TNL-8407 --- docker-compose-host.yml | 10 ++ .../0005-frontend-package-mounts.rst | 111 ++++++++++++++++++ 2 files changed, 121 insertions(+) create mode 100644 docs/decisions/0005-frontend-package-mounts.rst diff --git a/docker-compose-host.yml b/docker-compose-host.yml index 833b6bfbdb..6c87335db8 100644 --- a/docker-compose-host.yml +++ b/docker-compose-host.yml @@ -50,34 +50,44 @@ services: - edxapp_tox:/edx/app/edxapp/edx-platform/.tox - edxapp_uploads:/edx/var/edxapp/uploads - ${DEVSTACK_WORKSPACE}/src:/edx/src:cached + + # Note that frontends mount `src` to /edx/app/src instead of /edx/src. + # See ADR #5 for rationale. frontend-app-course-authoring: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-course-authoring:/edx/app/frontend-app-course-authoring:cached - frontend_app_course_authoring_node_modules:/edx/app/frontend-app-course-authoring/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached frontend-app-gradebook: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-gradebook:/edx/app/frontend-app-gradebook:cached - frontend_app_gradebook_node_modules:/edx/app/frontend-app-gradebook/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached frontend-app-learning: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-learning:/edx/app/frontend-app-learning:cached - frontend_app_learning_node_modules:/edx/app/frontend-app-learning/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached frontend-app-library-authoring: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-library-authoring:/edx/app/frontend-app-library-authoring:cached - frontend_app_library_authoring_node_modules:/edx/app/frontend-app-library-authoring/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached frontend-app-payment: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-payment:/edx/app/frontend-app-payment:cached - frontend_app_payment_node_modules:/edx/app/frontend-app-payment/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached frontend-app-program-console: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-program-console:/edx/app/frontend-app-program-console:cached - frontend_app_program_console_node_modules:/edx/app/frontend-app-program-console/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached frontend-app-publisher: volumes: - ${DEVSTACK_WORKSPACE}/frontend-app-publisher:/edx/app/frontend-app-publisher:cached - frontend_app_publisher_node_modules:/edx/app/frontend-app-publisher/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached volumes: credentials_node_modules: diff --git a/docs/decisions/0005-frontend-package-mounts.rst b/docs/decisions/0005-frontend-package-mounts.rst new file mode 100644 index 0000000000..fc189daf31 --- /dev/null +++ b/docs/decisions/0005-frontend-package-mounts.rst @@ -0,0 +1,111 @@ +5. Mounting frontend packages from src directory +------------------------------------------------ + +Synopsis +======== + +``${DEVSTACK_WORKSPACE}/src`` will be mounted at ``/edx/app/src`` within frontend containers, allowing locally-modified NPM packages to be tested via devstack. This will result in workflow changes for some frontend developers, which we will communicate via email. + +Status +====== + +Approved + + +Context +======= + +Current SOA: Local packages for backend services +************************************************ + +Backend devstack services currently mount the host folder ``${DEVSTACK_WORKSPACE}/src`` into their respective Docker containers at ``/edx/src``, making the contents of ``src`` available within the container. This enables developers to install local versions of Python packages into backend devstack services, as long as the package is placed within the host ``src`` folder. As a concrete user story: + +* A dev runs their devstack with ``~`` (home folder) as their ``${DEVSTACK_WORKSPACE}``. +* They would like to run edx-platform with a modified version of the ``completion`` Python package. +* So, they place their modified ``completion`` repository in ``~/src``. +* The dev's modified ``completion`` repository is now available to backend containers at ``/edx/src/completion``. +* Within ``make lms-shell``, they can now run ``pip install -e /edx/src/completion`` in order to install the modified package. + +This workflow is made possible via the ``${DEVSTACK_WORKSPACE}/src:/edx/src:cached`` volume declarations for each service that exist in docker-compose-host.yml. This line simply tells docker-compose to mount the ``src`` directory within the host devstack workspace to the ``/edx/src`` directory within a service's Docker container. + + +Current SOA: Local packages for frontends +***************************************** + +Unfortunately, this flow is currently *not* an option for frontend services (i.e., micro-frontends) when they're run via devstack. This was probably not an intentional omission; frontend services were added to devstack in a somewhat ad-hoc way, and the local-package workflow was probably overlooked. + +There is, however, an established strategy for using local packages when running frontends *outside* of devstack. This stategy is described in the `frontend-build documentation `_. Essentially, frontend package respositories can be placed anywhere in the host system, and each frontend's ``module.config.js`` can be pointed at those local respositories using a path relative to the frontend itself. For example: + +* A frontend dev has ``frontend-app-profile`` within their home folder (``~``). +* They would like to run a modified version of Paragon, located at ``~/paragon``. +* They create a ``module.config.js``, as recommended by the frontend-build docs, specifying ``../paragon`` as the path. +* They can now ``npm run build`` Paragon, and then install and start ``frontend-app-profile``, which will use their modified Paragon repository. + + +The issue: Making the frontend strategy work with devstack +********************************************************** + +With the acceptance of `ADR 4: Backend services now depend on frontend apps <./0004-backends-depend-on-frontends.rst>`_, it is more important than ever that devstack has a local package workflow for frontends. + +Unfortunately, the current backend and frontend strategies are incompatible in two ways: + +* The current frontend strategy allows package repositories to be placed anywhere in the filesystem, with the docs recommending them to be siblings of the ``frontend-app-...`` repositories. The backend strategy, on the other hand, requires packages to be placed within ``${DEVSTACK_WORKSPACE}/src``. +* The frontend strategy occurs entirely within the host system; directory mounting is not required. In the backend strategy, though, packages get mounted at ``/edx/src``. + +The implication of this is that local frontend package strategy for devstack will have to either: + +#. be slightly different than the current non-devstack local frontend package strategy, or +#. be implemented differently than devstack's current local backend package strategy. + + +Decision +======== + +We will introduce a local frontend package strategy to devstack that is (a) as similar in mechanism as possible to devstack's local backend package strategy, while (b) differing just enough to make it compatible with non-devstack frontend development. See **Consequences** for specifics. + +This is in observance of the `worse-is-better `_ design philosophy, which prioritizes simplicity of implementation over simplicity of interface. We hope that maintaining consistency with devstack's local package strategy will be worth the short-term frontend workflow confusion that this change may cause. + + +Consequences +============ + +In docker-compose-host.yml, each frontend service will be given a new volume declaration:: + + services: + + ... + + frontend-app-XX: + volumes: + - ${DEVSTACK_WORKSPACE}/frontend-app-XX:/edx/app/frontend-app-XX:cached + - frontend_app_XX_node_modules:/edx/app/frontend-app-XX/node_modules + - ${DEVSTACK_WORKSPACE}/src:/edx/app/src:cached # <--- This line is new! + +This will cause the ``${DEVSTACK_WORKSPACE}/src`` folder to mounted at ``/edx/app/src`` of each frontend service, similar to how that folder is mounted at ``/edx/src`` of each backend service. Via ``module.config.js``, frontend developers will then be able to specify ``../src/PACKAGE`` as the path of any local frontend package. This scheme has the benefit of: + +* working within a frontend Devstack container, since ``../src/PACKAGE`` resolves to ``/edx/app/src/PACKAGE``, and +* working oustide of Devstack, since ``../src/PACKAGE`` points to ``PACKAGE`` when ``src`` is a sibling of the frontend application repository. + +Developers will be informed of this scheme via a frontend-build documentation update and an email. + + +Rejected alternatives +===================== + + +Mount frontend packages at ``/edx/src`` +*************************************** + +One alternative would be to mount packages at ``/edx/src`` within frontend containers instead of ``/edx/app/src``. This approach would have been maximally consistent with the existing local backend package strategy. However, it would make it impossible for frontend developers to maintain a single ``module.config.js`` for both with-devstack and sans-devstack development. + +Concretely: Within a devstack container, in order to reference, say, ``/edx/src/paragon`` from an app running within ``/edx/app/frontend-app-profile``, one would need to specify the path ``../../src/paragon`` within ``module.config.js``. In order to reference the same package *outside* of devstack, the proper path would be ``../src/paragon`` (recall that ``src`` and ``frontend-app-profile`` are expected to be sibling directories, both within the devstack workspace). + + +Explicit frontend mounts in devstack workspace +********************************************** + +A more radical alternative would be to explicitly mount certain local frontend packages from the devstack workspace into each frontend container. For example, ``${DEVSTACK_WORKSPACE}/frontend-platform`` would be mounted into every frontend container (if it existed) at ``/edx/app/frontend-platform``. This would be done for a handful of other commonly-developed frontend packages, including Paragon and the branding packages. + +This approach would have been the most compatible with the existing local frontend package strategy, but it would sharply differ from devstack's backend package strategy. + +For reference, here is a draft implementation of this rejected approach: https://github.com/edx/devstack/pull/795.