Dockerize the setup

[amcewen]

As part of #16 (or maybe in prep for that), we should look at setting up the system on Docker. Probably want docker-compose with separate containers for:

• Postgres
• API server
• Website server
• Nginx as a front-end proxy

[ajlennon]

Sounds fun

[amcewen]

Assuming for now that we'll use docker-compose to set up three containers:

• web for the web server
• api for the API server
• db for a default postgres database server

The web and api servers will need their own Dockerfile to configure their container, we can use an off-the-shelf docker container for Postgres.

https://github.com/nodejs/docker-node/blob/main/README.md#how-to-use-this-image is a guide to using the official NodeJS container and extending it, and https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md seems a handy guide to read and apply for running them in production.

I've got the beginnings of a Dockerfile for the API server, but it's currently failing to install the dependencies, with node-sass seeming the most troublesome.

I think we might need to reorganise the directory structure of things, to split the API and web servers into separate projects with their own package.json and Dockerfile, but we'd need to work out how the dev and test environments would be affected by that. It can probably wait until I've got a working Dockerfile for at least the API server...

[amcewen]

This might have some useful tips - https://nickjanetakis.com/blog/best-practices-around-production-ready-web-apps-with-docker-compose

[amcewen]

Making some progress with this, all on the feature/17-dockerize-setup branch.

I'm splitting the package.json into two - one in api/ and one in website/, so that we can build them into separate docker containers.

In each folder there's also a Dockerfile which is responsible for building the container for that server. To build them you'd run (you might need to use sudo to run docker, I do on my Ubuntu machine):

cd api
docker build -t optimism/api .
cd ../website
docker build -t optimism/website .

Then running docker images should list two images: optimism/api and optimism/website. You need to then run the image to create the running container. There'll be a docker-compose config to manage that, but I've not done that yet. To test them individually I ran: docker run --name does-website -ti -p 3000:3000 --rm optimism/website or docker run --name does-api -ti -p 3001:3001 --rm optimism/api

If you're running one of the containers you can quit from it with Ctrl-P Ctrl-Q or run docker stop name-of-container in another terminal.

It won't do much (although the website container will serve you a perfectly good homepage) as the containers can't see each other, and the api container expects to connect to a Postgres database server, which will run in another container I've not set up yet. That plumbing will all come with the docker-compose setup

[amcewen]

I've added a docker-compose.yml config, which will build the docker images and then run up the three containers: optimism_db, optimism_api and optimism_website.

If you install docker-compose then you can run this to build and then start the containers:

docker-compose build
docker-compose up

There are an assortment of things still to do before this is ready:

• It should pull the database config from the environment, rather than having it hard-coded (Create test database with Knex #37 should resolve a lot of that, and we'll need to add the production environment)
• The environment variables in general need tidying up: the API URL is hard-coded in website/Dockerfile at present, and I'm not sure the other bits from .env get pulled in
• We need to work out how to run it for non-docker usage
• The package*.json files in root should be removed (unless some of the scripts to run the dev environment still need it)
• The postgres password needs to be changed, and moved into an environment variable
• We might want to move the code files in api/ and website/ into sub-folders so they're not sat at the same level as the package.json and Dockerfile? (not 100% sure this is necessary, but it feels like putting them in src/ would be neater?)
• Work out how we run migrations to set up the database (Create test database with Knex #37 will likely help here too)
• Work out how we populate the database

[ajlennon]

What you gonna run all this on? Balena? :)

[amcewen]

Hehe, nope. Digital Ocean is the current likely option.

[ajlennon]

Ooooh I can't stand them :)

[amcewen]

Well, you don't seem to be doing any of the work on it, so you're probably okay 😝

With the docker-compose setup, we can run the database migration like this: docker-compose exec api npx knex migrate:latest --env production and then seed the database (we'll want a better solution for this long-term, but for now...) with docker-compose exec api npx knex seed:run --env production

[amcewen]

Looks like npm workspaces might be the way to set up the dev environment now that we have api/package.json and website/package.json. However, that pulls in a requirement for npm v7, and my machine is currently running npm v6, so I'll need to upgrade things first.

I had a quick look at the Digital Ocean droplets, and there's a docker version available as a standard build. If no-one complains before I'm next working on this (which will be at least after the 9th July) then I'll delete the old droplet we were using to experiment with Odoo, etc. and spin up a new docker one to host optimism.

[ajlennon]

True enough. My main issue with it is you do all this work and then find you can’t export your droplets without massive problems….

I wish somebody had told me that before I started using them :)

[amcewen]

Worked tonight on getting the development environment working with the new re-jigged structure for docker.

Tried adding two npm workspaces for the api and website sub-projects, and they're getting symlinked in node_modules as optimism-api and optimism (because those are the names I've set up in their package.json files).

Running npm run api, npm run dev-all, etc. from the root folder seems to work fine, and finds the database, etc. However, none of the knex files are working, because it either:

• Can't find the knexfile.js because that's moved into api/ now, or
• If I run it and specify the location of the knexfile: npx knex --debug --knexfile api/knexfile.js migrate:latest then it fails because it's trying to open ./db/optimism.sqlite3 when it's in the api folder.

Actually, I've just tried creating another knexfile.js in the root, which pulls in the real one from api/ and that seems to have done the trick. Not sure if the migrations are in the right place or not yet - probably not right now. So I need to check that more in both the running locally and in docker versions to ensure that it's set up right.

So far, despite setting up the workspaces, I'm not sure they're getting used for much beyond the recursive installation of modules when you run npm install - I don't seem to be using it to run the dev servers (for example) and haven't worked out if there's a way to run them as part of the workspace.