Improve installation

[Didayolo]

Several issues are encountered when following this installation guide: https://github.com/codalab/codabench/wiki/Codabench-Installation

1. Email confirmation

• Improve the documentation.

When deploying a local instance, as the email server is not configured by default, it is tricky to validate user accounts.

One work-around is to:

1. Find the confirmation link in the Django logs using docker-compose logs -f django
2. Replace example.com by localhost to open it on the browser.

(Another work-around is to manage the account by hand in the Django container.)

2. Problem with storage

• We fixed MinIO buckets creation (Wait until MinIO is up before proceeding #783)
• We need to fix the policy setup to have directly the "read/write" policy on the public bucket (to fix the issue with logos not displayed). The fix: Use "mc anonymous set" instead of "mc policy set" for public bucket policy #786

codabench/docker-compose.yml

Line 76 in 6bf601a

/usr/bin/mc policy set download minio_docker/$AWS_STORAGE_BUCKET_NAME;

3. Step "run the following commands"

Maybe we just let it as it is.
At the end of the guide, we manually create the database and collect the static resources.
Can't this be handled directly by the docker-compose up -d command?

4. Endpoint URL

• Document this troubleshooting.

Some people (@bbearce) have to manually change the endpoint URL to have it working:

1. docker-compose logs -f minio
2. Grab first of these:

minio_1           | Browser Access:
minio_1           |    http://172.27.0.5:9000  http://127.0.0.1:9000

3. Set AWS_S3_ENDPOINT_URL=http://172.27.0.5:9000

[Didayolo]

About the storage problem, on my side the buckets creation failed.

At the beginning of the logs:

codabench-createbuckets-1   | Waiting for minio to startup...
codabench-createbuckets-1   | + echo 'Waiting for minio to startup...'
codabench-createbuckets-1   | + sleep 0.1
codabench-createbuckets-1   | + sleep 3
codabench-createbuckets-1   | + /usr/bin/mc config host add minio_docker http://minio:9000 testkey testsecret
codabench-createbuckets-1   | mc: <ERROR> Unable to initialize new alias from the provided credentials. Server not initialized, please try again.
codabench-createbuckets-1   | + /usr/bin/mc mb minio_docker/public
codabench-flower-1          | Traceback (most recent call last):
codabench-createbuckets-1   | Bucket created successfully `minio_docker/public`.
codabench-createbuckets-1   | + /usr/bin/mc mb minio_docker/private
codabench-createbuckets-1   | Bucket created successfully `minio_docker/private`.
codabench-createbuckets-1   | + /usr/bin/mc policy set download minio_docker/public
codabench-createbuckets-1   | mc: Please use 'mc anonymous'

When uploading a bundle:

codabench-django-1          | botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist

When login to MinIO at localhost:9000 and creating the buckets by hand, it then works: I am able to upload a bundle (but the competition logo is missing).

Note: this test was done on MacOS

[acletournel]

I have modified the wiki page with a few more hints for .env file, minio bucket policy configuration (the reason for logos not being dispayed) and the sequence of the different installation steps. @Didayolo @ihsaan-ullah : please, can you try again your local installation settings following these instructions ?

[Didayolo]

@acletournel

I see you have edited the following page:
https://github.com/codalab/codabench/wiki/How-to-deploy-Codabench-on-your-server

While I was working on improving the basic installation guide:
https://github.com/codalab/codabench/wiki/Codabench-Installation

The first one is about deploying a "production" environment, while the second is a simple local installation guide.

Maybe some information should be merged in order to have a complete setup guide. I'll have a look at it.

[Didayolo]

@acletournel

Please share your Flower error logs here.

[acletournel]

While working on local codabench instances, I found out that my flower docker container is unstable and continuously restarts:
I am using the current codabench/develop branch

CONTAINER ID   IMAGE                     COMMAND  CREATED            STATUS                                        PORTS                  NAMES
f995e89fc0d7   codabench_flower    "flower"     31 minutes ago   Restarting (1) 40 seconds ago                                codabench_flower_1

dockers logs show: ImportError:

cannot import name 'Command' from 'celery.bin.base' (/usr/local/lib/python3.11/site-packages/celery/bin/base.py)
which seems to be incompatibility with a new celery 5 version: mher/flower#1031

Flower is 0.9.3 according to Dockerfile.flower and celery is 4.2.1 according to requirements.txt ?

Codabench-test doesn't have the problem, but runs a flower dockerhub image mehr/flower and not a local built one.
Codabench-prod doesn't have the problem and run a local built image competitions-v2_flower.

[Didayolo]

@bbearce @acletournel

Some news. I realized that I'm able to access MinIO from http://localhost:9000 but not from http://minio:9000.

However, as you can see in the "create buckets" logs, Codabench tries to access it through http://minio:9000:

codabench-createbuckets-1   | + /usr/bin/mc config host add minio_docker http://minio:9000 testkey testsecret
codabench-createbuckets-1   | mc: <ERROR> Unable to initialize new alias from the provided credentials. Server not initialized, please try again.

This is not related to the AWS_S3_ENDPOINT_URL defined in .env file. It is actually hard coded in docker-compose.yml file:

codabench/docker-compose.yml

Line 73 in 46f68ee

/usr/bin/mc config host add minio_docker http://minio:$MINIO_PORT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY;

I guess there are two solutions:

• Edit the docker-compose.yml config
• Map minio to localhost in my local config

[Didayolo]

I think I found a fix!

[Didayolo]

I've added more information on the part "advanced configuration" of the guide:

https://github.com/codalab/codabench/wiki/Codabench-Installation