@@ -11,9 +14,57 @@ SciDash is a project that enables the reproducible execution and visualization o
SciDash is a geppetto / django-based client-server web application.
-## Installation
+## Installation with Docker
-We recommend you to use a Python 3 (*3.6 or newer at least*) virtual environment for the installation, so you can keep all the dependencies within that environment.
+This installation procedure relies on the use of docker and docker-compose, if you don't have these already installed in your machine you can follow these [link 1](https://docs.docker.com/install/) and [link 2](https://docs.docker.com/compose/install/).
+
+In order to install scidash in your machine you will need to clone the scidash repository
+
+```
+git clone https://github.com/metacell/scidash.git
+```
+
+Then, we need to access the repository at the folder scidash/service/docker, for instance if you are using linux you can type in your terminal the below:
+
+```
+cd ./scidash/service/docker
+```
+
+Inside this folder you will find the 3 docker files needed to build and run scidash locally.
+Running the 3 commands below we will build 3 docker images that we will re-use through our docker-compose.yml file in order to run scidash. You can now proceed with the 3 commands below to build the 3 images required (note, it might take roughly 1 hour to build all the 3 images and the time can vary depending on your internet connection speed since these steps will have to download other software from the web):
+
+```
+docker build -f Dockerfile-postgres -t metacell/scidash_db:my_local_tag .
+docker build -f Dockerfile-virgo -t metacell/scidash_virgo:my_local_tag .
+docker build -f Dockerfile-scidash -t metacell/scidash:my_local_tag .
+```
+
+Once the 3 images have been created we can now move to the deployment folder, on the same level of the current docker folder, for instance in linux we can run:
+
+```
+cd ../deployment
+```
+
+At this point we need to edit the docker-compose.yml with our favourite editor (vim obviously, unless you are one of those who use emacs, I am sorry for you) and replace the tag with the one used during the manual build.
+If you did a copy paste of the commands, you will need to replace the tag latest with the my_local_tag string, for instance:
+
+ image: metacell/scidash_db:latest
+
+will become:
+
+ image: metacell/scidash_db:my_local_tag
+
+Once we finish to edit the file with the correct tag we are ready to run scidash, so sit down, relax and run:
+
+```
+docker-compose up -d
+```
+
+Wait 1-2 mins that docker compose will bring up all the service and enjoy Scidash!
+
+## Installation in my machine
+
+We recommend you to use a Python 3.6 virtual environment for the installation, so you can keep all the dependencies within that environment.
**Dependencies**
@@ -47,7 +98,25 @@ cp service/dotenv/env-docker .env
source .env
```
-Just a reminder before going forward that this project requires at least a Python 3.6 version, if this requirement is not satisfied before proceeding further ensure you have Python 3.6 (or bigger) installed.
+As a developer you may add the git pre-commit hook to you .git repo. Besides updating the coverage badge, this hook runs some more commands before committing your changes. To install the pre-commit hook copy the hook to your .git folder
+
+```shell script
+cp service/hooks/pre-commit .git
+chmod +x .git/hooks/pre-commit
+```
+
+You should install the requirements-dev.txt to use the coverage and more code testing tools
+
+```shell script
+pip install -r requirements-dev.txt
+```
+
+To update the coverage badge manually run
+```shell script
+make coverage-badge
+```
+
+Just a reminder before going forward that this project requires a Python 3.6 version, if this requirement is not satisfied before proceeding further ensure you have Python 3.6 installed.
#### ***Configure Database***
In order to configure the database you need the PostgreSQL server installed as per dependencies listed above, then you can proceed with the steps below that will need to be run as postgres user:
@@ -63,7 +132,7 @@ sudo su postgres
logout
```
-#### ***Backend and Fronend Installation***
+#### ***Backend and Frontend Installation***
Once done with the database configuration you can proceed with the backend (first) and the frontend (second) installation.
First we start with the backend installation with the command below:
```
@@ -87,6 +156,7 @@ make run-dev
Go to http://localhost:8000/ and enjoy!
+
## Requirements to neuronunit test and model classes to work with scidash
> Note: to save compatibility with uploaded results you'll be able to save classes which are not meet requirements, but there is no guarantee that they will work with application
@@ -182,126 +252,8 @@ pq.UnitQuantity('megaohm', pq.ohm*1e6, symbol='Mohm') # custom unit
{'v': pq.V, 'i': pq.pA} # mapping
```
-## Deployment
-
-For scidash test deployment there are configurations in deploy folder `$PROJECT_ROOT/service/kubernetes/scidash`
-
-What is what:
+## Post install steps
-`scidash-service.yaml`
+For copying/cloning initial model instances and test instances please update the DEMO_USER_ID in the settings file
+to point to the user (id) from where the models and test instances should be cloned.
-This configuration describes kubernetes service ([what is service](https://kubernetes.io/docs/concepts/services-networking/service/)) for scidash deployment.
-
-Section with general information:
-
-```yaml
-kind: Service
-apiVersion: v1
-metadata:
- name: scidash
- namespace: scidash-testing
- labels:
- app: scidash
-```
-
-Section with port mappings and other important information:
-
-```
-spec:
- type: LoadBalancer
- ports:
- - port: 80
- targetPort: 8000
- selector:
- app: scidash
-```
-
-This service in general is k8 resource with load balancer which provides access to the open ports from your pods ([what is pods](https://kubernetes.io/docs/concepts/workloads/pods/pod/))
-
-`scidash-deployment.yaml`
-
-This file provides management for deploying (and updating) pod with containers (application container and redis container).
-
-To better understanding you can compare k8 pods with composition created by docker-compose.
-
-Section with general information:
-
-```
-apiVersion: extensions/v1beta1
-kind: Deployment
-metadata:
- labels:
- app: scidash
- name: scidash
- namespace: scidash-testing
-```
-
-Start of actual specification for deployment:
-
-```
-spec:
- replicas: 1 # Count of the similar pods that should by launched
- selector:
- matchLabels:
- app: scidash
- strategy:
- rollingUpdate:
- maxSurge: 50%
- maxUnavailable: 50%
- type: RollingUpdate
-```
-Containers descriptions start here in template:
-
-```
- template:
- metadata:
- labels:
- app: scidash
- spec:
- containers:
-
- - image: r.cfcr.io/tarelli/metacell/scidash:deployment
- imagePullPolicy: Always
- name: scidash
- ports:
- - containerPort: 8000 # Ports that should be exposed
- protocol: TCP
-```
-
-Also example for environment description. And as you can see here it uses secrets ([what is secret](https://kubernetes.io/docs/concepts/configuration/secret/)) as a source for sensible data.
-
-```
- env:
- - name: DB_USER
- valueFrom:
- secretKeyRef:
- name: scidash-secret
- key: DB_USER
- - name: DB_PASSWORD
- valueFrom:
- secretKeyRef:
- name: scidash-secret
- key: DB_PASSWORD
-```
-Every secret should be mounted as a volume
-
-```
- volumeMounts:
- - name: scidash-secret
- mountPath: /scidash-secret
-```
-
-And described in volumes section on the same level as containers:
-```
- volumes:
- - name: scidash-secret
- secret:
- secretName: scidash-secret
-```
-
-Also codefresh repository requires especial secret for pulling images:
-
-```
- imagePullSecrets:
- - name: codefresh-generated-r.cfcr.io-cfcr-scidash-testing
-```
diff --git a/coverage.svg b/coverage.svg
new file mode 100644
index 00000000..d79e240d
--- /dev/null
+++ b/coverage.svg
@@ -0,0 +1,21 @@
+
+
diff --git a/docker-compose.yml b/docker-compose.yml
deleted file mode 100644
index 0a0203f5..00000000
--- a/docker-compose.yml
+++ /dev/null
@@ -1,16 +0,0 @@
-version: '2'
-
-services:
- scidash-redis:
- image: redis
- scidash-postgres:
- image: metacell/scidash-db
- ports:
- - 5432:5432
- scidash:
- image: metacell/scidash:latest
- ports:
- - 80:8000
- depends_on:
- - scidash-redis
- - scidash-postgres
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 1e835830..698c17b3 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,3 +1,7 @@
flake8
yapf
isort
+coverage
+coverage-badge
+django_coverage_plugin
+pip-tools
diff --git a/requirements.in b/requirements.in
index 04185443..0f343887 100644
--- a/requirements.in
+++ b/requirements.in
@@ -1,16 +1,36 @@
-psycopg2
-django==1.11.7
-channels==2.1.2
+# Please upgrade pip to min v21, the default version v18 of py36 has package resolving issues
+# e.g. when installing numba the py37 version of numpy will be installed and fails
+django==1.11.29
+psycopg2-binary==2.8.6
+channels==2.3.1
djangorestframework==3.7.1
drf-writable-nested
django-filter==1.1.0
-djangorestframework-jwt
-django-extensions
+django-timezone-field==3.1
+git+https://github.com/zsinnema/django-rest-framework-jwt@master
+django-extensions==2.1.6
django-dotenv
rest-framework-cache
django-material
-celery[redis]
-django-celery-beat
-django-celery-results
+celery[redis]==4.2.2
+django-celery-beat==1.4.0
+django-celery-results==1.0.4
django-db-logger
-git+git://github.com/ddelpiano/neuronunit@4.0.0#egg=neuronunit
+neo==0.9.0
+elephant==0.10.0
+pynn==0.9.5
+Jinja2==2.11.3
+lmfit==1.0.2
+numpy==1.19.5
+numba==0.50.1
+git+https://github.com/scidash/neuronunit@metacell
+git+https://github.com/scidash/sciunit@metacell
+git+https://github.com/MetaCell/scidash-api.git@master
+wheel
+sentry-sdk
+django-ckeditor==5.9.0
+django-admin-sortable2==0.7.5
+social-auth-app-django==4.0.0 # will be installed through install script install-backend
+#git+git://github.com/scidash/python-quantities@master # waiting for this PR https://github.com/python-quantities/python-quantities/pull/186 to be merged
+quantities==0.12.1
+llvmlite>=0.33.0.dev0,<0.34
diff --git a/requirements.txt b/requirements.txt
index 813c6d41..5c7cc12b 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,47 +1,170 @@
-#
-# This file is autogenerated by pip-compile
-# To update, run:
-#
-# pip-compile
-#
-amqp==2.4.2 # via kombu
-asgiref==2.3.2 # via channels
-async-timeout==3.0.1 # via asgiref
-attrs==19.1.0 # via automat, twisted
-autobahn==19.3.3 # via daphne
-automat==0.7.0 # via twisted
-billiard==3.5.0.5 # via celery
-celery[redis]==4.2.2
-channels==2.1.2
-constantly==15.1.0 # via twisted
-daphne==2.2.5 # via channels
+airspeed==0.5.17
+amqp==2.6.1
+apipkg==1.5
+asgiref==3.3.4
+asteval==0.9.23
+async-generator==1.10
+attrs==21.2.0
+autobahn==21.2.1
+Automat==20.2.0
+backcall==0.2.0
+beautifulsoup4==4.9.3
+billiard==3.5.0.5
+bleach==3.3.0
+bokeh==2.3.2
+bs4==0.0.1
+cachetools==4.2.2
+celery==4.2.2
+Cerberus==1.3.4
+certifi==2021.5.30
+cffi==1.14.5
+channels==2.3.1
+chardet==4.0.0
+click==8.0.1
+cloudpickle==1.6.0
+constantly==15.1.0
+contextvars==2.4
+cryptography==3.4.7
+cycler==0.10.0
+daphne==2.5.0
+dask==2021.3.0
+deap==1.3.1
+decorator==5.0.9
+deepdiff==5.5.0
+defusedxml==0.7.1
+distributed==2021.3.0
+Django==1.11.29
+django-admin-sortable2==0.7.5
django-celery-beat==1.4.0
django-celery-results==1.0.4
-django-db-logger==0.1.7
+django-ckeditor==5.9.0
+django-db-logger==0.1.10
django-dotenv==1.4.2
django-extensions==2.1.6
django-filter==1.1.0
-django-material==1.5.2
-django-timezone-field==3.0 # via django-celery-beat
-django==1.11.7
-djangorestframework-jwt==1.11.0
+django-js-asset==1.2.2
+django-material==1.9.0
+django-timezone-field==3.1
djangorestframework==3.7.1
-drf-writable-nested==0.5.1
-git+git://github.com/ddelpiano/neuronunit@4.0.0#egg=neuronunit
-hyperlink==18.0.0 # via twisted
-idna==2.8 # via hyperlink
-incremental==17.5.0 # via twisted
-kombu==4.3.0 # via celery
-psycopg2==2.7.7
-pyhamcrest==1.9.0 # via twisted
-pyjwt==1.7.1 # via djangorestframework-jwt
-python-crontab==2.3.6 # via django-celery-beat
-python-dateutil==2.8.0 # via python-crontab
-pytz==2018.9 # via celery, django, django-timezone-field
-redis==2.10.6 # via celery
+git+https://github.com/zsinnema/django-rest-framework-jwt@master
+dpath==2.0.1
+drf-writable-nested==0.6.3
+elephant==0.10.0
+enforce==0.3.4
+entrypoints==0.3
+execnet==1.8.1
+fsspec==2021.5.0
+future==0.18.2
+gitdb==4.0.7
+GitPython==3.1.17
+graphviz==0.16
+HeapDict==1.0.1
+hyperlink==21.0.0
+idna==2.10
+igor==0.3
+immutables==0.15
+importlib-metadata==3.10.1
+incremental==21.3.0
+ipykernel==5.5.5
+ipython==7.16.1
+ipython-genutils==0.2.0
+jedi==0.18.0
+Jinja2==2.11.3
+jsonpickle==2.0.0
+jsonschema==3.2.0
+jupyter-client==6.1.12
+jupyter-core==4.7.1
+jupyterlab-pygments==0.1.2
+kiwisolver==1.3.1
+kombu==4.3.0
+lazyarray==0.4.0
+libNeuroML==0.2.55
+llvmlite==0.36.0
+lmfit==1.0.2
+locket==0.2.1
+lxml==4.6.3
+MarkupSafe==2.0.1
+matplotlib==3.3.4
+mistune==0.8.4
+msgpack==1.0.2
+nbclient==0.5.3
+nbconvert==6.0.7
+nbformat==5.1.3
+neo==0.9.0
+nest-asyncio==1.5.1
+neuromllite==0.3.2
+#neuronunit==0.19
+git+https://github.com/scidash/neuronunit@master
+#numba==0.53.1
+numba==0.50.1
+numpy==1.19.5
+oauthlib==3.1.1
+ordered-set==4.0.2
+packaging==20.9
+pandas==1.1.5
+pandocfilters==1.4.3
+parso==0.8.2
+partd==1.2.0
+pexpect==4.8.0
+pickleshare==0.7.5
+Pillow==8.2.0
+prompt-toolkit==3.0.18
+psutil==5.8.0
+psycopg2-binary==2.8.6
+ptyprocess==0.7.0
+pyasn1==0.4.8
+pyasn1-modules==0.2.8
+pycparser==2.20
+-e git+https://github.com/MetaCell/pygeppetto-django.git@c2fff0fad0c9b941b7eb7983038c0c5f1b2f83a9#egg=pygeppetto_django
+Pygments==2.9.0
+PyJWT==2.1.0
+PyLEMS==0.5.2
+pylmeasure==0.2.0
+pyNeuroML==0.5.11
+PyNN==0.9.5
+pyOpenSSL==20.0.1
+pyparsing==2.4.7
+pyrsistent==0.17.3
+python-crontab==2.5.1
+python-dateutil==2.8.1
+python3-openid==3.2.0
+pytz==2021.1
+PyYAML==5.4.1
+pyzmq==22.1.0
+quantities==0.12.4
+quantities-scidash==0.12.4.3
+redis==2.10.6
+requests==2.25.1
+requests-oauthlib==1.3.0
rest-framework-cache==0.1
-six==1.12.0 # via autobahn, automat, django-extensions, pyhamcrest, python-dateutil, txaio
-twisted==18.9.0 # via daphne
-txaio==18.8.1 # via autobahn
-vine==1.3.0 # via amqp
-zope.interface==4.6.0 # via twisted
+scidash-api==1.2.0
+scipy==1.5.4
+sciunit @ git+https://github.com/scidash/sciunit@42f7aa1dba657f0cfb7d2b96efd5dbb2a6493c72
+sentry-sdk==1.1.0
+service-identity==21.1.0
+six==1.16.0
+smmap==4.0.0
+social-auth-app-django==4.0.0
+social-auth-core==4.1.0
+sortedcontainers==2.4.0
+soupsieve==2.2.1
+tblib==1.7.0
+tqdm==4.64.1
+testpath==0.5.0
+toolz==0.11.1
+tornado==6.1
+traitlets==4.3.3
+Twisted==21.2.0
+txaio==21.2.1
+typing-extensions==3.10.0.0
+uncertainties==3.1.5
+urllib3==1.26.5
+validators==0.18.2
+vine==1.3.0
+wcwidth==0.2.5
+webencodings==0.5.1
+websocket-client==1.0.1
+wrapt==1.12.1
+zict==2.0.0
+zipp==3.4.1
+zope.interface==5.4.0
diff --git a/scidash/account/api/views.py b/scidash/account/api/views.py
index 2a012c85..abc6fc74 100644
--- a/scidash/account/api/views.py
+++ b/scidash/account/api/views.py
@@ -22,7 +22,16 @@ def get_object(self):
class CheckIsLoggedView(views.APIView):
def get(self, request, format=None):
-
return response.Response(
{'is_logged': self.request.user.is_authenticated()}
)
+
+
+class SetShowInstructions(views.APIView):
+ def post(self, request, format=None):
+ show = request.data.get('show', None)
+ if show is not None:
+ user = ScidashUser.objects.get(id=request.user.id)
+ user.show_instructions = show
+ user.save()
+ return response.Response()
diff --git a/scidash/account/static/css/main.css b/scidash/account/static/css/main.css
index ab45a818..6e67671f 100644
--- a/scidash/account/static/css/main.css
+++ b/scidash/account/static/css/main.css
@@ -68,10 +68,9 @@ body {
.login-container {
position: fixed !important;
- top: 50%;
+ top: 40%;
left: 50%;
transform: translate(-50%, -50%) !important;
- width: 250px;
}
.password-reset-container {
@@ -96,4 +95,15 @@ body {
left: 50%;
transform: translate(-50%, -50%) !important;
width: 250px;
+}
+
+.btn.icon-btn {
+ float: left;
+ letter-spacing: 0px;
+ margin-top: 16px;
+ margin-right: 16px;
+ width: 208px;
+ display: flex;
+ justify-content: center;
+ align-items: center;
}
\ No newline at end of file
diff --git a/scidash/account/templates/registration/base-template.html b/scidash/account/templates/registration/base-template.html
index 4b921a0c..3b10fef0 100644
--- a/scidash/account/templates/registration/base-template.html
+++ b/scidash/account/templates/registration/base-template.html
@@ -3,6 +3,17 @@
+
+
+
+
+
+
diff --git a/scidash/account/templates/registration/login.html b/scidash/account/templates/registration/login.html
index 01f13a6a..fe1b3ce6 100644
--- a/scidash/account/templates/registration/login.html
+++ b/scidash/account/templates/registration/login.html
@@ -1,25 +1,57 @@
+{% include 'registration/base-template.html' %}
{% include 'material/includes/material_css.html' %}
{% include 'material/includes/material_js.html' %}
-{% include 'registration/base-template.html' %}
-
-{% load material_form %}
+{% load material_form %}
{% block content %}
{% endblock %}
-
-
diff --git a/scidash/account/templates/registration/password-reset-confirm.html b/scidash/account/templates/registration/password-reset-confirm.html
new file mode 100644
index 00000000..7024f2d0
--- /dev/null
+++ b/scidash/account/templates/registration/password-reset-confirm.html
@@ -0,0 +1,37 @@
+{% include 'registration/base-template.html' %}
+{% include 'material/includes/material_css.html' %}
+{% include 'material/includes/material_js.html' %}
+
+{% load material_form %}
+
+{% block content %}
+
+ {% if validlink %}
+
+
SCIDASH RESET PASSWORD
+
Please enter your new password twice so we can verify you typed it in correctly.
+
+
+
+ {% else %}
+
+
The password reset link was invalid, possibly because it has already been used. Please request a new password reset.
+
+ {% endif %}
+
+
+{% endblock %}
diff --git a/scidash/account/templates/registration/password-reset-done.html b/scidash/account/templates/registration/password-reset-done.html
index 7e4bc432..b756fa60 100644
--- a/scidash/account/templates/registration/password-reset-done.html
+++ b/scidash/account/templates/registration/password-reset-done.html
@@ -1,9 +1,9 @@
+{% include 'registration/base-template.html' %}
{% include 'material/includes/material_css.html' %}
{% include 'material/includes/material_js.html' %}
-{% include 'registration/base-template.html' %}
{% block content %}
-
+
PASSWORD RESET SENT
We've emailed you instructions for setting your password, if an account exists with the email you entered. You should receive them shortly.
diff --git a/scidash/account/templates/registration/password-reset.html b/scidash/account/templates/registration/password-reset.html
index 504dafe1..d901d46d 100644
--- a/scidash/account/templates/registration/password-reset.html
+++ b/scidash/account/templates/registration/password-reset.html
@@ -1,6 +1,6 @@
+{% include 'registration/base-template.html' %}
{% include 'material/includes/material_css.html' %}
{% include 'material/includes/material_js.html' %}
-{% include 'registration/base-template.html' %}
{% load material_form %}
@@ -10,6 +10,7 @@
SCIDASH RESET PASSWORD
Go home
diff --git a/scidash/account/templates/registration/signup.html b/scidash/account/templates/registration/signup.html
index 2efc9edb..773e0ae1 100644
--- a/scidash/account/templates/registration/signup.html
+++ b/scidash/account/templates/registration/signup.html
@@ -1,6 +1,6 @@
+{% include 'registration/base-template.html' %}
{% include 'material/includes/material_css.html' %}
{% include 'material/includes/material_js.html' %}
-{% include 'registration/base-template.html' %}
{% load material_form %}
diff --git a/scidash/account/views.py b/scidash/account/views.py
index f567d58b..7a96346f 100644
--- a/scidash/account/views.py
+++ b/scidash/account/views.py
@@ -1,8 +1,10 @@
from django.contrib.auth import authenticate, login
+from django.conf import settings
from django.shortcuts import redirect, render
from scidash.account.forms import ScidashUserCreationForm
-
+from scidash.sciunitmodels.models import ModelInstance
+from scidash.sciunittests.models import TestInstance
def signup(request):
if request.method == 'POST':
@@ -12,8 +14,22 @@ def signup(request):
username = form.cleaned_data.get('username')
raw_password = form.cleaned_data.get('password1')
user = authenticate(username=username, password=raw_password)
+
+ # clone initial models and tests to the new user
+ _clone_demo_models_and_tests(user)
+
login(request, user)
return redirect('index')
else:
form = ScidashUserCreationForm()
return render(request, 'registration/signup.html', {'form': form})
+
+
+def _clone_demo_models_and_tests(user):
+ demo_user_id = settings.SCIDASH_DEMO_USER_ID
+ if demo_user_id:
+ for mi in ModelInstance.objects.filter(owner_id=demo_user_id):
+ mi.clone_to_user(user).save()
+
+ for ti in TestInstance.objects.filter(owner_id=demo_user_id):
+ ti.clone_to_user(user).save()
diff --git a/scidash/general/admin.py b/scidash/general/admin.py
index f73fb977..3fa4a7f0 100644
--- a/scidash/general/admin.py
+++ b/scidash/general/admin.py
@@ -2,10 +2,24 @@
from django.contrib.auth.admin import UserAdmin
from django.contrib.contenttypes.models import ContentType
-from scidash.general.models import ScidashUser, Tag
+from adminsortable2.admin import SortableAdminMixin
+from scidash.general.models import ScidashUser, Tag, ContentItem
+
# Register your models here.
-admin.site.register(ScidashUser, UserAdmin)
+class ScidashUserAdmin(UserAdmin):
+ fieldsets = UserAdmin.fieldsets + (
+ (None, {'fields': ('show_instructions',)}),
+ )
+
+
+class ContentItemAdmin(SortableAdminMixin, admin.ModelAdmin):
+ list_display = ["name", "display_from", "display_to"]
+ date_hierarchy = "display_from"
+
+
+admin.site.register(ScidashUser, ScidashUserAdmin)
admin.site.register(ContentType)
admin.site.register(Tag)
+admin.site.register(ContentItem, ContentItemAdmin)
diff --git a/scidash/general/api/views.py b/scidash/general/api/views.py
index aaebb164..f60214ee 100644
--- a/scidash/general/api/views.py
+++ b/scidash/general/api/views.py
@@ -9,6 +9,7 @@
from rest_framework.views import APIView
from scidash.general.helpers import import_class
+from scidash.general.models import ContentItem
from scidash.sciunitmodels.helpers import download_and_save_model
from scidash.sciunitmodels.models import ModelInstance
from scidash.sciunittests.models import ScoreInstance, TestInstance
@@ -153,3 +154,26 @@ def post(self, request):
ScoreInstance.objects.bulk_create(scores)
return Response(result)
+
+
+class InstructionsView(APIView):
+ def get(self, request):
+ html = ""
+ for instructions in ContentItem.active_objects.all():
+ html += instructions.content
+ return Response(
+ {
+ 'instructions': html
+ }, 200
+ )
+
+class SettingsView(APIView):
+ def get(self, request):
+ return Response(
+ {
+ 'sentry': {
+ 'dsn': s.SENTRY_DSN,
+ 'env': s.SENTRY_ENV
+ }
+ }, 200
+ )
diff --git a/scidash/general/migrations/0008_contentitem.py b/scidash/general/migrations/0008_contentitem.py
new file mode 100644
index 00000000..3955c44f
--- /dev/null
+++ b/scidash/general/migrations/0008_contentitem.py
@@ -0,0 +1,27 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.23 on 2020-03-03 21:06
+from __future__ import unicode_literals
+
+import ckeditor.fields
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('general', '0007_tag'),
+ ]
+
+ operations = [
+ migrations.CreateModel(
+ name='ContentItem',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('content_order', models.PositiveIntegerField(default=0)),
+ ('content', ckeditor.fields.RichTextField()),
+ ],
+ options={
+ 'ordering': ['content_order'],
+ },
+ ),
+ ]
diff --git a/scidash/general/migrations/0009_contentitem_name.py b/scidash/general/migrations/0009_contentitem_name.py
new file mode 100644
index 00000000..7e436a58
--- /dev/null
+++ b/scidash/general/migrations/0009_contentitem_name.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.23 on 2020-03-03 21:16
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ('general', '0008_contentitem'),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name='contentitem',
+ name='name',
+ field=models.CharField(max_length=100),
+ preserve_default=False,
+ ),
+ ]
diff --git a/scidash/general/migrations/0010_auto_20200305_2013.py b/scidash/general/migrations/0010_auto_20200305_2013.py
new file mode 100644
index 00000000..8eaa3068
--- /dev/null
+++ b/scidash/general/migrations/0010_auto_20200305_2013.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.23 on 2020-03-05 20:13
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('general', '0009_contentitem_name'),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name='contentitem',
+ name='display_from',
+ field=models.DateTimeField(blank=True, null=True),
+ ),
+ migrations.AddField(
+ model_name='contentitem',
+ name='display_to',
+ field=models.DateTimeField(blank=True, null=True),
+ ),
+ ]
diff --git a/scidash/general/migrations/0011_scidashuser_show_instructions.py b/scidash/general/migrations/0011_scidashuser_show_instructions.py
new file mode 100644
index 00000000..0a6a0951
--- /dev/null
+++ b/scidash/general/migrations/0011_scidashuser_show_instructions.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.23 on 2020-03-16 17:00
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('general', '0010_auto_20200305_2013'),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name='scidashuser',
+ name='show_instructions',
+ field=models.BooleanField(default=True),
+ ),
+ ]
diff --git a/scidash/general/models.py b/scidash/general/models.py
index 70066e04..88eecf7e 100644
--- a/scidash/general/models.py
+++ b/scidash/general/models.py
@@ -1,10 +1,19 @@
+import datetime
+import json
+import re
+import urllib.request
+
from django.contrib.auth.models import AbstractUser
from django.contrib.contenttypes.fields import GenericForeignKey
from django.contrib.contenttypes.models import ContentType
from django.db import models
+from ckeditor.fields import RichTextField
+
class ScidashUser(AbstractUser):
+ show_instructions = models.BooleanField(default=True)
+
class Meta:
verbose_name = "Scidash user"
verbose_name_plural = "Scidash users"
@@ -21,3 +30,45 @@ class Tag(models.Model):
def __str__(self):
return self.name
+
+
+def oembedReplace(match):
+ url = "http://ckeditor.iframe.ly/api/oembed?url=" + \
+ re.search(r'(.*)', match.group()).group(1)
+ req = urllib.request.Request(url,
+ headers={"Referer": "https://scidash.org"})
+ f = urllib.request.urlopen(req)
+ obj = json.loads(f.read())
+ return obj['html']
+
+
+class ActiveContentItemManager(models.Manager):
+ def get_queryset(self):
+ now = datetime.datetime.now()
+ return super().get_queryset().filter(
+ models.Q(display_from__isnull=True) |
+ models.Q(display_from__lte=now),
+ models.Q(display_to__isnull=True) |
+ models.Q(display_to__gte=now),
+ ).order_by('content_order')
+
+
+class ContentItem(models.Model):
+ content_order = models.PositiveIntegerField(default=0, blank=False,
+ null=False)
+ name = models.CharField(max_length=100)
+ content = RichTextField()
+ display_from = models.DateTimeField(null=True, blank=True)
+ display_to = models.DateTimeField(null=True, blank=True)
+
+ objects = models.Manager()
+ active_objects = ActiveContentItemManager()
+
+ def save(self, *args, **kwargs):
+ # replace the oembed tags
+ p = re.compile(r'.*?')
+ self.content = p.sub(oembedReplace, self.content)
+ super().save(*args, **kwargs)
+
+ class Meta(object):
+ ordering = ['content_order']
diff --git a/scidash/general/serializers.py b/scidash/general/serializers.py
index 6a68ddff..b58a8bec 100644
--- a/scidash/general/serializers.py
+++ b/scidash/general/serializers.py
@@ -1,4 +1,4 @@
-from rest_framework import serializers
+from rest_framework import serializers, fields
from scidash.general.models import Tag
@@ -7,3 +7,49 @@ class TagSerializer(serializers.ModelSerializer):
class Meta:
model = Tag
fields = ('name', )
+
+
+class SerializerWritableMethodField(fields.ModelField):
+ """
+ A writable (ModelField base) SerializerMethodField that get its
+ representation from calling a method on the parent serializer class. The
+ method called will be of the form "get_{field_name}", and should take a
+ single argument, which is the object being serialized.
+
+ For example:
+
+ class ExampleSerializer(self):
+ class_name = SciDashSerializerWritableMethodField(
+ model_field=TestClass()._meta.get_field('class_name'))
+
+ def get_class_name(self, obj):
+ return ... # Calculate some data to return.
+ """
+
+ def __init__(self, method_name=None, **kwargs):
+ self.method_name = method_name
+ super(SerializerWritableMethodField, self).__init__(**kwargs)
+
+ def bind(self, field_name, parent):
+ # In order to enforce a consistent style, we error if a redundant
+ # 'method_name' argument has been used. For example:
+ # my_fld = serializer.SerializerMethodField(method_name='get_my_fld')
+ default_method_name = 'get_{field_name}'.format(field_name=field_name)
+ assert self.method_name != default_method_name, (
+ "It is redundant to specify `%s` on SerializerMethodField '%s' in "
+ "serializer '%s', because it is the same as the default method "
+ "name. Remove the `method_name` argument." %
+ (self.method_name, field_name, parent.__class__.__name__)
+ )
+
+ # The method name should default to `get_{field_name}`.
+ if self.method_name is None:
+ self.method_name = default_method_name
+
+ super(SerializerWritableMethodField, self).bind(
+ field_name, parent)
+
+ def to_representation(self, value):
+ method = getattr(self.parent, self.method_name)
+ return method(value)
+
diff --git a/scidash/general/tasks.py b/scidash/general/tasks.py
index d0ed042a..7eb376f3 100644
--- a/scidash/general/tasks.py
+++ b/scidash/general/tasks.py
@@ -4,8 +4,10 @@
import platform
from celery import shared_task
+from scidash.main.celery import app
from django.conf import settings as s
from websocket import WebSocketTimeoutException
+from sentry_sdk import capture_exception, capture_message
import pygeppetto_gateway as pg
from pygeppetto_gateway.interpreters.helpers import interpreter_detector
@@ -28,100 +30,125 @@ def get_project_id(raw_data):
def get_error(raw_data):
return raw_data
-def send_score_to_geppetto(score):
- db_logger.info(f'Processing score with ID {score.pk}')
- model_name = os.path.basename(score.model_instance.url)
- interpreter = import_class(interpreter_detector(score.model_instance.url))
-
- project_builder = pg.GeppettoProjectBuilder(
- score=score,
- interpreter=interpreter,
- project_location=f"{s.PYGEPPETTO_BUILDER_PROJECT_BASE_URL}/{score.owner}/{score.pk}/project.json", # noqa:E501
- xmi_location=f"{s.PYGEPPETTO_BUILDER_PROJECT_BASE_URL}/{score.owner}/{score.pk}/model.xmi", # noqa:E501
- model_file_location=f"{s.PYGEPPETTO_BUILDER_PROJECT_BASE_URL}/{score.owner}/{score.pk}/{model_name}", # noqa:E501
- )
-
- project_url = project_builder.build_project()
-
- servlet_manager = pg.GeppettoServletManager.get_instance('scheduler')
- servlet_manager.handle(S.LOAD_PROJECT_FROM_URL, project_url)
-
- project_loaded = False
- model_loaded = False
-
- project_id = None
-
- while not project_loaded and not model_loaded:
- try:
- response = json.loads(servlet_manager.read())
- except Exception as e:
- return e
-
- response_type = response.get('type')
-
- db_logger.info(response_type)
-
- if response_type == SR.GENERIC_ERROR or response_type == SR.ERROR_LOADING_PROJECT: # noqa: E501
- error = get_error(response.get('data'))
- db_logger.error(error)
-
- return error
-
- project_loaded = response_type == SR.PROJECT_LOADED
- model_loaded = response_type == SR.GEPPETTO_MODEL_LOADED
-
- if project_loaded:
- project_id = get_project_id(response.get('data'))
- db_logger.info(project_id)
- if project_id is None:
- return "Project not found"
-
- servlet_manager.handle(
- S.RUN_EXPERIMENT,
- json.dumps({
- 'projectId': project_id,
- 'experimentId': 1
- })
- )
-
- finished = False
- experiment_loaded = False
-
-
- while not finished:
- try:
- response = json.loads(servlet_manager.read())
- except WebSocketTimeoutException:
- db_logger.info('Successfully started experiment')
- finished = True
- except Exception as e:
- db_logger.error(e)
- score.error = e
- score.status = score.FAILED
- score.save()
-
- response_type = response.get('type')
-
- db_logger.info(response_type)
-
- if response_type == SR.ERROR_RUNNING_EXPERIMENT:
- error = get_error(response.get('data'))
- db_logger.error(error)
- score.error = error
- score.status = score.FAILED
- score.test_instance.build_info = f' {platform.system()}-{platform.release()}/{platform.system()}' # noqa: E501
- score.test_instance.hostname = 'Scidash Host'
- score.save()
- finished = True
-
- experiment_loaded = response_type == SR.EXPERIMENT_LOADED
-
- if experiment_loaded:
- db_logger.info(f'Score with ID {score.pk} successfully sent')
-
-
-@shared_task
+def send_score_to_geppetto(score):
+ try:
+ db_logger.info(f'Processing score with ID {score.pk}')
+ model_name = os.path.basename(score.model_instance.url)
+ interpreter = import_class(
+ interpreter_detector(score.model_instance.url))
+
+ project_builder = pg.GeppettoProjectBuilder(
+ score=score,
+ interpreter=interpreter,
+ project_location=f"{s.PYGEPPETTO_BUILDER_PROJECT_BASE_URL}"
+ f"/"
+ f"{score.owner}"
+ f"/"
+ f"{score.pk}"
+ f"/"
+ f"project.json",
+ xmi_location=f"{s.PYGEPPETTO_BUILDER_PROJECT_BASE_URL}"
+ f"/"
+ f"{score.owner}"
+ f"/"
+ f"{score.pk}"
+ f"/"
+ f"model.xmi",
+ model_file_location=f"{s.PYGEPPETTO_BUILDER_PROJECT_BASE_URL}"
+ f"/"
+ f"{score.owner}"
+ f"/"
+ f"{score.pk}"
+ f"/"
+ f"{model_name}",
+ )
+
+ project_url = project_builder.build_project()
+
+ servlet_manager = pg.GeppettoServletManager.get_instance('scheduler')
+ servlet_manager.handle(S.LOAD_PROJECT_FROM_URL, project_url)
+
+ project_loaded = False
+ model_loaded = False
+
+ project_id = None
+
+ while not project_loaded and not model_loaded:
+ try:
+ response = json.loads(servlet_manager.read())
+ except Exception as e:
+ return e
+
+ response_type = response.get('type')
+
+ db_logger.info(response_type)
+
+ if response_type == SR.GENERIC_ERROR or response_type == SR.ERROR_LOADING_PROJECT: # noqa: E501
+ error = get_error(response.get('data'))
+ db_logger.error(error)
+
+ return error
+
+ project_loaded = response_type == SR.PROJECT_LOADED
+ model_loaded = response_type == SR.GEPPETTO_MODEL_LOADED
+
+ if project_loaded:
+ project_id = get_project_id(response.get('data'))
+ db_logger.info(project_id)
+
+ if project_id is None:
+ return "Project not found"
+
+ servlet_manager.handle(
+ S.RUN_EXPERIMENT,
+ json.dumps({
+ 'projectId': project_id,
+ 'experimentId': 1
+ })
+ )
+
+ finished = False
+
+ while not finished:
+ try:
+ response = json.loads(servlet_manager.read())
+ except WebSocketTimeoutException as e:
+ capture_exception(e)
+ db_logger.info('Successfully started experiment')
+ finished = True
+
+ response_type = response.get('type')
+
+ db_logger.info(response_type)
+
+ if response_type == SR.ERROR_RUNNING_EXPERIMENT:
+ error = get_error(response.get('data'))
+ db_logger.error(error)
+ capture_message(error, app=True)
+ score.error = error
+ score.status = score.FAILED
+ score.test_instance.build_info = f' {platform.system()}' \
+ f'-' \
+ f'{platform.release()}' \
+ f'/' \
+ f'{platform.system()}'
+ score.test_instance.hostname = 'Scidash Host'
+ score.save()
+ finished = True
+
+ if response_type == SR.EXPERIMENT_LOADED:
+ db_logger.info(f'Score with ID {score.pk} successfully sent')
+
+ except Exception as e:
+ db_logger.error(e)
+ capture_exception(e)
+ score.error = e
+ score.status = Score.FAILED
+ score.save()
+
+
+@app.task
def run_experiment():
scores = list(Score.objects.filter(status=Score.SCHEDULED))
diff --git a/scidash/general/tests.py b/scidash/general/tests/__init__.py
similarity index 100%
rename from scidash/general/tests.py
rename to scidash/general/tests/__init__.py
diff --git a/scidash/general/tests/test_geppetto_servlet.py b/scidash/general/tests/test_geppetto_servlet.py
new file mode 100644
index 00000000..82b8df46
--- /dev/null
+++ b/scidash/general/tests/test_geppetto_servlet.py
@@ -0,0 +1,15 @@
+from django.test import TestCase
+
+from pygeppetto_gateway.base import GeppettoServletManager
+
+
+class GeppettoServletTest(TestCase):
+ @classmethod
+ def setUpClass(cls):
+ super(GeppettoServletTest, cls).setUpClass()
+ cls.servlet_manager = GeppettoServletManager()
+
+ def test_ws_address(self):
+ self.assertEqual(
+ self.servlet_manager.host,
+ "ws://scidash-virgo:8080/org.geppetto.frontend/GeppettoServlet")
diff --git a/scidash/general/urls.py b/scidash/general/urls.py
index b8d85ac8..7fb4bf47 100644
--- a/scidash/general/urls.py
+++ b/scidash/general/urls.py
@@ -4,5 +4,5 @@
urlpatterns = [
url(r'^upload/(?P[^/]+)$', FileUploadView.as_view()),
- url(r'^experiment-result/handle/$', GeppettoHandlerView.as_view())
+ url(r'^experiment-result/handle/$', GeppettoHandlerView.as_view()),
]
diff --git a/scidash/general/views.py b/scidash/general/views.py
index abae3972..3a55d2c6 100644
--- a/scidash/general/views.py
+++ b/scidash/general/views.py
@@ -1,6 +1,7 @@
import collections
import copy
import json
+import jsonpickle
import logging
import os
import re
@@ -47,6 +48,11 @@ def put(self, request, filename):
)
def populate_data(self, data, request):
+ related_data = data.get('related_data')
+ if related_data and isinstance(related_data, dict):
+ # related data is a dict, make it a string
+ data.update({'related_data': jsonpickle.encode(related_data, unpicklable=True, make_refs=False)})
+
score_serializer = ScoreInstanceSerializer(
data=data, context={'request': request}
)
diff --git a/scidash/logviewer/migrations/0003_auto_20200303_2106.py b/scidash/logviewer/migrations/0003_auto_20200303_2106.py
new file mode 100644
index 00000000..02a85575
--- /dev/null
+++ b/scidash/logviewer/migrations/0003_auto_20200303_2106.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.23 on 2020-03-03 21:06
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('logviewer', '0002_auto_20190520_1728'),
+ ]
+
+ operations = [
+ migrations.AlterField(
+ model_name='logfile',
+ name='path',
+ field=models.FilePathField(allow_folders=True, match='.*\\.log$', path='/opt/projects/metacell/scidash_new', recursive=True),
+ ),
+ ]
diff --git a/scidash/logviewer/migrations/0004_auto_20210603_1115.py b/scidash/logviewer/migrations/0004_auto_20210603_1115.py
new file mode 100644
index 00000000..c6dedcba
--- /dev/null
+++ b/scidash/logviewer/migrations/0004_auto_20210603_1115.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.29 on 2021-06-03 11:15
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('logviewer', '0003_auto_20200303_2106'),
+ ]
+
+ operations = [
+ migrations.AlterField(
+ model_name='logfile',
+ name='path',
+ field=models.FilePathField(allow_folders=True, match='.*\\.log$', path='/opt/projects/metacell/scidash/scidash', recursive=True),
+ ),
+ ]
diff --git a/scidash/main/celery.py b/scidash/main/celery.py
index aabd27a2..1126d14c 100644
--- a/scidash/main/celery.py
+++ b/scidash/main/celery.py
@@ -14,7 +14,7 @@
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'scidash.main.settings')
-app = Celery('proj', broker=os.environ.get('REDIS_URL', 'redis://'))
+app = Celery('proj', broker=os.environ.get('REDIS_URL', 'redis://'), include=['scidash.general.tasks',])
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
diff --git a/scidash/main/sentry.py b/scidash/main/sentry.py
new file mode 100644
index 00000000..9caa75e7
--- /dev/null
+++ b/scidash/main/sentry.py
@@ -0,0 +1,27 @@
+import os
+import sentry_sdk
+
+from django.conf import settings
+from sentry_sdk.integrations.django import DjangoIntegration
+from sentry_sdk.integrations.celery import CeleryIntegration
+from sentry_sdk.integrations.redis import RedisIntegration
+
+
+def init():
+ # sentry_sdk.init(
+ # dsn=settings.SENTRY_DSN,
+ # environment=settings.SENTRY_ENV,
+ # integrations=[CeleryIntegration(),
+ # DjangoIntegration(),
+ # RedisIntegration()],
+
+ # # Set traces_sample_rate to 1.0 to capture 100%
+ # # of transactions for performance monitoring.
+ # # We recommend adjusting this value in production.
+ # traces_sample_rate=1.0,
+
+ # # If you wish to associate users to errors (assuming you are using
+ # # django.contrib.auth) you may enable sending PII data.
+ # # send_default_pii=True
+ # )
+ pass
diff --git a/scidash/main/settings.py b/scidash/main/settings.py
index 26f700b7..f66a999a 100644
--- a/scidash/main/settings.py
+++ b/scidash/main/settings.py
@@ -16,6 +16,10 @@
import dotenv
from django.urls import reverse
+SENTRY_ENV = os.environ.get("ENVIRONMENT", "Production")
+SENTRY_DSN = os.environ.get('SENTRY_DSN', "")
+from .sentry import init as sentry_init
+
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -27,19 +31,22 @@
)
)
+# SECURITY WARNING: don't run with debug turned on in production!
+# Set via OS env, defaults to False
+DEBUG = TEMPLATE_DEBUG = os.environ.get('DEVELOPMENT', '0') == '1'
+
+sentry_init()
+
+
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.9/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '4*0@ca#ocm*(1=12m(bfb2p8e$sk-%i4xlj=%$wkj3*&gs!%sr'
-# SECURITY WARNING: don't run with debug turned on in production!
-# Se via OS env, defaults to False
-DEBUG = TEMPLATE_DEBUG = os.environ.get('DEBUG', False)
-
ALLOWED_HOSTS = [
- "*"
- ]
+ "*"
+]
# Application definition
@@ -63,7 +70,11 @@
'material',
'django_celery_beat',
'django_celery_results',
- 'django_db_logger'
+ 'django_db_logger',
+ 'ckeditor',
+ 'adminsortable2',
+ # Add the following django-allauth apps
+ 'social_django',
]
SCIDASH_APPS = [
@@ -87,16 +98,70 @@
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
+SOCIAL_AUTH_POSTGRES_JSONFIELD = True
+SOCIAL_AUTH_URL_NAMESPACE = 'social'
+SOCIAL_AUTH_LOGIN_REDIRECT_URL = '/'
+SOCIAL_AUTH_REDIRECT_IS_HTTPS = os.environ.get('HTTPS', '0') == '1' # production is behind a reverse proxy with https
+
+# see https://python-social-auth.readthedocs.io/en/latest/backends/index.html
+# for configation of social backends
+
+def get_secret(secret):
+ try:
+ sec_path = os.getenv('SECRETS_PATH','/etc/secrets')
+ with open(os.path.join(sec_path, secret)) as fh:
+ return fh.read()
+ except:
+ # if no secrets folder exists
+ return ''
+
+# GOOGLE
+# https://python-social-auth.readthedocs.io/en/latest/backends/google.html
+# see https://developers.google.com/identity/protocols/oauth2?csw=1#Registering
+# to get google client id (key) and secret
+SOCIAL_AUTH_GOOGLE_OAUTH2_KEY = get_secret('SOCIAL_AUTH_GOOGLE_OAUTH2_KEY')
+SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET = get_secret('SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET')
+
+# TWITTER
+# https://python-social-auth.readthedocs.io/en/latest/backends/twitter.html
+SOCIAL_AUTH_TWITTER_KEY = get_secret('SOCIAL_AUTH_TWITTER_KEY')
+SOCIAL_AUTH_TWITTER_SECRET = get_secret('SOCIAL_AUTH_TWITTER_SECRET')
+
+# GITHUB
+# https://python-social-auth.readthedocs.io/en/latest/backends/github.html
+SOCIAL_AUTH_GITHUB_KEY = get_secret('SOCIAL_AUTH_GITHUB_KEY')
+SOCIAL_AUTH_GITHUB_SECRET = get_secret('SOCIAL_AUTH_GITHUB_SECRET')
+
+AUTHENTICATION_BACKENDS = (
+ 'social_core.backends.open_id.OpenIdAuth',
+ 'social_core.backends.google.GoogleOAuth2',
+ 'social_core.backends.twitter.TwitterOAuth',
+ 'social_core.backends.github.GithubOAuth2',
+ 'django.contrib.auth.backends.ModelBackend',
+)
+
+JWT_AUTH = {
+ # how long the original token is valid for
+ 'JWT_EXPIRATION_DELTA': datetime.timedelta(hours=2),
+
+ # allow refreshing of tokens
+ 'JWT_ALLOW_REFRESH': True,
+
+ # this is the maximum time AFTER the token was issued that
+ # it can be refreshed. exprired tokens can't be refreshed.
+ 'JWT_REFRESH_EXPIRATION_DELTA': datetime.timedelta(days=7),
+}
+
REST_FRAMEWORK = {
# Use Django's standard `django.contrib.auth` permissions,
# or allow read-only access for unauthenticated users.
'DEFAULT_AUTHENTICATION_CLASSES': [
'scidash.account.auth.CsrfExemptSessionAuthentication',
- 'rest_framework_jwt.authentication.JSONWebTokenAuthentication'
- ],
+ 'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
+ ],
'DEFAULT_FILTER_BACKENDS': [
'django_filters.rest_framework.DjangoFilterBackend'
- ]
+ ]
}
ROOT_URLCONF = 'scidash.main.urls'
@@ -111,6 +176,8 @@
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
+ 'social_django.context_processors.backends',
+ 'social_django.context_processors.login_redirect',
'django.contrib.messages.context_processors.messages',
],
},
@@ -123,16 +190,15 @@
# https://docs.djangoproject.com/en/1.9/ref/settings/#databases
DATABASES = {
- 'default': {
- 'ENGINE': 'django.db.backends.postgresql',
- 'NAME': os.environ.get('DB_NAME'),
- 'USER': os.environ.get('DB_USER'),
- 'PASSWORD': os.environ.get('DB_PASSWORD'),
- 'HOST': os.environ.get('DB_HOST'),
- 'PORT': os.environ.get('DB_PORT'),
- }
- }
-
+ 'default': {
+ 'ENGINE': 'django.db.backends.postgresql',
+ 'NAME': os.environ.get('DB_NAME'),
+ 'USER': os.environ.get('DB_USER'),
+ 'PASSWORD': os.environ.get('DB_PASSWORD'),
+ 'HOST': os.environ.get('DB_HOST'),
+ 'PORT': os.environ.get('DB_PORT'),
+ }
+}
# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators
@@ -152,7 +218,6 @@
},
]
-
# Internationalization
# https://docs.djangoproject.com/en/1.9/topics/i18n/
@@ -193,7 +258,7 @@
},
},
'loggers': {
- 'django': {
+ 'django': {
'handlers': ['db_log'],
'level': 'ERROR',
'propagate': False,
@@ -208,7 +273,7 @@
AUTH_USER_MODEL = 'general.ScidashUser'
REST_FRAMEWORK_CACHE = {
- 'DEFAULT_CACHE_TIMEOUT': 86400, # Default is 1 day
+ 'DEFAULT_CACHE_TIMEOUT': 86400, # Default is 1 day
}
PYGEPPETTO_SOCKET_URL = 'org.geppetto.frontend/GeppettoServlet'
@@ -218,15 +283,16 @@
GEPPETTO_SERVLET_URL = os.environ.get(
'GEPPETTO_SERVLET_URL',
- 'ws://localhost:8080/org.geppetto.frontend/GeppettoServlet'
+ 'ws://scidash-virgo:8080/org.geppetto.frontend/GeppettoServlet'
)
GEPPETTO_BASE_URL = os.environ.get(
- 'GEPPETTO_BASE_URL', 'http://localhost:8080/org.geppetto.frontend/geppetto'
+ 'GEPPETTO_BASE_URL',
+ 'http://scidash-virgo:8080/org.geppetto.frontend/geppetto'
)
BASE_PROJECT_FILES_HOST = os.environ.get(
'BASE_PROJECT_FILES_HOST',
- 'http://localhost:8000/static/projects/'
+ 'http://scidash:8000/static/projects/'
)
ACCEPTABLE_SCORE_INSTANCES_AMOUNT = 50
@@ -236,10 +302,10 @@
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp.gmail.com'
-EMAIL_PORT = 465
-EMAIL_HOST_USER = 'user here'
-EMAIL_HOST_PASSWORD = 'password here'
-EMAIL_USE_SSL = True
+EMAIL_PORT = 587
+EMAIL_HOST_USER = 'email_username'
+EMAIL_HOST_PASSWORD = 'email_password'
+EMAIL_USE_TLS = True
POPULATE_USERS = True
@@ -250,3 +316,74 @@
CELERY_RESULT_BACKEND = 'django-db'
NO_IMPORT_TAG = 'unschedulable'
+
+# SCIDASH
+# DEMO user id used to clone initial models and tests from
+SCIDASH_DEMO_USER_ID = None
+
+# Initial search number of quarters to search back in time.
+SCIDASH_INITIAL_SEARCH_QUARTERS = 12
+
+CKEDITOR_CONFIGS = {
+ 'default': {
+ 'skin': 'moono',
+ # 'skin': 'office2013',
+ 'toolbar_Basic': [
+ ['Source', '-', 'Bold', 'Italic']
+ ],
+ 'toolbar_YourCustomToolbarConfig': [
+ {'name': 'document', 'items': ['Source', '-', 'Save', 'NewPage', 'Preview', 'Print', '-', 'Templates']},
+ {'name': 'clipboard', 'items': ['Cut', 'Copy', 'Paste', 'PasteText', 'PasteFromWord', '-', 'Undo', 'Redo']},
+ {'name': 'editing', 'items': ['Find', 'Replace', '-', 'SelectAll']},
+ {'name': 'forms',
+ 'items': ['Form', 'Checkbox', 'Radio', 'TextField', 'Textarea', 'Select', 'Button', 'ImageButton',
+ 'HiddenField']},
+ '/',
+ {'name': 'basicstyles',
+ 'items': ['Bold', 'Italic', 'Underline', 'Strike', 'Subscript', 'Superscript', '-', 'RemoveFormat']},
+ {'name': 'paragraph',
+ 'items': ['NumberedList', 'BulletedList', '-', 'Outdent', 'Indent', '-', 'Blockquote', 'CreateDiv', '-',
+ 'JustifyLeft', 'JustifyCenter', 'JustifyRight', 'JustifyBlock', '-', 'BidiLtr', 'BidiRtl',
+ 'Language']},
+ {'name': 'links', 'items': ['Link', 'Unlink', 'Anchor']},
+ {'name': 'insert',
+ 'items': ['Image', 'Flash', 'Table', 'HorizontalRule', 'Smiley', 'SpecialChar', 'PageBreak', 'Iframe']},
+ '/',
+ {'name': 'styles', 'items': ['Styles', 'Format', 'Font', 'FontSize']},
+ {'name': 'colors', 'items': ['TextColor', 'BGColor']},
+ {'name': 'tools', 'items': ['Maximize', 'ShowBlocks']},
+ {'name': 'about', 'items': ['About']},
+ '/', # put this to force next toolbar on new line
+ {'name': 'yourcustomtools', 'items': [
+ # put the name of your editor.ui.addButton here
+ 'Preview',
+ 'Maximize',
+ ]},
+ ],
+ 'toolbar': 'YourCustomToolbarConfig', # put selected toolbar config here
+ # 'toolbarGroups': [{ 'name': 'document', 'groups': [ 'mode', 'document', 'doctools' ] }],
+ # 'height': 291,
+ # 'width': '100%',
+ # 'filebrowserWindowHeight': 725,
+ # 'filebrowserWindowWidth': 940,
+ # 'toolbarCanCollapse': True,
+ # 'mathJaxLib': '//cdn.mathjax.org/mathjax/2.2-latest/MathJax.js?config=TeX-AMS_HTML',
+ 'tabSpaces': 4,
+ 'extraPlugins': ','.join([
+ 'uploadimage', # the upload image feature
+ # your extra plugins here
+ 'div',
+ 'autolink',
+ 'autoembed',
+ 'embedsemantic',
+ 'autogrow',
+ # 'devtools',
+ 'widget',
+ 'lineutils',
+ 'clipboard',
+ 'dialog',
+ 'dialogui',
+ 'elementspath'
+ ]),
+ }
+}
diff --git a/scidash/main/urls.py b/scidash/main/urls.py
index a9c2777e..3695d01c 100644
--- a/scidash/main/urls.py
+++ b/scidash/main/urls.py
@@ -21,7 +21,9 @@
from rest_framework_cache.registry import cache_registry
from rest_framework_jwt.views import obtain_jwt_token
-from scidash.account.api.views import CheckIsLoggedView, UserViewSet
+from scidash.account.api.views import CheckIsLoggedView, \
+ UserViewSet, \
+ SetShowInstructions
from scidash.account.views import signup
from scidash.general.api import views as general_views
from scidash.sciunitmodels.api import views as models_views
@@ -64,6 +66,7 @@
urlpatterns = [
url(r'^admin/', admin.site.urls),
url(r'^api/login/$', obtain_jwt_token),
+ url('', include('social_django.urls', namespace='social')),
url(r'^data/', include('scidash.general.urls')),
url(r'^api/date-range/$', DateRangeView.as_view(), name='date-range-view'),
url(r'^api/', include(router.urls)),
@@ -83,6 +86,14 @@
),
name='password-reset'
),
+ url(
+ r'^reset/(?P[0-9A-Za-z_\-]+)/(?P[0-9A-Za-z]{1,13}-[0-9A-Za-z]{1,20})/$',
+ auth_views.PasswordResetConfirmView.as_view(
+ template_name='registration/password-reset-confirm.html',
+ success_url="/",
+ post_reset_login=True
+ ),
+ name='password_reset_confirm'),
url(r'^auth/sign-up/$', signup, name='sign-up'),
url(
r'^api/users/me/$',
@@ -95,6 +106,21 @@
CheckIsLoggedView.as_view(),
name='is-logged'
),
+ url(
+ r'^api/users/toggle-show-instructions/$',
+ SetShowInstructions.as_view(),
+ name='set-show-instructions'
+ ),
+ url(
+ r'^api/instructions/$',
+ general_views.InstructionsView.as_view(),
+ name='instructions-view'
+ ),
+ url(
+ r'^api/settings/$',
+ general_views.SettingsView.as_view(),
+ name='instructions-view'
+ ),
url(r'^api/parameters/$', models_views.ModelParametersView.as_view()),
url(
r'^api/compatibility/$',
diff --git a/scidash/sciunitmodels/migrations/0022_auto_20200303_2106.py b/scidash/sciunitmodels/migrations/0022_auto_20200303_2106.py
new file mode 100644
index 00000000..f0fcea71
--- /dev/null
+++ b/scidash/sciunitmodels/migrations/0022_auto_20200303_2106.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.23 on 2020-03-03 21:06
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('sciunitmodels', '0021_auto_20190712_1041'),
+ ]
+
+ operations = [
+ migrations.AlterField(
+ model_name='modelclass',
+ name='import_path',
+ field=models.TextField(blank=True, default='', null=True),
+ ),
+ migrations.AlterField(
+ model_name='modelinstance',
+ name='timestamp',
+ field=models.DateTimeField(auto_now=True),
+ ),
+ ]
diff --git a/scidash/sciunitmodels/models.py b/scidash/sciunitmodels/models.py
index dd3d438b..7b3b86b7 100644
--- a/scidash/sciunitmodels/models.py
+++ b/scidash/sciunitmodels/models.py
@@ -84,7 +84,8 @@ def populate_capabilities(self):
): # noqa: E501
self.capabilities.add(capability_model)
else:
- extra_capability_model, created = ExtraCapabilityModelThrough.objects.get_or_create( # noqa: E501
+ extra_capability_model, created = ExtraCapabilityModelThrough.objects.get_or_create(
+ # noqa: E501
capability=capability_model,
model_class=self,
extra_check=extra_capabilities[capability]
@@ -133,3 +134,8 @@ class Meta:
def __str__(self):
return self.name
+
+ def clone_to_user(self, user):
+ self.pk = None
+ self.owner = user
+ return self
diff --git a/scidash/sciunitmodels/serializers.py b/scidash/sciunitmodels/serializers.py
index 3ab207cc..b82fcec2 100644
--- a/scidash/sciunitmodels/serializers.py
+++ b/scidash/sciunitmodels/serializers.py
@@ -1,3 +1,4 @@
+from scidash.general.helpers import import_class
from drf_writable_nested import WritableNestedModelSerializer
from rest_framework import serializers
@@ -21,11 +22,19 @@ class Meta:
class ModelClassSerializer(
GetByKeyOrCreateMixin, WritableNestedModelSerializer
):
- key = 'import_path'
+ key = 'url'
capabilities = CapabilitySerializer(many=True)
url = serializers.URLField(
allow_null=True, allow_blank=True, validators=[]
)
+ tooltip = serializers.SerializerMethodField()
+
+ def get_tooltip(self, model_class):
+ try:
+ c = import_class(model_class.import_path)
+ return c.description if c.description else ''
+ except Exception as e:
+ return ''
class Meta:
model = ModelClass
diff --git a/scidash/sciunitmodels/tests/test_data/score_object.json b/scidash/sciunitmodels/tests/test_data/score_object.json
index 70ad446b..9402ff18 100644
--- a/scidash/sciunitmodels/tests/test_data/score_object.json
+++ b/scidash/sciunitmodels/tests/test_data/score_object.json
@@ -1,11 +1,30 @@
{
- "score_class": {"class_name": "ZScore", "url": "http://test-url.for/not-spaming-data/in-database"},
+ "score_class": {
+ "class_name": "ZScore",
+ "url": "http://test-url.for/not-spaming-data/in-database"
+ },
"hash_id": "111",
"model_instance": {
"model_class": {
"class_name": "ReducedModel",
- "url": "http://test-url.for/not-spaming-data/in-database",
+ "url": "https://github.com/scidash/neuronunit/blob/master/neuronunit/models/NeuroML2/LEMS_2007One.xml",
+ "import_path": "neuronunit.models.static.StaticModel",
+ "memo": null,
+ "tooltip": "",
+ "extra_capabilities": [],
"capabilities": [
+ {
+ "class_name": "ReceivesSquareCurrent"
+ },
+ {
+ "class_name": "ProducesActionPotentials"
+ },
+ {
+ "class_name": "ProducesSpikes"
+ },
+ {
+ "class_name": "ProducesMembranePotential"
+ },
{
"class_name": "CanBeReduced"
}
@@ -25,18 +44,33 @@
"test_instance": {
"description": null,
"hash_id": "111",
- "test_suites": [{
- "hash": "testhash",
- "name": "ReducedSuite"
- }],
+ "test_suites": [
+ {
+ "hash": "testhash",
+ "name": "ReducedSuite"
+ }
+ ],
"test_class": {
- "class_name": "MyTest",
- "url": "http://test-url.for/not-spaming-data/in-database"
+ "class_name": "InputResistanceTest",
+ "url": "http://github.com/scidash/neuronunit.git",
+ "import_path": "neuronunit.tests.passive.InputResistanceTest",
+ "observation_schema": [
+ ["Mean, Standard Deviation, N", {"n": {"min": 1, "type": "integer"}, "std": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}],
+ ["Mean, Standard Error, N", {"n": {"min": 1, "type": "integer", "required": true}, "sem": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}]
+ ]
},
"observation": {
- "mean":"8",
- "std": "3",
- "url": ""
+ "mean": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "125.0",
+ "units": "megaohm"}},
+ "n": 10,
+ "std": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "40.0",
+ "units": "megaohm"}}
},
"verbose": 1
}
diff --git a/scidash/sciunitmodels/tests/test_sciunit_models.py b/scidash/sciunitmodels/tests/test_sciunit_models.py
index d0b825ac..91810f4a 100644
--- a/scidash/sciunitmodels/tests/test_sciunit_models.py
+++ b/scidash/sciunitmodels/tests/test_sciunit_models.py
@@ -1,5 +1,6 @@
import json
import os
+from scidash.sciunittests.models import TestClass
from django.test import Client, RequestFactory, TestCase
from django.urls import reverse
@@ -20,6 +21,13 @@ def setUpClass(cls):
factory = RequestFactory()
request = factory.get('/data/upload/sample_json.json')
+
+ cls.test_class = TestClass.objects.create(
+ class_name="InputResistanceTest",
+ import_path="neuronunit.tests.passive.InputResistanceTest",
+ url="http://github.com/scidash/neuronunit.git",
+ )
+
cls.user = ScidashUser.objects.create_user(
'admin', 'a@a.cc', 'montecarlo'
)
@@ -64,7 +72,7 @@ def test_if_capabilities_endpoint_works_correctly(self):
parsed_response = parsed_response.pop()
self.scrub(parsed_response, 'id')
capabilities_data = data.get('model_instance') \
- .get('model_class').get('capabilities').pop()
+ .get('model_class').get('capabilities').pop()
for key in capabilities_data.keys():
self.assertTrue(key in parsed_response)
@@ -86,7 +94,7 @@ def test_if_model_class_endpoint_works_correctly(self):
parsed_response = parsed_response.pop()
self.scrub(parsed_response, 'id')
model_class_data = data.get('model_instance') \
- .get('model_class')
+ .get('model_class')
for key in model_class_data.keys():
self.assertTrue(key in parsed_response)
@@ -118,18 +126,17 @@ def test_if_model_instance_endpoint_works_correctly(self):
class SciunitModelMatchingClassObjects(TestCase):
@classmethod
- def setUpClass(cls):
- super(SciunitModelMatchingClassObjects, cls).setUpClass()
-
- cls.model_class = {
+ def setUp(self):
+ self.model_class = {
"class_name": "ScoreModelClass",
"capabilities": [{
"class_name": "TestCapability"
}],
+ "import_path": "neuronunit.models.static.StaticModel",
"url": "http://test-score.url"
}
- cls.user = ScidashUser.objects.create_user(
+ self.user = ScidashUser.objects.create_user(
'admin', 'a@a.cc', 'montecarlo'
)
@@ -139,12 +146,6 @@ def test_is_model_class_match_the_same_object(self):
model_class_serializer = ModelClassSerializer(data=self.model_class)
- if model_class_serializer.is_valid():
- model_class_serializer.save()
-
- model_class_serializer = None
- model_class_serializer = ModelClassSerializer(data=self.model_class)
-
if model_class_serializer.is_valid():
model_class_serializer.save()
diff --git a/scidash/sciunittests/admin.py b/scidash/sciunittests/admin.py
index 5c2433f8..bb0131f0 100644
--- a/scidash/sciunittests/admin.py
+++ b/scidash/sciunittests/admin.py
@@ -8,8 +8,12 @@
# Register your models here.
+class ScoreInstanceAdmin(admin.ModelAdmin):
+ list_display = ('model_instance', 'test_instance', 'status', 'test_instance')
+ exclude = ('related_data',)
class TestClassModelAdmin(admin.ModelAdmin):
+ search_fields = ['class_name']
readonly_fields = [
'observation_schema', 'test_parameters_schema', 'params_units',
'units', 'memo'
@@ -44,7 +48,7 @@ class TestSuiteAdmin(admin.ModelAdmin):
admin.site.register(TestSuite, TestSuiteAdmin)
-admin.site.register(ScoreInstance)
+admin.site.register(ScoreInstance, ScoreInstanceAdmin)
admin.site.register(ScoreClass)
admin.site.register(TestInstance, TestInstanceModelAdmin)
admin.site.register(TestClass, TestClassModelAdmin)
diff --git a/scidash/sciunittests/api/views.py b/scidash/sciunittests/api/views.py
index 9213ef0e..2b94a6d4 100644
--- a/scidash/sciunittests/api/views.py
+++ b/scidash/sciunittests/api/views.py
@@ -21,14 +21,25 @@
class ScoreViewSet(viewsets.ReadOnlyModelViewSet):
- queryset = ScoreInstance.objects.all()
+ queryset = ScoreInstance.objects.all(). \
+ select_related('score_class','model_instance__model_class','model_instance__owner','test_instance__test_class','test_instance__owner','owner'). \
+ prefetch_related('model_instance__model_class__capabilities','model_instance__model_class__extra_capabilities',
+ 'test_instance__test_suites',
+ 'owner__user_permissions','owner__user_permissions__content_types','owner__groups',
+ 'model_instance__owner__user_permissions','model_instance__owner__user_permissions__content_types','model_instance__owner__groups',
+ 'model_instance__tags','model_instance__tags__content_type','model_instance__tags__content_object',
+ 'test_instance__owner__user_permissions','test_instance__owner__user_permissions__content_types','test_instance__owner__groups',
+ 'test_instance__test_suites__owner__user_permissions','test_instance__test_suites__owner__user_permissions__content_types','test_instance__test_suites__owner__groups',
+ 'test_instance__tags','test_instance__tags__content_type','test_instance__tags__content_object')
serializer_class = ScoreInstanceSerializer
permission_classes = (permissions.AllowAny, )
filter_class = ScoreFilter
class TestInstanceViewSet(viewsets.ModelViewSet):
- queryset = TestInstance.objects.all()
+ queryset = TestInstance.objects.all(). \
+ select_related('test_class','owner'). \
+ prefetch_related('test_suites','tags','tags__content_type','tags__content_object')
serializer_class = TestInstanceSerializer
permission_classes = (permissions.IsAuthenticatedOrReadOnly, )
filter_class = TestInstanceFilter
@@ -64,7 +75,7 @@ def filter_queryset(self, queryset):
class TestSuiteViewSet(viewsets.ReadOnlyModelViewSet):
- queryset = TestSuite.objects.all()
+ queryset = TestSuite.objects.all().select_related('owner')
serializer_class = TestSuiteSerializer
permission_classes = (permissions.AllowAny, )
filter_class = TestSuiteFilter
diff --git a/scidash/sciunittests/filters.py b/scidash/sciunittests/filters.py
index a75e2753..e24dd51c 100644
--- a/scidash/sciunittests/filters.py
+++ b/scidash/sciunittests/filters.py
@@ -78,7 +78,11 @@ def model_class_name_filter(self, queryset, name, value):
)
def with_suites_filter(self, queryset, name, value):
- tests = TestInstance.objects.prefetch_related('test_suites')
+ tests = TestInstance.objects. \
+ prefetch_related('test_suites',
+ 'owner__user_permissions','owner__user_permissions__content_type','owner__groups',
+ 'tags','tags__content_type','tags__content_object'). \
+ select_related('owner','test_class')
tests = tests.annotate(Count('test_suites'))
tests = tests.filter(test_suites__count__gt=0)
diff --git a/scidash/sciunittests/migrations/0047_auto_20191219_1217.py b/scidash/sciunittests/migrations/0047_auto_20191219_1217.py
new file mode 100644
index 00000000..3d8f5f1b
--- /dev/null
+++ b/scidash/sciunittests/migrations/0047_auto_20191219_1217.py
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.7 on 2019-12-19 12:17
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('sciunittests', '0046_auto_20190827_0221'),
+ ]
+
+ operations = [
+ migrations.AddIndex(
+ model_name='scoreinstance',
+ index=models.Index(fields=['-timestamp'], name='sciunittest_timesta_aa7614_idx'),
+ ),
+ ]
diff --git a/scidash/sciunittests/migrations/0048_scoreinstance_related_data.py b/scidash/sciunittests/migrations/0048_scoreinstance_related_data.py
new file mode 100644
index 00000000..765952ad
--- /dev/null
+++ b/scidash/sciunittests/migrations/0048_scoreinstance_related_data.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.29 on 2021-06-03 11:15
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('sciunittests', '0047_auto_20191219_1217'),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name='scoreinstance',
+ name='related_data',
+ field=models.TextField(blank=True, null=True),
+ ),
+ ]
diff --git a/scidash/sciunittests/models.py b/scidash/sciunittests/models.py
index 2f17f852..37579f59 100644
--- a/scidash/sciunittests/models.py
+++ b/scidash/sciunittests/models.py
@@ -11,7 +11,8 @@
from scidash.general.helpers import import_class
from scidash.sciunittests.constants import TEST_PARAMS_UNITS_TYPE
from scidash.sciunittests.helpers import (
- build_destructured_unit, get_observation_schema, get_test_parameters_schema,
+ build_destructured_unit, get_observation_schema,
+ get_test_parameters_schema,
get_units, get_default_params
)
@@ -20,13 +21,14 @@
db_logger = logging.getLogger('db')
+
class JSONEncoder(json.JSONEncoder):
def default(self, obj):
quantities.set_default_units(time='s', current='A')
if isinstance(obj, quantities.quantity.Quantity):
return float(obj.simplified.magnitude)
elif isinstance(obj, numbers.Number):
- # Noticed Sciunit does not use always quantities,
+ # Noticed Sciunit does not use always quantities,
# this will avoid the entire UI to explode
return float(obj)
else:
@@ -56,7 +58,7 @@ class TestClass(models.Model):
units = models.TextField(null=True, blank=True)
memo = models.TextField(null=True, blank=True)
params_units = JSONField(null=True, blank=True)
- default_params = JSONField(encoder=JSONEncoder , null=True, blank=True)
+ default_params = JSONField(encoder=JSONEncoder, null=True, blank=True)
class Meta:
verbose_name = 'Test class'
@@ -127,7 +129,7 @@ def clean_fields(self, exclude=None):
if params_schema is not None:
for key in params_schema:
params_units[key] = TEST_PARAMS_UNITS_TYPE[params_schema[key]
- ['type']]
+ ['type']]
self.units = units
self.params_units = params_units
@@ -172,6 +174,11 @@ class Meta:
def __str__(self):
return f"{self.name} - {self.test_class.class_name} instance"
+ def clone_to_user(self, user):
+ self.pk = None
+ self.owner = user
+ return self
+
class ScoreClass(models.Model):
class_name = models.CharField(max_length=200)
@@ -219,6 +226,7 @@ class ScoreInstance(models.Model):
timestamp = models.DateTimeField(default=date.today)
owner = models.ForeignKey(general_models.ScidashUser, default=None)
error = models.TextField(null=True, blank=True)
+ related_data = models.TextField(null=True, blank=True)
def save(self, *args, **kwargs):
super().save(*args, **kwargs)
@@ -243,6 +251,9 @@ def prediction(self):
class Meta:
ordering = ['-timestamp']
+ indexes = [
+ models.Index(fields=['-timestamp', ]),
+ ]
def __str__(self):
return "Score for {0} in {1} test instance".format(
diff --git a/scidash/sciunittests/serializers.py b/scidash/sciunittests/serializers.py
index b5471bbb..5cf6ff8e 100644
--- a/scidash/sciunittests/serializers.py
+++ b/scidash/sciunittests/serializers.py
@@ -1,23 +1,21 @@
-import json
+import jsonpickle
-import numpy as np
from drf_writable_nested import WritableNestedModelSerializer
-from rest_framework import serializers
+from rest_framework import serializers, fields
-import sciunit
from scidash.account.serializers import ScidashUserSerializer
from scidash.general.helpers import import_class
from scidash.general.mixins import GetByKeyOrCreateMixin, GetOrCreateMixin
-from scidash.general.serializers import TagSerializer
+from scidash.general.serializers import TagSerializer, \
+ SerializerWritableMethodField
from scidash.sciunitmodels.serializers import ModelInstanceSerializer
-from scidash.sciunittests.helpers import build_destructured_unit
+from scidash.sciunittests.validators import TestInstanceValidator
from scidash.sciunittests.models import (
ScoreClass, ScoreInstance, TestClass, TestInstance, TestSuite
)
class TestSuiteSerializer(GetOrCreateMixin, WritableNestedModelSerializer):
-
owner = ScidashUserSerializer(
default=serializers.CurrentUserDefault(), read_only=True
)
@@ -28,11 +26,28 @@ class Meta:
class TestClassSerializer(
- GetByKeyOrCreateMixin, WritableNestedModelSerializer
-):
+ GetByKeyOrCreateMixin, WritableNestedModelSerializer):
+ class_name = SerializerWritableMethodField(
+ model_field=TestClass()._meta.get_field('class_name'))
units_name = serializers.CharField(required=False)
key = 'import_path'
+ tooltip = serializers.SerializerMethodField()
+
+ def get_tooltip(self, test_class):
+ try:
+ c = import_class(test_class.import_path)
+ return c.description if c.description else ''
+ except Exception as e:
+ return ''
+
+ def get_class_name(self, obj):
+ # return class_name + ( first part of import_path )
+ return obj.class_name + (
+ ' (' +
+ '.'.join((obj.import_path if obj.import_path else ''
+ ).split('.')[0:-1]) + ')').replace(' ()', '')
+
class Meta:
model = TestClass
fields = '__all__'
@@ -51,72 +66,12 @@ class TestInstanceSerializer(
key = 'hash_id'
def validate(self, data):
- sciunit.settings['PREVALIDATE'] = True
-
- class_data = data.get('test_class')
-
- if not class_data.get('import_path', False):
- return data
-
- test_class = import_class(class_data.get('import_path'))
-
- try:
- destructured = json.loads(class_data.get('units'))
- except json.JSONDecodeError:
- quantity = import_class(class_data.get('units'))
- else:
- if destructured.get('name', False):
- quantity = build_destructured_unit(destructured)
- else:
- quantity = destructured
-
- observations = data.get('observation')
- without_units = []
-
- def filter_units(schema):
- result = []
- for key, rules in schema.items():
- if not rules.get('units', False):
- result.append(key)
-
- return result
-
- if isinstance(test_class.observation_schema, list):
- for schema in test_class.observation_schema:
- if isinstance(schema, tuple):
- without_units += filter_units(schema[1])
- else:
- without_units += filter_units(schema)
- else:
- without_units = filter_units(test_class.observation_schema)
-
- def process_obs(obs):
- try:
- obs = int(obs)
- except ValueError:
- obs = np.array(json.loads(obs))
-
- return obs
-
- if not isinstance(quantity, dict):
- obs_with_units = {
- x: (
- process_obs(y) * quantity
- if x not in without_units else process_obs(y)
- )
- for x, y in observations.items()
- }
- else:
- obs_with_units = {
- x: (
- process_obs(y) * import_class(quantity[x])
- if x not in without_units else process_obs(y)
- )
- for x, y in observations.items()
- }
+ fallback_observation_schema = TestClass.objects.get(
+ import_path=data.get('test_class').get('import_path')
+ ).observation_schema
try:
- test_class(obs_with_units)
+ TestInstanceValidator.validate(data, fallback_observation_schema)
except Exception as e:
raise serializers.ValidationError(
f"Can't instantiate class, reason candidates: {e}"
@@ -132,8 +87,38 @@ class Meta:
class ScoreClassSerializer(
GetByKeyOrCreateMixin, WritableNestedModelSerializer
):
+ def create(self, validated_data):
+ model = self.Meta.model
+ relations, reverse_relations = self._extract_relations(validated_data)
+
+ self.update_or_create_direct_relations(
+ validated_data,
+ relations,
+ )
- key = 'class_name'
+ key_url = validated_data.get("url")
+ key_class_name = validated_data.get("class_name")
+
+ if key_url != "" and key_class_name != "":
+ try:
+ model_instance = model.objects.get(**{"url": key_url, "class_name": key_class_name})
+ instance = super(GetByKeyOrCreateMixin,
+ self).update(model_instance, validated_data)
+ except model.DoesNotExist:
+ instance = super(GetByKeyOrCreateMixin,
+ self).create(validated_data)
+ else:
+ if not validated_data.get('id', False):
+ instance = super(GetByKeyOrCreateMixin,
+ self).create(validated_data)
+ else:
+ model_instance = model.objects.get(pk=validated_data.get('id'))
+ instance = super(GetByKeyOrCreateMixin,
+ self).update(model_instance, validated_data)
+
+ self.update_or_create_reverse_relations(instance, reverse_relations)
+
+ return instance
class Meta:
model = ScoreClass
@@ -154,6 +139,14 @@ class ScoreInstanceSerializer(
key = 'hash_id'
+ def __init__(self, *args, **kwargs):
+ # Instantiate the superclass normally
+ super(ScoreInstanceSerializer, self).__init__(*args, **kwargs)
+
+ related_data = getattr(self.context.get('request',{}), 'query_params', {}).get('related_data',None)
+ if related_data is None:
+ self.fields.pop("related_data")
+
def get_prediction(self, obj):
if obj.prediction_numeric is not None:
return obj.prediction_numeric
@@ -177,5 +170,5 @@ class Meta:
model = ScoreInstance
exclude = (
'prediction_dict',
- 'prediction_numeric',
+ 'prediction_numeric'
)
diff --git a/scidash/sciunittests/tests/test_data/score_object.json b/scidash/sciunittests/tests/test_data/score_object.json
index 70ad446b..04a51158 100644
--- a/scidash/sciunittests/tests/test_data/score_object.json
+++ b/scidash/sciunittests/tests/test_data/score_object.json
@@ -1,11 +1,35 @@
{
- "score_class": {"class_name": "ZScore", "url": "http://test-url.for/not-spaming-data/in-database"},
+ "score_class": {
+ "class_name": "ZScore",
+ "url": "http://test-url.for/not-spaming-data/in-database"
+ },
"hash_id": "111",
"model_instance": {
"model_class": {
"class_name": "ReducedModel",
"url": "http://test-url.for/not-spaming-data/in-database",
+ "import_path": "neuronunit.models.reduced.ReducedModel",
+ "memo": null,
+ "tooltip": "",
+ "extra_capabilities": [
+ 24
+ ],
"capabilities": [
+ {
+ "class_name": "Runnable"
+ },
+ {
+ "class_name": "ReceivesSquareCurrent"
+ },
+ {
+ "class_name": "ProducesActionPotentials"
+ },
+ {
+ "class_name": "ProducesSpikes"
+ },
+ {
+ "class_name": "ProducesMembranePotential"
+ },
{
"class_name": "CanBeReduced"
}
@@ -14,6 +38,8 @@
"attributes": {},
"hash_id": "111",
"name": "Izhikevich",
+ "status": "l",
+ "tags": [],
"run_params": {},
"url": "https://github.com/scidash/neuronunit/blob/master/neuronunit/models/NeuroML2/LEMS_2007One.xml",
"backend": "JNeuroML"
@@ -23,21 +49,40 @@
"sort_key": 0.738882680363527,
"score_type": "ZType",
"test_instance": {
+ "name": "Default Name",
"description": null,
"hash_id": "111",
- "test_suites": [{
- "hash": "testhash",
- "name": "ReducedSuite"
- }],
+ "status": "l",
+ "tags": [],
+ "params": null,
+ "verbose": 1,
+ "test_suites": [
+ {
+ "hash": "testhash",
+ "name": "ReducedSuite"
+ }
+ ],
"test_class": {
- "class_name": "MyTest",
- "url": "http://test-url.for/not-spaming-data/in-database"
+ "class_name": "InputResistanceTest",
+ "url": "http://github.com/scidash/neuronunit.git",
+ "import_path": "neuronunit.tests.passive.InputResistanceTest",
+ "observation_schema": [
+ ["Mean, Standard Deviation, N", {"n": {"min": 1, "type": "integer"}, "std": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}],
+ ["Mean, Standard Error, N", {"n": {"min": 1, "type": "integer", "required": true}, "sem": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}]
+ ]
},
"observation": {
- "mean":"8",
- "std": "3",
- "url": ""
- },
- "verbose": 1
+ "mean": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "125.0",
+ "units": "megaohm"}},
+ "n": 10,
+ "std": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "40.0",
+ "units": "megaohm"}}
+ }
}
}
diff --git a/scidash/sciunittests/tests/test_data/score_objects_list.json b/scidash/sciunittests/tests/test_data/score_objects_list.json
index 9ea540a3..dbaa1d4a 100644
--- a/scidash/sciunittests/tests/test_data/score_objects_list.json
+++ b/scidash/sciunittests/tests/test_data/score_objects_list.json
@@ -1,14 +1,38 @@
[
{
- "score_class": {"class_name": "ZScore", "url": "http://test-url.for/not-spaming-data/in-database"},
+ "score_class": {
+ "class_name": "ZScore",
+ "url": "http://test-url.for/not-spaming-data/in-database"
+ },
"hash_id": "111",
"model_instance": {
"model_class": {
"class_name": "ReducedModel_1",
"url": "http://test-url.for/not-spaming-data/in-database-one",
+ "import_path": "neuronunit.models.reduced.ReducedModel",
+ "memo": null,
+ "tooltip": "",
+ "extra_capabilities": [
+ 14
+ ],
"capabilities": [
{
- "class_name": "CanBeReduced_1"
+ "class_name": "Runnable"
+ },
+ {
+ "class_name": "ReceivesSquareCurrent"
+ },
+ {
+ "class_name": "ProducesActionPotentials"
+ },
+ {
+ "class_name": "ProducesSpikes"
+ },
+ {
+ "class_name": "ProducesMembranePotential"
+ },
+ {
+ "class_name": "CanBeReduced"
}
]
},
@@ -16,6 +40,8 @@
"attributes": {},
"name": "Izhikevich",
"run_params": {},
+ "status": "l",
+ "tags": [],
"url": "https://github.com/scidash/neuronunit/blob/master/neuronunit/models/NeuroML2/LEMS_2007One.xml",
"backend": "JNeuroML"
},
@@ -25,33 +51,67 @@
"score_type": "ZType_1",
"test_instance": {
"description": null,
- "test_suites": [{
- "hash": "testhash",
- "name": "ReducedSuite_1"
- }],
+ "test_suites": [
+ {
+ "hash": "testhash",
+ "name": "ReducedSuite_1"
+ }
+ ],
"test_class": {
- "class_name": "MyTest_1",
- "url": "http://test-url.for/not-spaming-data/in-database"
+ "class_name": "InputResistanceTest ZS3",
+ "url": "http://github.com/scidash/neuronunit.git",
+ "import_path": "neuronunit.tests.passive.InputResistanceTest",
+ "observation_schema": [
+ ["Mean, Standard Deviation, N", {"n": {"min": 1, "type": "integer"}, "std": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}],
+ ["Mean, Standard Error, N", {"n": {"min": 1, "type": "integer", "required": true}, "sem": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}]
+ ]
},
"observation": {
- "mean":"8",
- "std": "3",
- "url": ""
+ "mean": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "125.0",
+ "units": "megaohm"}},
+ "n": 10,
+ "std": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "40.0",
+ "units": "megaohm"}}
},
"hash_id": "111",
"verbose": 1
}
},
{
- "score_class": {"class_name": "ZScore", "url": "http://test-url.for/not-spaming-data/in-database"},
+ "score_class": {
+ "class_name": "ZScore",
+ "url": "http://test-url.for/not-spaming-data/in-database"
+ },
"hash_id": "222",
"model_instance": {
"model_class": {
"class_name": "ReducedModel_2",
"url": "http://test-url.for/not-spaming-data/in-database-two",
+ "import_path": "neuronunit.models.static.StaticModel",
+ "memo": null,
+ "tooltip": "",
+ "extra_capabilities": [],
"capabilities": [
{
- "class_name": "CanBeReduced_2"
+ "class_name": "ReceivesSquareCurrent"
+ },
+ {
+ "class_name": "ProducesActionPotentials"
+ },
+ {
+ "class_name": "ProducesSpikes"
+ },
+ {
+ "class_name": "ProducesMembranePotential"
+ },
+ {
+ "class_name": "CanBeReduced"
}
]
},
@@ -59,6 +119,8 @@
"attributes": {},
"name": "Izhikevich",
"run_params": {},
+ "status": "l",
+ "tags": [],
"url": "https://github.com/scidash/neuronunit/blob/master/neuronunit/models/NeuroML2/LEMS_2007One.xml",
"backend": "JNeuroML"
},
@@ -69,30 +131,64 @@
"test_instance": {
"description": null,
"hash_id": "111",
- "test_suites": [{
- "hash": "testhash",
- "name": "ReducedSuite_2"
- }],
+ "test_suites": [
+ {
+ "hash": "testhash",
+ "name": "ReducedSuite_2"
+ }
+ ],
"test_class": {
- "class_name": "MyTest_2",
- "url": "http://test-url.for/not-spaming-data/in-database"
+ "class_name": "InputResistanceTest ZS1",
+ "url": "http://github.com/scidash/neuronunit.git",
+ "import_path": "neuronunit.tests.passive.InputResistanceTest",
+ "observation_schema": [
+ ["Mean, Standard Deviation, N", {"n": {"min": 1, "type": "integer"}, "std": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}],
+ ["Mean, Standard Error, N", {"n": {"min": 1, "type": "integer", "required": true}, "sem": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}]
+ ]
},
"observation": {
- "mean":"8",
- "std": "3",
- "url": ""
+ "mean": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "125.0",
+ "units": "megaohm"}},
+ "n": 10,
+ "std": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "40.0",
+ "units": "megaohm"}}
},
"verbose": 1
}
},
{
- "score_class": {"class_name": "ZScore", "url": "http://test-url.for/not-spaming-data/in-database"},
+ "score_class": {
+ "class_name": "ZScore",
+ "url": "http://test-url.for/not-spaming-data/in-database"
+ },
"hash_id": "333",
"model_instance": {
"model_class": {
"class_name": "ReducedModel",
"url": "http://test-url.for/not-spaming-data/in-database-three",
+ "import_path": "neuronunit.models.static.StaticModel",
+ "memo": null,
+ "tooltip": "",
+ "extra_capabilities": [],
"capabilities": [
+ {
+ "class_name": "ReceivesSquareCurrent"
+ },
+ {
+ "class_name": "ProducesActionPotentials"
+ },
+ {
+ "class_name": "ProducesSpikes"
+ },
+ {
+ "class_name": "ProducesMembranePotential"
+ },
{
"class_name": "CanBeReduced"
}
@@ -102,6 +198,8 @@
"hash_id": "311",
"name": "Izhikevich",
"run_params": {},
+ "status": "l",
+ "tags": [],
"url": "https://github.com/scidash/neuronunit/blob/master/neuronunit/models/NeuroML2/LEMS_2007One.xml",
"backend": "JNeuroML"
},
@@ -112,18 +210,33 @@
"test_instance": {
"description": null,
"hash_id": "111",
- "test_suites": [{
- "hash": "testhash",
- "name": "ReducedSuite"
- }],
+ "test_suites": [
+ {
+ "hash": "testhash",
+ "name": "ReducedSuite"
+ }
+ ],
"test_class": {
- "class_name": "MyTest",
- "url": "http://test-url.for/not-spaming-data/in-database"
+ "class_name": "InputResistanceTest ZS2",
+ "url": "http://github.com/scidash/neuronunit.git",
+ "import_path": "neuronunit.tests.passive.InputResistanceTest",
+ "observation_schema": [
+ ["Mean, Standard Deviation, N", {"n": {"min": 1, "type": "integer"}, "std": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}],
+ ["Mean, Standard Error, N", {"n": {"min": 1, "type": "integer", "required": true}, "sem": {"min": 0, "units": true, "required": true}, "mean": {"units": true, "required": true}}]
+ ]
},
"observation": {
- "mean":"8",
- "std": "3",
- "url": ""
+ "mean": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "125.0",
+ "units": "megaohm"}},
+ "n": 10,
+ "std": {
+ "py/object": "quantities.quantity.Quantity",
+ "py/state": {
+ "base": "40.0",
+ "units": "megaohm"}}
},
"verbose": 1
}
diff --git a/scidash/sciunittests/tests/test_sciunit_tests.py b/scidash/sciunittests/tests/test_sciunit_tests.py
index 5a404c13..c5ba6095 100644
--- a/scidash/sciunittests/tests/test_sciunit_tests.py
+++ b/scidash/sciunittests/tests/test_sciunit_tests.py
@@ -6,18 +6,17 @@
from django.urls import reverse
from scidash.general.models import ScidashUser
-from scidash.sciunittests.models import ScoreInstance
+from scidash.sciunittests.models import ScoreInstance, TestClass
from scidash.sciunittests.serializers import (
ScoreClassSerializer, ScoreInstanceSerializer, TestClassSerializer
)
SAMPLE_OBJECT = os.path.join(
- os.path.dirname(os.path.dirname(__file__)), 'test_data/score_object.json'
+ os.path.dirname(__file__), 'test_data/score_object.json'
)
SAMPLE_OBJECT_LIST = os.path.join(
- os.path.dirname(os.path.dirname(__file__)),
- 'test_data/score_objects_list.json'
+ os.path.dirname(__file__), 'test_data/score_objects_list.json'
)
@@ -28,6 +27,12 @@ def setUpClass(cls):
factory = RequestFactory()
request = factory.get('/data/upload/score_object.json')
+
+ cls.test_class = TestClass.objects.create(
+ class_name="InputResistanceTest",
+ import_path="neuronunit.tests.passive.InputResistanceTest",
+ url="http://github.com/scidash/neuronunit.git",
+ )
cls.user = ScidashUser.objects.create_user(
'admin', 'a@a.cc', 'montecarlo'
)
@@ -78,11 +83,19 @@ def test_if_scores_endpoint_works_correctly(self):
self.scrub(parsed_response, 'build_info')
self.scrub(parsed_response, 'hostname')
self.scrub(parsed_response, 'owner')
+ self.scrub(parsed_response, 'units_name')
parsed_keys = parsed_response.keys()
+ self.maxDiff=None
for key in data.keys():
self.assertTrue(key in parsed_keys)
- self.assertEqual(data.get(key), parsed_response.get(key))
+ if key == "test_instance":
+ # we skip testing of the test instance, it fails because
+ # of the extra fields and enhancing the class_name with
+ # the import path
+ pass
+ else:
+ self.assertEqual(data.get(key), parsed_response.get(key))
def test_if_test_instance_endpoint_works_correctly(self):
client = Client()
@@ -100,14 +113,21 @@ def test_if_test_instance_endpoint_works_correctly(self):
self.scrub(parsed_response, 'id')
self.scrub(parsed_response, 'timestamp')
self.scrub(parsed_response, 'owner')
+ self.scrub(parsed_response, 'units_name')
parsed_keys = parsed_response.keys()
test_instance_data = data.get('test_instance')
for key in data.get('test_instance').keys():
self.assertTrue(key in parsed_keys)
- self.assertEqual(
- test_instance_data.get(key), parsed_response.get(key)
- )
+ if key == "test_class":
+ # we skip testing of the test instance, it fails because
+ # of the extra fields and enhancing the class_name with
+ # the import path
+ pass
+ else:
+ self.assertEqual(
+ test_instance_data.get(key), parsed_response.get(key)
+ )
def test_if_test_class_endpoint_works_correctly(self):
client = Client()
@@ -129,9 +149,14 @@ def test_if_test_class_endpoint_works_correctly(self):
for key in data.get('test_instance').get('test_class').keys():
self.assertTrue(key in parsed_keys)
- self.assertEqual(
- test_classes_data.get(key), parsed_response.get(key)
- )
+ if key == "class_name":
+ self.assertEqual(
+ test_classes_data.get(key), parsed_response.get(key).split(" ")[0]
+ )
+ else:
+ self.assertEqual(
+ test_classes_data.get(key), parsed_response.get(key)
+ )
def test_if_test_suite_endpoint_works_correctly(self):
client = Client()
@@ -166,6 +191,12 @@ def setUpClass(cls):
factory = RequestFactory()
request = factory.get('/data/upload/score_object_list.json')
+
+ cls.test_class = TestClass.objects.create(
+ class_name="InputResistanceTest",
+ import_path="neuronunit.tests.passive.InputResistanceTest",
+ url="http://github.com/scidash/neuronunit.git",
+ )
cls.user = ScidashUser.objects.create_user(
'admin', 'a@a.cc', 'montecarlo'
)
@@ -234,7 +265,7 @@ def test_scores_endpoint_filters_get_by_class_name(self):
parsed_response = response.json()
first_element = parsed_response[2]
model_class_name = first_element.get('model_instance') \
- .get('model_class').get('class_name')
+ .get('model_class').get('class_name')
filtered_url = '{}?model={}'.format(
reverse('score-list'), model_class_name
@@ -318,6 +349,12 @@ def setUpClass(cls):
factory = RequestFactory()
request = factory.get('/data/upload/score_object_list.json')
+
+ cls.test_class = TestClass.objects.create(
+ class_name="InputResistanceTest",
+ import_path="neuronunit.tests.passive.InputResistanceTest",
+ url="http://github.com/scidash/neuronunit.git",
+ )
cls.user = ScidashUser.objects.create_user(
'admin', 'a@a.cc', 'montecarlo'
)
diff --git a/scidash/sciunittests/validators.py b/scidash/sciunittests/validators.py
new file mode 100644
index 00000000..7330751d
--- /dev/null
+++ b/scidash/sciunittests/validators.py
@@ -0,0 +1,167 @@
+import json
+import importlib
+from scidash.sciunittests.models import TestClass
+import numpy as np
+import quantities as pq
+import sciunit
+
+import jsonpickle
+from copy import deepcopy
+
+# Register the sciunit quantity handlers
+jsonpickle.handlers.register(pq.Quantity, handler=sciunit.base.QuantitiesHandler)
+jsonpickle.handlers.register(pq.UnitQuantity, handler=sciunit.base.UnitQuantitiesHandler)
+
+
+def build_destructured_unit(unit_dict):
+ unit = pq.UnitQuantity(
+ unit_dict.get('name'),
+ import_class(unit_dict.get('base').get('quantity')) *
+ unit_dict.get('base').get('coefficient'), unit_dict.get('symbol')
+ )
+
+ return unit
+
+
+def import_class(import_path: str) -> object:
+ """ Import class from import_path
+
+ :type str:
+ :param import_path: path to module similar to path.to.module.ClassName
+
+ :returns: imported class
+ """
+
+ splitted = import_path.split('.')
+
+ class_name = splitted[-1:][0]
+ module_path = ".".join(splitted[:-1])
+
+ imported_module = importlib.import_module(module_path)
+ klass = getattr(imported_module, class_name)
+
+ return klass
+
+
+class TestInstanceValidator:
+
+ @classmethod
+ def validate(cls, data, fallback_observation_schema=None):
+ try:
+ # old style
+ sciunit.settings['PREVALIDATE'] = True
+ except:
+ # new style
+ try:
+ sciunit.config_set('PREVALIDATE', True)
+ except:
+ sciunit.config.set('PREVALIDATE', True)
+
+ class_data = data.get('test_class')
+ if not class_data:
+ class_data = data.get('test').get('_class')
+
+ if not class_data.get('import_path', False):
+ return data
+
+ test_class = import_class(class_data.get('import_path'))
+
+ observation_schema = test_class.observation_schema
+ if not observation_schema:
+ observation_schema = class_data.get("observation_schema")
+ if not observation_schema:
+ observation_schema = fallback_observation_schema
+
+ observation = deepcopy(data.get('observation')) # json of observation
+
+ # Thicken the JSON with metadata required for deserialization
+ for key, value in observation.items():
+ if isinstance(value, dict) and 'units' in value:
+ observation[key] = {'py/object': 'quantities.quantity.Quantity',
+ 'py/state': value}
+
+ # Check observations for values without units
+ has_units = False
+ for key, value in observation.items():
+ if isinstance(value, dict):
+ has_units = True
+
+ observation = json.dumps(observation) # As string for decoding
+ observation = jsonpickle.decode(observation) # decode
+
+ if has_units:
+ # some/all observation items have units, use the observation
+ obs_with_units = observation
+ else:
+ # none of the observation items have units, let try to get them from the (class)template
+ obs_with_units = cls.add_units_to_observation(class_data, test_class, observation, fallback_observation_schema)
+
+ test_class(obs_with_units)
+
+ return data
+
+ @classmethod
+ def add_units_to_observation(cls, class_data, test_class, observation, fallback_observation_schema=None):
+ without_units = []
+ if class_data.get("units"):
+ try:
+ destructured = json.loads(class_data.get('units'))
+ except json.JSONDecodeError:
+ quantity = import_class(class_data.get('units'))
+ else:
+ if destructured.get('name', False):
+ quantity = build_destructured_unit(destructured)
+ else:
+ quantity = destructured
+ else:
+ quantity = 1
+
+
+ def filter_units(schema):
+ result = []
+ for key, rules in schema.items():
+ if not rules.get('units', False):
+ result.append(key)
+
+ return result
+
+ observation_schema = test_class.observation_schema
+ if not observation_schema:
+ observation_schema = class_data.get("observation_schema")
+ if not observation_schema:
+ observation_schema = fallback_observation_schema
+
+ if isinstance(observation_schema, list):
+ for schema in observation_schema:
+ if isinstance(schema, tuple) or len(schema)>1:
+ without_units += filter_units(schema[1])
+ else:
+ without_units += filter_units(schema)
+ elif observation_schema:
+ without_units = filter_units(observation_schema)
+
+ def process_obs(obs):
+ try:
+ obs = int(obs)
+ except ValueError:
+ obs = np.array(json.loads(obs))
+
+ return obs
+
+ if not isinstance(quantity, dict):
+ obs_with_units = {
+ x: (
+ process_obs(y) * quantity
+ if x not in without_units else process_obs(y)
+ )
+ for x, y in observation.items()
+ }
+ else:
+ obs_with_units = {
+ x: (
+ process_obs(y) * import_class(quantity[x])
+ if x not in without_units else process_obs(y)
+ )
+ for x, y in observation.items()
+ }
+ return obs_with_units
diff --git a/scidash/sciunittests/views.py b/scidash/sciunittests/views.py
index 99e91b50..e3109d9d 100644
--- a/scidash/sciunittests/views.py
+++ b/scidash/sciunittests/views.py
@@ -9,7 +9,21 @@
class DateRangeView(APIView):
def get(self, request, *args, **kwargs):
- three_month_period = datetime.timedelta(3 * 365 / 12)
+ """Returns the initial search period (acceptable_period).
+ for the first N (settings.ACCEPTABLE_SCORE_INSTANCES_AMOUNT) scores.
+
+ Parameters
+ ----------
+ -
+
+ Returns
+ -------
+ JSON object
+ {
+ "current_date": "",
+ "acceptable_period": ""
+ }
+ """
current_date = datetime.date.today() + datetime.timedelta(days=1)
current_date_iso = datetime.datetime(
year=current_date.year,
@@ -17,20 +31,20 @@ def get(self, request, *args, **kwargs):
day=current_date.day
)
- acceptable_period = None
-
- for quarter in range(1, s.SCIDASH_INITIAL_SEARCH_QUARTERS + 1):
- period = current_date_iso - quarter*three_month_period
- count = ScoreInstance.objects.filter(
- timestamp__gte=period, timestamp__lt=current_date_iso
- ).count()
-
- if count > s.ACCEPTABLE_SCORE_INSTANCES_AMOUNT:
- acceptable_period = period
- break
-
- if acceptable_period is None:
- acceptable_period = period
+ scores = ScoreInstance.objects.filter(
+ timestamp__lt=current_date_iso).order_by('-timestamp') \
+ [:s.ACCEPTABLE_SCORE_INSTANCES_AMOUNT]
+ if scores:
+ # found scores, acceptable period is scores last.timestamp
+ # because sorting is DESC timestamp
+ acceptable_period = scores.reverse()[0].timestamp
+ else:
+ # acceptable period defaults to current date - 1 year
+ acceptable_period = datetime.datetime(
+ year=current_date.year-1,
+ month=current_date.month,
+ day=current_date.day
+ )
return Response(
{
diff --git a/service/deployment/docker-compose.yml b/service/deployment/docker-compose.yml
index 1d0810c8..dea00f4f 100644
--- a/service/deployment/docker-compose.yml
+++ b/service/deployment/docker-compose.yml
@@ -1,32 +1,80 @@
-version: '2'
+version: '3'
services:
scidash-redis:
image: redis
+ healthcheck:
+ test: ["CMD", "redis-cli","ping"]
+ ports:
+ - 6379:6379
expose:
- - 6379
+ - 6379
scidash-postgres:
image: metacell/scidash_db:latest
+ container_name: scidash_db
+ build:
+ dockerfile: service/docker/Dockerfile-postgres
+ context: ../..
+ environment:
+ - SCIDASH_BRANCH=development
+ - POSTGRES_PASSWORD=password
+ ports:
+ - 5432:5432
+ healthcheck:
+ test: ["CMD-SHELL", "pg_isready -U postgres"]
+ interval: 15s
+ timeout: 10s
+ retries: 5
expose:
- - 5432
+ - 5432
volumes:
- - ./database:/var/lib/postgresql/
+ - ./database:/var/lib/postgresql/
scidash-virgo:
- image: metacell/scidash_virgo:v1.0.0
+ image: metacell/scidash_geppetto:latest
+ build:
+ dockerfile: service/docker/Dockerfile-virgo
+ context: ../..
+ container_name: scidash_geppetto
+ volumes:
+ - geppettoTmp-volume:/opt/virgo/geppettoTmp
+ ports:
+ - 8080:8080
expose:
- - 8080
+ - 8080
mem_reservation: 5120m
mem_limit: 10240m
privileged: true
shm_size: 512M
depends_on:
- - scidash-redis
- - scidash-postgres
+ - scidash-redis
+ - scidash-postgres
scidash:
image: metacell/scidash:latest
+ build:
+ dockerfile: service/docker/Dockerfile-scidash
+ context: ../..
+ environment:
+ - EMAIL_USERNAME=scidashasu@gmail.com
+ - EMAIL_PASSWORD=Zh0UzMZDI7ic
+ - ENVIRONMENT=production
+ - DOMAIN=dash.scidash.org
+ container_name: scidash
+ cap_add:
+ - SYS_ADMIN
+ volumes:
+ - geppettoTmp-volume:/opt/virgo/geppettoTmp
+ - ./secrets:/etc/secrets
ports:
- - 8000:8000
+ - 8000:8000
depends_on:
- - scidash-redis
- - scidash-postgres
- - scidash-virgo
+ scidash-redis:
+ condition: service_healthy
+ scidash-postgres:
+ condition: service_healthy
+
+volumes:
+ geppettoTmp-volume:
+ driver_opts:
+ o: uid=1000,gid=1000
+ device: tmpfs
+ type: tmpfs
diff --git a/service/deployment/secrets/SOCIAL_AUTH_GITHUB_KEY b/service/deployment/secrets/SOCIAL_AUTH_GITHUB_KEY
new file mode 100644
index 00000000..1aadabed
--- /dev/null
+++ b/service/deployment/secrets/SOCIAL_AUTH_GITHUB_KEY
@@ -0,0 +1 @@
+key
\ No newline at end of file
diff --git a/service/deployment/secrets/SOCIAL_AUTH_GITHUB_SECRET b/service/deployment/secrets/SOCIAL_AUTH_GITHUB_SECRET
new file mode 100644
index 00000000..536aca34
--- /dev/null
+++ b/service/deployment/secrets/SOCIAL_AUTH_GITHUB_SECRET
@@ -0,0 +1 @@
+secret
\ No newline at end of file
diff --git a/service/deployment/secrets/SOCIAL_AUTH_GOOGLE_OAUTH2_KEY b/service/deployment/secrets/SOCIAL_AUTH_GOOGLE_OAUTH2_KEY
new file mode 100644
index 00000000..1aadabed
--- /dev/null
+++ b/service/deployment/secrets/SOCIAL_AUTH_GOOGLE_OAUTH2_KEY
@@ -0,0 +1 @@
+key
\ No newline at end of file
diff --git a/service/deployment/secrets/SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET b/service/deployment/secrets/SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET
new file mode 100644
index 00000000..536aca34
--- /dev/null
+++ b/service/deployment/secrets/SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET
@@ -0,0 +1 @@
+secret
\ No newline at end of file
diff --git a/service/deployment/secrets/SOCIAL_AUTH_TWITTER_KEY b/service/deployment/secrets/SOCIAL_AUTH_TWITTER_KEY
new file mode 100644
index 00000000..1aadabed
--- /dev/null
+++ b/service/deployment/secrets/SOCIAL_AUTH_TWITTER_KEY
@@ -0,0 +1 @@
+key
\ No newline at end of file
diff --git a/service/deployment/secrets/SOCIAL_AUTH_TWITTER_SECRET b/service/deployment/secrets/SOCIAL_AUTH_TWITTER_SECRET
new file mode 100644
index 00000000..536aca34
--- /dev/null
+++ b/service/deployment/secrets/SOCIAL_AUTH_TWITTER_SECRET
@@ -0,0 +1 @@
+secret
\ No newline at end of file
diff --git a/service/docker/Dockerfile-postgres b/service/docker/Dockerfile-postgres
index aca1f1c5..9425f4e9 100644
--- a/service/docker/Dockerfile-postgres
+++ b/service/docker/Dockerfile-postgres
@@ -1,7 +1,12 @@
-FROM postgres:9.4
+FROM postgres:12
ARG ROOT=/
ARG APP_DIR=/app
+ARG targetBranch=development
+ARG originBranch=development
+ARG defaultBranch=development
+ARG SCIDASH_BRANCH
+ARG POSTGRES_PASSWORD
WORKDIR $ROOT
@@ -13,7 +18,10 @@ RUN apt-get update && \
RUN mkdir $APP_DIR
WORKDIR $APP_DIR
-RUN git clone -b deployment https://github.com/MetaCell/scidash
+RUN git clone https://github.com/ddelpiano/travis_utils
+RUN cp travis_utils/copy.sh $APP_DIR
+RUN chmod 777 /tmp
+RUN ./copy.sh https://github.com/MetaCell/scidash "${targetBranch}" "${originBranch}" "${defaultBranch}"
RUN cp scidash/service/database/backup_database.sh /backup_database.sh
RUN chmod 0644 /backup_database.sh
-RUN cp scidash/service/scripts/db-create-psql.sh /docker-entrypoint-initdb.d/db-create-psql.sh
\ No newline at end of file
+RUN cp scidash/service/scripts/db-create-psql.sh /docker-entrypoint-initdb.d/db-create-psql.sh
diff --git a/service/docker/Dockerfile-scidash b/service/docker/Dockerfile-scidash
index 93aa6cbf..10603b09 100644
--- a/service/docker/Dockerfile-scidash
+++ b/service/docker/Dockerfile-scidash
@@ -3,13 +3,18 @@ FROM python:3.6
USER root
# BUILD VARIABLES
+ARG EMAIL_USERNAME
+ARG EMAIL_PASSWORD
ARG ROOT=/
ARG APP_DIR=/app
ARG NRN_SYMLINK=/Applications/NEURON-7.6/nrn/
ARG DOTENV_FILE=env-docker
ARG STATIC_DIR=$APP_DIR/scidash/static
ARG GEPPETTO_DIR=$STATIC_DIR/org.geppetto.frontend/src/main/webapp
-ARG SCIDASH_BRANCH=feature/400
+ARG targetBranch=geppetto-scidash
+ARG originBranch=geppetto-scidash
+ARG defaultBranch=development
+
ENV SERVER_HOME $APP_DIR/virgo-tomcat-server
RUN useradd -ms /bin/bash developer
@@ -17,37 +22,55 @@ ENV HOME /home/developer
WORKDIR $ROOT
+RUN echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf
+RUN mkdir -p /etc/secrets
+
# INSTALLING REQUIREMENTS
-RUN apt-get install -y curl wget
-RUN curl -sL https://deb.nodesource.com/setup_9.x | bash
-RUN apt-get update && apt-get -y install nodejs
-RUN curl https://www.npmjs.com/install.sh | sh
+RUN apt-get update
+RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
+ libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 \
+ libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 \
+ libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 \
+ libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 \
+ lsb-release xdg-utils wget curl llvm python3-tk llvm
+RUN curl -sL https://deb.nodesource.com/setup_10.x | bash
+RUN apt-get update && apt-get -y install nodejs npm
+# RUN npm --version
+# RUN curl https://www.npmjs.com/install.sh | npm_install=6.4.0 sh
+RUN npm install npm@7.20.3 -g
+RUN pip install virtualenv
+RUN mkdir $APP_DIR
+RUN chown developer $APP_DIR
+RUN mkdir -p /home/developer/.config
+RUN chown -R developer /home/developer/.config
+RUN git clone https://github.com/ddelpiano/travis_utils
+RUN cp travis_utils/copy.sh $APP_DIR
-#COPYING PROJECT
-WORKDIR $APP_DIR
-RUN git clone -b $SCIDASH_BRANCH https://github.com/MetaCell/scidash
+# COPYING PROJECT
+RUN mkdir -p $APP_DIR/scidash
+RUN chown developer:developer $APP_DIR/scidash
+COPY --chown=developer . $APP_DIR/scidash
+USER developer
+# set git email and name for use with git merge
+RUN git config --global user.email "scidash@metacell.us"
+RUN git config --global user.name "SciDash"
WORKDIR $APP_DIR/scidash
-RUN pip install virtualenv
-RUN virtualenv venv-py -p python3.6
WORKDIR $APP_DIR/scidash
-RUN make install-backend-with-env
-RUN make install-frontend
+RUN make ARGS="-b $targetBranch" install-backend-with-env
+RUN make ARGS="-b $targetBranch" install-frontend
+
+RUN sed -i "s/email_username/$EMAIL_USERNAME/g" ./scidash/main/settings.py
+RUN sed -i "s/email_password/$EMAIL_PASSWORD/g" ./scidash/main/settings.py
WORKDIR $GEPPETTO_DIR
-RUN ls -la
RUN npm run build-dev-noTest
WORKDIR $APP_DIR/scidash
-RUN rm -rf .git &&\
- rm -rf static/org.geppetto.frontend/.git &&\
- rm -rf static/org.geppetto.frontend/extension/geppetto-scidash/.git &&\
- rm -rf sciunit/.git &&\
- rm -rf neuronunit/.git
-RUN chown -R developer ./
RUN cp ./service/dotenv/scidash_env .env
-USER developer
+RUN bash -c "source venv/bin/activate && pip list "
+
CMD ./service/scripts/run.sh
diff --git a/service/docker/Dockerfile-virgo b/service/docker/Dockerfile-virgo
index a8b0f34d..03274f7e 100644
--- a/service/docker/Dockerfile-virgo
+++ b/service/docker/Dockerfile-virgo
@@ -4,44 +4,52 @@ ARG APP_DIR=/app
ARG SOURCES_DIR=$APP_DIR/sources
ARG VIRGO_DIR=/opt/virgo
ARG SCIDASH_REPO_FOLDER=/git
-ARG GEPPETTO_REPO=http://github.com/openworm/org.geppetto
-ARG ALPHA_TAG=v0.4.2-alpha
-ARG SIMFLAG_TAG=v0.4.2-simflag
-ARG SCIDASH_BRANCH=development
+ARG GEPPETTO_REPO=http://github.com/openworm/org.geppetto.git
+ARG SCIDASH_BRANCH=geppetto-scidash
+ARG targetBranch=scidash-backend
+ARG originBranch=scidash-backend
+ARG defaultBranch=scidash-backend
# -== Install OpenJDK and certificates ==-
WORKDIR /
RUN useradd -ms /bin/bash developer
ENV HOME /home/developer
+
+WORKDIR /home/developer
+RUN wget -q https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u282-b08/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz && \
+ tar xzf OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz
+# -== Setup JAVA_HOME ==-
+RUN export JAVA_HOME=/home/developer/jdk8u282-b08
+RUN export PATH=$JAVA_HOME/bin:$PATH
+ENV JAVA_HOME=/home/developer/jdk8u282-b08
+
+WORKDIR /
+
RUN printf "\ndeb http://deb.debian.org/debian/ sid main\n" >> /etc/apt/sources.list
RUN apt-get update && \
- apt-get install -y openjdk-8-jdk && \
apt-get install -y ant && \
apt-get clean;
RUN apt-get update && \
- apt-get install ca-certificates-java && \
apt-get clean && \
update-ca-certificates -f;
-# -== Setup JAVA_HOME ==-
-ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
-RUN export JAVA_HOME
-
# -== Create various folder for this deployment ==-
RUN mkdir $APP_DIR
RUN mkdir $SCIDASH_REPO_FOLDER
WORKDIR $APP_DIR
RUN mkdir $SOURCES_DIR
WORKDIR $SCIDASH_REPO_FOLDER
+RUN git clone https://github.com/ddelpiano/travis_utils
+RUN cp travis_utils/copy.sh $APP_DIR
# -== Download SCIDASH repo ==-
-RUN git clone -b $SCIDASH_BRANCH https://github.com/MetaCell/scidash
+RUN $APP_DIR/copy.sh http://github.com/Metacell/scidash.git "development" "development" "development"
# -== INSTALL MAVEN ==-
WORKDIR /tmp
-RUN wget http://archive.apache.org/dist/maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz
+RUN wget -q http://archive.apache.org/dist/maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz
RUN tar xzf apache-maven-3.5.2-bin.tar.gz -C /opt/
RUN ln -s /opt/apache-maven-3.5.2 /opt/maven
RUN ln -s /opt/maven/bin/mvn /usr/local/bin
@@ -52,8 +60,11 @@ RUN mvn --version
# -== INSTALL NEURON ==-
RUN mkdir /nrn
WORKDIR /nrn
-RUN apt-get install -y libreadline5 libreadline-gplv2-dev lsof
-RUN wget -O nrn-7.6.tar.gz https://github.com/ddelpiano/neuron/raw/master/nrn-7.6.tar.gz
+RUN apt -y update
+RUN apt-get install -y readline-common # libreadline5
+RUN apt-get install -y libreadline-dev # libreadline-gplv2-dev
+RUN apt-get install -y lsof
+RUN wget -q -O nrn-7.6.tar.gz https://github.com/ddelpiano/neuron/raw/master/nrn-7.6.tar.gz
RUN tar xzvf nrn-7.6.tar.gz
WORKDIR nrn-7.6
RUN ./configure \
@@ -92,50 +103,61 @@ RUN cat /git/scidash/service/geppetto/tomcat-server.xml | sed 's/127.0.0.1/0.0.0
RUN chmod u+x $VIRGO_DIR/bin/*.sh
ENV SERVER_HOME $VIRGO_DIR
RUN chmod 777 -R $VIRGO_DIR
-RUN mkdir rm $VIRGO_DIR/repository/usr
+RUN mkdir $VIRGO_DIR/repository/usr
RUN mvn --version
# -== INSTALL GEPPETTO ==-
WORKDIR $SOURCES_DIR
-RUN git clone $GEPPETTO_REPO -b $ALPHA_TAG
-WORKDIR $SOURCES_DIR/org.geppetto/utilities/source_setup
-RUN rm $SOURCES_DIR/org.geppetto/utilities/source_setup/config.json
+RUN $APP_DIR/copy.sh http://github.com/openworm/org.geppetto.git "scidash-backend" "scidash-backend" "scidash-backend"
RUN cp /git/scidash/service/geppetto/config.json $SOURCES_DIR/org.geppetto/utilities/source_setup
RUN cp /git/scidash/service/geppetto/setup.py $SOURCES_DIR/org.geppetto/utilities/source_setup
RUN cp /git/scidash/service/geppetto/geppetto.plan $SOURCES_DIR/org.geppetto/
-RUN python2 setup.py
-
-WORKDIR $SOURCES_DIR/org.geppetto.core
-RUN git checkout $SIMFLAG_TAG
-RUN rm ./src/main/java/META-INF/spring/app-config.xml
-RUN cp /git/scidash/service/geppetto/core/app-config.xml ./src/main/java/META-INF/spring
-
-WORKDIR $SOURCES_DIR/org.geppetto.frontend
-RUN git checkout $SIMFLAG_TAG
-RUN rm ./src/main/webapp/WEB-INF/spring/app-config.xml
-RUN cp /git/scidash/service/geppetto/frontend/app-config.xml ./src/main/webapp/WEB-INF/spring/
-
-WORKDIR $SOURCES_DIR/org.geppetto.model
-RUN git checkout $ALPHA_TAG
-
-WORKDIR $SOURCES_DIR/org.geppetto.model.neuroml
-RUN git checkout $ALPHA_TAG
-
-WORKDIR $SOURCES_DIR/org.geppetto.simulation
-RUN git checkout $SIMFLAG_TAG
-
-WORKDIR $SOURCES_DIR/org.geppetto.simulator.external
-RUN git checkout $ALPHA_TAG
-
-WORKDIR $SOURCES_DIR/org.geppetto.simulator.scidash
-RUN git checkout development
-# RUN rm ./src/main/java/META-INF/spring/app-config.xml
-# RUN cp /git/scidash/service/geppetto/simulator.scidash/app-config.xml ./src/main/java/META-INF/spring
-WORKDIR $SOURCES_DIR/org.geppetto
-RUN mvn -Dhttps.protocols=TLSv1.2 -DcontextPath=org.geppetto.frontend -DuseSsl=false -DskipTests install
+RUN $APP_DIR/copy.sh https://github.com/openworm/org.geppetto.model.git scidash-backend scidash-backend scidash-backend &&\
+ cd org.geppetto.model &&\
+ /bin/echo -e "\e[96mMaven install org.geppetto.model\e[0m" &&\
+ mvn -Dhttps.protocols=TLSv1.2 -DskipTests --quiet install &&\
+ rm -rf src && cd ../
+
+RUN $APP_DIR/copy.sh https://github.com/openworm/org.geppetto.core.git scidash-backend scidash-backend scidash-backend &&\
+ cd org.geppetto.core && cp /git/scidash/service/geppetto/core/app-config.xml ./src/main/java/META-INF/spring &&\
+ /bin/echo -e "\e[96mMaven install org.geppetto.core\e[0m" &&\
+ mvn -Dhttps.protocols=TLSv1.2 -DskipTests --quiet install &&\
+ rm -rf src && cd ../
+
+RUN $APP_DIR/copy.sh https://github.com/openworm/org.geppetto.model.neuroml.git scidash-backend scidash-backend scidash-backend &&\
+ cd org.geppetto.model.neuroml &&\
+ /bin/echo -e "\e[96mMaven install org.geppetto.model.neuroml\e[0m" &&\
+ mvn -Dhttps.protocols=TLSv1.2 -DskipTests --quiet install &&\
+ rm -rf src && cd ../
+
+RUN $APP_DIR/copy.sh https://github.com/openworm/org.geppetto.simulation.git scidash-backend scidash-backend scidash-backend &&\
+ cd org.geppetto.simulation &&\
+ /bin/echo -e "\e[96mMaven install org.geppetto.simulation\e[0m" &&\
+ mvn -Dhttps.protocols=TLSv1.2 -DskipTests --quiet install &&\
+ rm -rf src && cd ../
+
+RUN $APP_DIR/copy.sh https://github.com/openworm/org.geppetto.simulator.external.git scidash-backend scidash-backend scidash-backend &&\
+ cd org.geppetto.simulator.external &&\
+ /bin/echo -e "\e[96mMaven install org.geppetto.simulator.external\e[0m" &&\
+ mvn -Dhttps.protocols=TLSv1.2 -DskipTests --quiet install &&\
+ rm -rf src && cd ../
+
+RUN $APP_DIR/copy.sh https://github.com/Metacell/org.geppetto.simulator.scidash.git scidash-backend scidash-backend scidash-backend &&\
+ cd org.geppetto.simulator.scidash && cp /git/scidash/service/geppetto/simulator.scidash/app-config.xml ./src/main/java/META-INF/spring &&\
+ /bin/echo -e "\e[96mMaven install org.geppetto.simulator.scidash\e[0m" &&\
+ mvn -Dhttps.protocols=TLSv1.2 -DskipTests --quiet install &&\
+ rm -rf src && cd ../
+
+RUN $APP_DIR/copy.sh https://github.com/openworm/org.geppetto.frontend.git scidash-backend scidash-backend scidash-backend &&\
+ cd org.geppetto.frontend && cp /git/scidash/service/geppetto/frontend/app-config.xml ./src/main/webapp/WEB-INF/spring/ &&\
+ cat ./src/main/webapp/package-lock.json | grep -i superagent &&\
+ /bin/echo -e "\e[96mMaven install org.geppetto.frontend\e[0m" &&\
+ mvn -Dhttps.protocols=TLSv1.2 -DskipTests --quiet install &&\
+ rm -rf src && cd ../
WORKDIR $SOURCES_DIR/org.geppetto/utilities/source_setup
+RUN apt install -y python2
RUN python2 update_server.py
WORKDIR $APP_DIR
diff --git a/service/docker/build_all.sh b/service/docker/build_all.sh
index 7f0e00f7..da7cf182 100755
--- a/service/docker/build_all.sh
+++ b/service/docker/build_all.sh
@@ -1,5 +1,7 @@
#!/bin/bash
./build_database.sh
+./build_virgo_base.sh
./build_virgo.sh
./build_scidash.sh
+
diff --git a/service/docker/build_codefresh.sh b/service/docker/build_codefresh.sh
index 13d3ebf3..4ad07c45 100755
--- a/service/docker/build_codefresh.sh
+++ b/service/docker/build_codefresh.sh
@@ -1,5 +1,5 @@
#!/bin/bash
-docker build -f Dockerfile-virgo -t metacell/scidash_virgo:latest
-docker build -f Dockerfile-postgres -t metacell/scidash_db:latest
-docker build -f Dockerfile-scidash -t metacell/scidash:latest
+docker build -f Dockerfile-virgo -t metacell/scidash_virgo:latest ../..
+docker build -f Dockerfile-postgres -t metacell/scidash_db:latest ../..
+docker build -f Dockerfile-scidash -t metacell/scidash:latest ../..
diff --git a/service/docker/build_database.sh b/service/docker/build_database.sh
index cd6191c4..4614b0bc 100755
--- a/service/docker/build_database.sh
+++ b/service/docker/build_database.sh
@@ -1,4 +1,11 @@
#!/bin/bash
+
+branchname=$1
+if [ -z "$branchname" ]
+then
+ branchname="development"
+fi
+
echo "We are going to build the scidash Database, be carefull since"
echo " this can overwrite the existing container if another one is already running"
while true; do
@@ -6,9 +13,9 @@ while true; do
case $yn in
[Yy]* ) read -p "Please type the tag you want to use for this build (default will use the latest and overwrite this). [latest/user_input] > " tag;
if [[ -z "$tag" ]]; then
- docker build --no-cache -f Dockerfile-postgres -t metacell/scidash_db:latest .
+ docker build --build-arg SCIDASH_BRANCH=${branchname} -f Dockerfile-postgres -t metacell/scidash_db:latest ../..
else
- docker build --no-chace -f Dockerfile-postgres -t metacell/scidash_db:$tag .
+ docker build --build-arg SCIDASH_BRANCH=${branchname} -f Dockerfile-postgres -t metacell/scidash_db:$tag ../..
fi
break;;
[Nn]* ) exit;;
diff --git a/service/docker/build_scidash.sh b/service/docker/build_scidash.sh
index d54db7dc..6b527d93 100755
--- a/service/docker/build_scidash.sh
+++ b/service/docker/build_scidash.sh
@@ -6,9 +6,9 @@ while true; do
case $yn in
[Yy]* ) read -p "Please type the tag you want to use for this build (default will use the latest and overwrite this). [latest/user_input] >" tag;
if [[ -z "$tag" ]]; then
- docker build --no-cache -f Dockerfile-scidash -t metacell/scidash:latest .
+ docker build --no-cache -f Dockerfile-scidash -t metacell/scidash:latest ../..
else
- docker build --no-cache -f Dockerfile-scidash -t metacell/scidash:$tag .
+ docker build --no-cache -f Dockerfile-scidash -t metacell/scidash:$tag ../..
fi
break;;
[Nn]* ) exit;;
diff --git a/service/docker/build_virgo.sh b/service/docker/build_virgo.sh
index 9d282afb..9af558fb 100755
--- a/service/docker/build_virgo.sh
+++ b/service/docker/build_virgo.sh
@@ -6,9 +6,9 @@ while true; do
case $yn in
[Yy]* ) read -p "Please type the tag you want to use for this build (default will use the latest and overwrite this). [default/user_input] > " tag;
if [[ -z "$tag" ]]; then
- docker build --no-cache -f Dockerfile-virgo -t metacell/scidash_virgo:latest .
+ docker build --no-cache -f Dockerfile-virgo -t metacell/scidash_virgo:latest ../..
else
- docker build --no-cache -f Dockerfile-virgo -t metacell/scidash_virgo:$tag .
+ docker build --no-cache -f Dockerfile-virgo -t metacell/scidash_virgo:$tag ../..
fi
break;;
[Nn]* ) exit;;
diff --git a/service/dotenv/dev_env b/service/dotenv/dev_env
new file mode 100644
index 00000000..8368a47a
--- /dev/null
+++ b/service/dotenv/dev_env
@@ -0,0 +1,29 @@
+DEVELOPMENT=1
+PRODUCTION=0
+HTTP=0
+
+DB_HOST='scidash-postgres'
+DB_PORT=5432
+DB_NAME='scidash'
+DB_USER=scidash_admin
+DB_PASSWORD=scidash_local_password
+
+REDIS_URL=redis://scidash-redis:6379
+
+STATIC_URL='/static/'
+STATIC_DIR='static'
+
+# for localhost
+# GEPPETTO_HOST=localhost
+# for docker
+GEPPETTO_HOST=scidash-virgo
+GEPPETTO_PORT=8080
+GEPPETTO_SERVLET_URL=ws://${GEPPETTO_HOST}:${GEPPETTO_PORT}/org.geppetto.frontend/GeppettoServlet
+GEPPETTO_BASE_URL=http://${GEPPETTO_HOST}:${GEPPETTO_PORT}/org.geppetto.frontend/geppetto
+
+BASE_PROJECT_FILES_HOST='http://scidash:8000/static/projects/'
+
+# ToDo: change in development sentry dsn
+SENTRY_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # base of all errors
+SENTRY_SYS_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # system/infra errors
+SENTRY_APP_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # application errors
diff --git a/service/dotenv/env-docker b/service/dotenv/env-docker
index 9965aa72..9b4cb575 100644
--- a/service/dotenv/env-docker
+++ b/service/dotenv/env-docker
@@ -21,3 +21,8 @@ GEPPETTO_SERVLET_URL=ws://${GEPPETTO_HOST}:${GEPPETTO_PORT}/org.geppetto.fronten
GEPPETTO_BASE_URL=http://${GEPPETTO_HOST}:${GEPPETTO_PORT}/org.geppetto.frontend/geppetto
BASE_PROJECT_FILES_HOST='http://scidash:8000/static/projects/'
+
+# ToDo: change in development sentry dsn
+SENTRY_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # base of all errors
+SENTRY_SYS_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # system/infra errors
+SENTRY_APP_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # application errors
diff --git a/service/dotenv/scidash_env b/service/dotenv/scidash_env
index 9e12c06d..a8d35e2d 100644
--- a/service/dotenv/scidash_env
+++ b/service/dotenv/scidash_env
@@ -1,5 +1,6 @@
DEVELOPMENT=1
PRODUCTION=0
+HTTPS=1
DB_HOST='scidash-postgres'
DB_PORT=5432
@@ -21,3 +22,8 @@ GEPPETTO_SERVLET_URL=ws://${GEPPETTO_HOST}:${GEPPETTO_PORT}/org.geppetto.fronten
GEPPETTO_BASE_URL=http://${GEPPETTO_HOST}:${GEPPETTO_PORT}/org.geppetto.frontend/geppetto
BASE_PROJECT_FILES_HOST='http://scidash:8000/static/projects/'
+
+# ToDo: change in development sentry dsn
+SENTRY_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # base of all errors
+SENTRY_SYS_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # system/infra errors
+SENTRY_APP_DSN="https://f25ea2fac2da427684e6875c7de5c296@sentry.metacell.us/2" # application errors
diff --git a/service/geppetto/GeppettoConfiguration.json b/service/geppetto/GeppettoConfiguration.json
index e336de2e..a53e5eed 100644
--- a/service/geppetto/GeppettoConfiguration.json
+++ b/service/geppetto/GeppettoConfiguration.json
@@ -1,7 +1,7 @@
{
- "_README": "http://docs.geppetto.org/en/latest/build.html",
+ "_README": "https://docs.geppetto.org/en/latest/build.html",
"contextPath": "org.geppetto.frontend",
- "useSsl": false,
+ "useSsl": true,
"embedded": false,
"embedderURL": ["/"],
"rootRedirect": "",
@@ -18,8 +18,8 @@
"title": "SciDash",
"description": "SciDash is a project that enables the reproducible execution and visualization of data-driven unit test (SciUnit) for assessing model quality.",
"type": "website",
- "url": "http://scidash.github.io/",
- "icon": "http://scidash.github.io/assets/icons/favicon-32x32.png",
- "image": "http://scidash.github.io/assets/scidash-text.png"
+ "url": "https://scidash.github.io/",
+ "icon": "https://scidash.github.io/assets/icons/favicon-32x32.png",
+ "image": "https://scidash.github.io/assets/scidash-text.png"
}
}
diff --git a/service/geppetto/app-config.xml b/service/geppetto/app-config.xml
index 366a69de..542b9dc7 100644
--- a/service/geppetto/app-config.xml
+++ b/service/geppetto/app-config.xml
@@ -9,7 +9,7 @@
-
+
diff --git a/service/geppetto/simulator.scidash/app-config.xml b/service/geppetto/simulator.scidash/app-config.xml
index 09447198..2513b884 100644
--- a/service/geppetto/simulator.scidash/app-config.xml
+++ b/service/geppetto/simulator.scidash/app-config.xml
@@ -9,13 +9,13 @@
-
+
-
+
diff --git a/service/hooks/pre-commit b/service/hooks/pre-commit
index 963e6a18..8de93ec1 100644
--- a/service/hooks/pre-commit
+++ b/service/hooks/pre-commit
@@ -2,4 +2,7 @@
set -e
-make lint
+#make lint # not for no, first need to fixup pep8
+
+make coverage-badge
+git add coverage.svg
diff --git a/service/k8s/ingress.yaml b/service/k8s/ingress.yaml
new file mode 100644
index 00000000..dd7b36c7
--- /dev/null
+++ b/service/k8s/ingress.yaml
@@ -0,0 +1,38 @@
+apiVersion: cert-manager.io/v1alpha2
+kind: Issuer
+metadata:
+ name: "letsencrypt-scidash"
+spec:
+ acme:
+ server: https://acme-v02.api.letsencrypt.org/directory
+ email: filippo@metacell.us
+ privateKeySecretRef:
+ name: letsencrypt-scidash
+ solvers:
+ - http01:
+ ingress:
+ class: nginx
+---
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+ annotations:
+ cert-manager.io/issuer: letsencrypt-scidash
+ kubernetes.io/ingress.class: nginx
+ kubernetes.io/tls-acme: "true"
+ nginx.ingress.kubernetes.io/proxy-body-size: 512m
+ name: scidash-web-nginx-ingress
+spec:
+ rules:
+ - host: {{DOMAIN}}
+ http:
+ paths:
+ - backend:
+ serviceName: scidash
+ servicePort: 8000
+ path: /
+ pathType: ImplementationSpecific
+ tls:
+ - hosts:
+ - {{DOMAIN}}
+ secretName: scidash-tls
diff --git a/service/k8s/scidash-postgres.yaml b/service/k8s/scidash-postgres.yaml
new file mode 100644
index 00000000..4c195faf
--- /dev/null
+++ b/service/k8s/scidash-postgres.yaml
@@ -0,0 +1,69 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: scidash-db
+ labels:
+ app: scidash-db
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 10G
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: scidash-postgres
+ labels:
+ app: scidash-postgres
+ usesvolume: scidash-db
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: scidash-postgres
+ template:
+ metadata:
+ labels:
+ app: scidash-postgres
+ usesvolume: scidash-db
+ spec:
+ containers:
+ - name: scidash-postgres
+ image: us.gcr.io/metacellllc/scidash_db:{{TAG}}
+ imagePullPolicy: IfNotPresent
+ env:
+ - name: PGDATA
+ value: /opt/scidash-db/data
+ ports:
+ - containerPort: 5432
+ volumeMounts:
+ - name: scidash-db
+ mountPath: /opt/scidash-db
+ affinity:
+ podAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ - labelSelector:
+ matchExpressions:
+ - key: usesvolume
+ operator: In
+ values:
+ - scidash-db
+ topologyKey: "kubernetes.io/hostname"
+ volumes:
+ - name: scidash-db
+ persistentVolumeClaim:
+ claimName: scidash-db
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: scidash-postgres
+spec:
+ type: ClusterIP
+ ports:
+ - port: 5432
+ targetPort: 5432
+ selector:
+ app: scidash-postgres
diff --git a/service/k8s/scidash-redis.yaml b/service/k8s/scidash-redis.yaml
new file mode 100644
index 00000000..dffea033
--- /dev/null
+++ b/service/k8s/scidash-redis.yaml
@@ -0,0 +1,35 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: scidash-redis
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: scidash-redis
+ template:
+ metadata:
+ labels:
+ app: scidash-redis
+ spec:
+ containers:
+ - name: scidash-redis
+ image: registry.hub.docker.com/library/redis:latest
+ imagePullPolicy: IfNotPresent
+ env:
+ - name: CF_SHORT_REVISION
+ value: "{{CF_SHORT_REVISION}}"
+ ports:
+ - containerPort: 6379
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: scidash-redis
+spec:
+ type: ClusterIP
+ ports:
+ - port: 6379
+ targetPort: 6379
+ selector:
+ app: scidash-redis
diff --git a/service/k8s/scidash-virgo.yaml b/service/k8s/scidash-virgo.yaml
new file mode 100644
index 00000000..4ef154bb
--- /dev/null
+++ b/service/k8s/scidash-virgo.yaml
@@ -0,0 +1,67 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: scidash-geppettotmp
+ labels:
+ app: scidash-geppettotmp
+ usesvolume: scidash-geppettotmp
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 2G
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: scidash-virgo
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: scidash-virgo
+ template:
+ metadata:
+ labels:
+ app: scidash-virgo
+ usesvolume: scidash-geppettotmp
+ spec:
+ securityContext:
+ runAsUser: 1000
+ fsGroup: 1000
+ containers:
+ - name: scidash-virgo
+ image: us.gcr.io/metacellllc/scidash_geppetto:{{TAG}}
+ imagePullPolicy: IfNotPresent
+ ports:
+ - containerPort: 8080
+ volumeMounts:
+ - name: scidash-geppettotmp
+ mountPath: /opt/virgo/geppettoTmp
+ affinity:
+ podAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ - labelSelector:
+ matchExpressions:
+ - key: usesvolume
+ operator: In
+ values:
+ - scidash-geppettotmp
+ topologyKey: "kubernetes.io/hostname"
+ volumes:
+ - name: scidash-geppettotmp
+ persistentVolumeClaim:
+ claimName: scidash-geppettotmp
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: scidash-virgo
+spec:
+ type: LoadBalancer
+ ports:
+ - port: 8080
+ targetPort: 8080
+ selector:
+ app: scidash-virgo
diff --git a/service/k8s/scidash.yaml b/service/k8s/scidash.yaml
new file mode 100644
index 00000000..d72e34f7
--- /dev/null
+++ b/service/k8s/scidash.yaml
@@ -0,0 +1,87 @@
+apiVersion: v1
+kind: Secret
+metadata:
+ name: scidash
+type: Opaque
+stringData:
+ SOCIAL_AUTH_GOOGLE_OAUTH2_KEY: {{OAUTH_GOOGLE_KEY}}
+ SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET: {{OAUTH_GOOGLE_SECRET}}
+ SOCIAL_AUTH_TWITTER_KEY: {{OAUTH_TWITTER_KEY}}
+ SOCIAL_AUTH_TWITTER_SECRET: {{OAUTH_TWITTER_SECRET}}
+ SOCIAL_AUTH_GITHUB_KEY: {{OAUTH_GITHUB_KEY}}
+ SOCIAL_AUTH_GITHUB_SECRET: {{OAUTH_GITHUB_SECRET}}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: scidash-geppettotmp
+ labels:
+ app: scidash-geppettotmp
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 2G
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: scidash
+ labels:
+ usesvolume: scidash-geppettotmp
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: scidash
+ template:
+ metadata:
+ labels:
+ app: scidash
+ usesvolume: scidash-geppettotmp
+ spec:
+ containers:
+ - name: scidash
+ image: us.gcr.io/metacellllc/scidash:{{TAG}}
+ imagePullPolicy: IfNotPresent
+ env:
+ - name: ENVIRONMENT
+ value: "{{ENVIRONMENT}}"
+ ports:
+ - containerPort: 8000
+ volumeMounts:
+ - name: scidash-geppettotmp
+ mountPath: /opt/virgo/geppettoTmp
+ - name: secrets
+ mountPath: "/etc/secrets"
+ readOnly: true
+ affinity:
+ podAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ - labelSelector:
+ matchExpressions:
+ - key: usesvolume
+ operator: In
+ values:
+ - scidash-geppettotmp
+ topologyKey: "kubernetes.io/hostname"
+ volumes:
+ - name: scidash-geppettotmp
+ persistentVolumeClaim:
+ claimName: scidash-geppettotmp
+ - name: secrets
+ secret:
+ secretName: scidash
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: scidash
+spec:
+ type: LoadBalancer
+ ports:
+ - port: 8000
+ targetPort: 8000
+ selector:
+ app: scidash
diff --git a/service/scripts/db-create-psql.sh b/service/scripts/db-create-psql.sh
index 0c9a6a0c..c26decd4 100755
--- a/service/scripts/db-create-psql.sh
+++ b/service/scripts/db-create-psql.sh
@@ -1,8 +1,24 @@
-#!/bin/bash
+#!/bin/sh -x
+psql -d template1 -c 'create extension hstore;'
+
+psql < /dev/null
-git clone $pygeppetto_django_repo;
+if [ $? -eq 0 ]; then
+ git clone -b $pygeppetto_branch $pygeppetto_django_repo $pygeppetto_folder
+else
+ git clone -b geppetto-scidash $pygeppetto_django_repo $pygeppetto_folder
+fi
cd $pygeppetto_folder;
diff --git a/service/scripts/install-frontend.sh b/service/scripts/install-frontend.sh
index f76218b4..2a95e28f 100755
--- a/service/scripts/install-frontend.sh
+++ b/service/scripts/install-frontend.sh
@@ -4,22 +4,66 @@ node -v
root_path=$PWD;
-geppetto_repo="https://github.com/openworm/org.geppetto.frontend.git -b v0.4.2-beta";
-extension_repo="https://github.com/MetaCell/geppetto-scidash.git -b 4.0.2";
+geppetto_repo="https://github.com/openworm/org.geppetto.frontend.git"
+geppetto_branch="geppetto-scidash"
+extension_repo="https://github.com/MetaCell/geppetto-scidash.git"
+extension_branch="geppetto-scidash"
+geppetto_client_repo="https://github.com/openworm/geppetto-client.git"
+geppetto_client_branch="geppetto-scidash"
+
+POSITIONAL=()
+while [[ $# -gt 0 ]]
+do
+key="$1"
+
+case $key in
+ -b|--branch)
+ geppetto_branch="$2"
+ extension_branch="$2"
+ shift # past argument
+ shift # past value
+ ;;
+ *) # unknown option
+ POSITIONAL+=("$1") # save it in an array for later
+ shift # past argument
+ ;;
+esac
+done
+set -- "${POSITIONAL[@]}" # restore positional parameters
geppetto_path="./static/org.geppetto.frontend";
-geppetto_app_path="$geppetto_path/src/main/webapp";
-extension_path="$geppetto_app_path/extensions/geppetto-scidash";
-sample_config_path="./service/geppetto/GeppettoConfiguration.json";
-actual_config_path="$geppetto_app_path/GeppettoConfiguration.json";
+geppetto_app_path="$geppetto_path/src/main/geppetto-scidash";
+
+# The lines below can be commented out when we will be in line with the latest geppetto, differently
+# if we try to test development this won't work since we are diverging from geppetto frontend at the moment
+git ls-remote --heads --tags $geppetto_repo | grep -E 'refs/(heads|tags)/'$geppetto_branch > /dev/null
+
+if [ $? -eq 0 ]; then
+ git clone -b $geppetto_branch $geppetto_repo $geppetto_path;
+else
+ git clone -b development $geppetto_repo $geppetto_path;
+fi
+
+git ls-remote --heads --tags $extension_repo | grep -E 'refs/(heads|tags)/'$extension_branch > /dev/null
+
+if [ $? -eq 0 ]; then
+ git clone -b $extension_branch $extension_repo $geppetto_app_path;
+else
+ git clone -b development $extension_repo $geppetto_app_path;
+fi
-git clone $geppetto_repo $geppetto_path;
-git clone $extension_repo $extension_path;
+cd $geppetto_path/src/main;
+rm -rf webapp;
+mv geppetto-scidash webapp;
+cd webapp;
-cd $geppetto_app_path;
+git ls-remote --heads --tags $geppetto_client_repo | grep -E 'refs/(heads|tags)/'$geppetto_client_branch > /dev/null
-npm install
+if [ $? -eq 0 ]; then
+ git clone -b $geppetto_client_branch $geppetto_client_repo;
+else
+ git clone -b development $geppetto_client_repo;
+fi
-cd $root_path;
+npm ci
-cp $sample_config_path $actual_config_path;
diff --git a/service/scripts/trigger-travis.sh b/service/scripts/trigger-travis.sh
new file mode 100755
index 00000000..a1a08ead
--- /dev/null
+++ b/service/scripts/trigger-travis.sh
@@ -0,0 +1,126 @@
+#!/bin/sh -f
+
+# Trigger a new Travis-CI job.
+# Ordinarily, a new Travis job is triggered when a commit is pushed to a
+# GitHub repository. The trigger-travis.sh script provides a programmatic
+# way to trigger a new Travis job.
+
+# Usage:
+# trigger-travis.sh [--pro] [--branch BRANCH] GITHUBID GITHUBPROJECT TRAVIS_ACCESS_TOKEN [MESSAGE]
+# For example:
+# trigger-travis.sh typetools checker-framework `cat ~/private/.travis-access-token` "Trigger for testing"
+#
+# where --pro means to use travis-ci.com instead of travis-ci.org, and
+# where TRAVIS_ACCESS_TOKEN is, or ~/private/.travis-access-token contains,
+# the Travis access token.
+#
+# Your Travis access token is the text after "Your access token is " in
+# the output of this compound command:
+# travis login && travis token
+# (If the travis program isn't installed, then use either of these two commands:
+# gem install travis
+# sudo apt-get install ruby-dev && sudo gem install travis
+# Don't do "sudo apt-get install travis" which installs a trajectory analyzer.)
+# Note that the Travis access token output by `travis token` differs from the
+# Travis token available at https://travis-ci.org/profile .
+# If you store it in in a file, make sure the file is not readable by others,
+# for example by running: chmod og-rwx ~/private/.travis-access-token
+
+# To use this script to trigger a dependent build in Travis, do two things:
+#
+# 1. Set an environment variable TRAVIS_ACCESS_TOKEN by navigating to
+# https://travis-ci.org/MYGITHUBID/MYGITHUBPROJECT/settings
+# The TRAVIS_ACCESS_TOKEN environment variable will be set when Travis runs
+# the job, but won't be visible to anyone browsing https://travis-ci.org/.
+#
+# 2. Add the following to your .travis.yml file, where you replace
+# OTHERGITHUB* by a specific downstream project, but you leave
+# $TRAVIS_ACCESS_TOKEN as literal text:
+#
+# jobs:
+# include:
+# - stage: trigger downstream
+# jdk: oraclejdk8
+# script: |
+# echo "TRAVIS_BRANCH=$TRAVIS_BRANCH TRAVIS_PULL_REQUEST=$TRAVIS_PULL_REQUEST"
+# if [[ ($TRAVIS_BRANCH == master) &&
+# ($TRAVIS_PULL_REQUEST == false) ]] ; then
+# curl -LO --retry 3 https://raw.github.com/mernst/plume-lib/master/bin/trigger-travis.sh
+# sh trigger-travis.sh OTHERGITHUBID OTHERGITHUBPROJECT $TRAVIS_ACCESS_TOKEN
+# fi
+
+# TODO: Show how to use the --branch command-line argument.
+# TODO: Enable the script to clone a particular branch rather than master.
+# This would require a way to know the relationships among branches in
+# different GitHub projects. It's easier to run all your tests within a
+# single Travis job, if they fit within Travis's 50-minute time limit.
+
+# An alternative to this script would be to install the Travis command-line
+# client and then run:
+# travis restart -r OTHERGITHUBID/OTHERGITHUBPROJECT
+# That is undesirable because it restarts an old job, destroying its history,
+# rather than starting a new job which is our goal.
+
+# Parts of this script were originally taken from
+# http://docs.travis-ci.com/user/triggering-builds/
+
+
+if [ "$#" -lt 3 ] || [ "$#" -ge 7 ]; then
+ echo "Wrong number of arguments $# to trigger-travis.sh; run like:"
+ echo " trigger-travis.sh [--pro] [--branch BRANCH] GITHUBID GITHUBPROJECT TRAVIS_ACCESS_TOKEN [MESSAGE]" >&2
+ exit 1
+fi
+
+if [ "$1" = "--pro" ] ; then
+ TRAVIS_URL=travis-ci.com
+ shift
+else
+ TRAVIS_URL=travis-ci.org
+fi
+
+if [ "$1" = "--branch" ] ; then
+ shift
+ BRANCH="$1"
+ shift
+else
+ BRANCH=master
+fi
+
+USER=$1
+REPO=$2
+TOKEN=$3
+if [ $# -eq 4 ] ; then
+ MESSAGE=",\"message\": \"$4\""
+elif [ -n "$TRAVIS_REPO_SLUG" ] ; then
+ MESSAGE=",\"message\": \"Triggered by upstream build of $TRAVIS_REPO_SLUG commit "`git rev-parse --short HEAD`"\""
+else
+ MESSAGE=""
+fi
+## For debugging:
+# echo "USER=$USER"
+# echo "REPO=$REPO"
+# echo "TOKEN=$TOKEN"
+# echo "MESSAGE=$MESSAGE"
+
+body="{
+\"request\": {
+ \"branch\":\"$BRANCH\"
+ $MESSAGE
+}}"
+
+# It does not work to put / in place of %2F in the URL below. I'm not sure why.
+curl -s -X POST \
+ -H "Content-Type: application/json" \
+ -H "Accept: application/json" \
+ -H "Travis-API-Version: 3" \
+ -H "Authorization: token ${TOKEN}" \
+ -d "$body" \
+ https://api.${TRAVIS_URL}/repo/${USER}%2F${REPO}/requests \
+ | tee /tmp/travis-request-output.$$.txt
+
+if grep -q '"@type": "error"' /tmp/travis-request-output.$$.txt; then
+ exit 1
+fi
+if grep -q 'access denied' /tmp/travis-request-output.$$.txt; then
+ exit 1
+fi
