Skip to content
This repository was archived by the owner on Mar 6, 2023. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
eca19c9
add pyproject.toml
LukeWeidenwalker Jun 2, 2022
45cfc3c
update .gitignore to full list from github/gitignore
LukeWeidenwalker Jun 2, 2022
6ca0d26
add poetry.lock to .gitignore
LukeWeidenwalker Jun 2, 2022
97db7f5
add odc-algo dependency
LukeWeidenwalker Jun 2, 2022
eb0bf0b
add github actions workflow
LukeWeidenwalker Jun 2, 2022
eab9062
minor cleanups to trigger workflow
LukeWeidenwalker Jun 2, 2022
b3ba101
Merge pull request #1 from LukeWeidenwalker/setup-ci
LukeWeidenwalker Jun 2, 2022
60c9292
actually there is no debian GH runner :(
Jun 2, 2022
0aef1ab
attempt at installing gdal in ci
Jun 2, 2022
496dc9e
remove extra python versions for now
Jun 2, 2022
ae147af
remove another cheeky sudo
Jun 2, 2022
7e2f8ca
add sudo back in, seems necessary
Jun 2, 2022
44e8341
allow failrue for pytest to check rest of pipeline
Jun 2, 2022
03a234c
add name to test
Jun 2, 2022
e7fe618
add devcontainer
Jun 2, 2022
bd4cd52
add python3-venv to devcontainer
Jun 2, 2022
1624c2e
Correctly install the openeo_package
Jun 3, 2022
4bbf066
fix venv location
Jun 3, 2022
b3198a2
tests now work in devcontainer
Jun 3, 2022
8b89c14
update tests with matrix strategy
Jun 7, 2022
952ebf0
add mypy
Jun 7, 2022
fb276ce
add mypy config
Jun 8, 2022
02386ae
add poetry.lock to check how much it speeds up build
Jun 8, 2022
66be3f8
add gdal python dependency
Jun 8, 2022
6499f09
add older python versions for nox to docker image
Jun 8, 2022
8da5225
get rid of unnecessary stage
Jun 8, 2022
f2d9a54
add workflow for building devcontainer
Jun 8, 2022
3f2f380
name and add workflow_dispatch trigger
Jun 8, 2022
2ed011f
update image name to work on forks
Jun 8, 2022
a3587d1
fix image name templating
Jun 8, 2022
693547b
fix typo
Jun 8, 2022
d3c7a69
minor edit to test workflow trigger
Jun 8, 2022
bde545d
change devcontainer name
Jun 8, 2022
756850f
merge pipelines
Jun 8, 2022
e4846a0
add the latest tag
Jun 8, 2022
fbd4f3b
fix using output of non-existant job
Jun 8, 2022
4457529
fix docker tag lowercasing
Jun 8, 2022
09f5aff
fix syntax error
Jun 8, 2022
142d3c9
specify that we want to use ghcr.io cache!
Jun 8, 2022
25566be
fix ghcr.io/ reference for tests job too
Jun 8, 2022
7564343
try different cache-from config
Jun 8, 2022
8cc84fb
edit cache-to
Jun 8, 2022
53666f3
add login step to allow caching
Jun 8, 2022
4c47d37
pull image if missing
Jun 8, 2022
4afaea0
remove writing to cache in first step again
Jun 8, 2022
207efec
explicitly pull image before
Jun 9, 2022
9a3f64c
also build with cache in tests
Jun 9, 2022
80ff836
output to local docker to be available to run
Jun 9, 2022
cd0ddad
mount GITHUB_WORKSPACE into devcontainer
Jun 13, 2022
c07b8cd
pass NOXSESSION into container too!
Jun 13, 2022
1558131
Update README to new install
Jun 13, 2022
10386fe
fix user permissions issues
Jun 17, 2022
9db9ce4
add shebang to bashscript
Jun 17, 2022
59edddc
minor formatting
Jun 17, 2022
159f4ec
try passing user id etc as build arg
Jun 17, 2022
f980291
fix curly brackets
Jun 17, 2022
8e48fe0
ls statement for debugging permissions issue
Jun 17, 2022
6f01225
fix wrong syntax on COPY layer and get rid of 3.5 in image
Jun 20, 2022
2ae4a5a
add a few debugging statements
Jun 20, 2022
7ebf0f7
fix bracket type ..
Jun 20, 2022
4df0633
provide build args as strings (see https://github.com/docker/build-pu…
Jun 20, 2022
3ad4816
try other type of bracket
Jun 20, 2022
7db1b1e
try with manually set env variables
Jun 20, 2022
4fd8446
provide as strings
Jun 20, 2022
06abb3c
use =
Jun 20, 2022
4ea2ceb
uff, need to do that in both places !
Jun 20, 2022
be691cf
remove debugging echo statements again
Jun 20, 2022
ca98998
add UID things to readme.
Jun 20, 2022
be2ca59
Update readme with local docker build
Jun 22, 2022
d6f17e9
update README with additional instructions
Jun 23, 2022
602663f
update devcontainer.json with intial settings
Jun 23, 2022
8d9d4e0
Skip broken tests
Jun 23, 2022
165cb9b
get_time_dimension_from_data: raise exception instead of returning None
Jun 23, 2022
904b8ca
Ignore typing failures to get CI towards green
Jun 23, 2022
fdd1707
add missing import for pyyaml
Jun 23, 2022
7751a4c
update dependencies for pyyaml to resolve
Jun 23, 2022
c563ba4
fix mypy failures
Jun 24, 2022
d5e173c
update image build workflow to build the whole devcontainer
Jun 24, 2022
b5df3bc
change .venv location to in-project
Jun 24, 2022
cb218de
add $HOME/.local to path to get rid of pip warnings
Jun 24, 2022
3a370a7
add another folder to path
Jun 24, 2022
2e3da46
try username instead
Jun 24, 2022
76b2434
syntax fix
Jun 24, 2022
0903f48
set PATH after changing user
Jun 24, 2022
fc6fcb7
mount files to correct dir
Jun 24, 2022
a72c173
add missing slash
Jun 24, 2022
cfcad16
Skip failing test
Jun 24, 2022
1700f56
get rid of push step and use gha cache
Jun 24, 2022
f378250
add load back in
Jun 24, 2022
b6edba4
get rid of postCreateCommand
Jun 27, 2022
d4cc488
Update readme to reflect latest changes
Jun 27, 2022
31737d9
change order in readme.
Jun 28, 2022
779af0d
change user to ubuntu
Jun 28, 2022
9fbc431
remove remote user field, I misunderstood what this does
Jun 29, 2022
b305423
add optional dependencies with extras
Jun 30, 2022
03a9fe9
Revert "add optional dependencies with extras"
Jul 1, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
FROM osgeo/gdal:ubuntu-small-3.5.0 AS base

ARG DEBIAN_FRONTEND=noninteractive
ARG USERNAME=ubuntu
ARG GROUPNAME=ubuntu
ARG USER_UID=1000
ARG USER_GID=1000
ENV TZ=Etc/GMT

ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=off \
PIP_DISABLE_PIP_VERSION_CHECK=on \
PIP_DEFAULT_TIMEOUT=100 \
POETRY_HOME="/opt/poetry/" \
POETRY_NO_INTERACTION=1 \
POETRY_VIRTUALENVS_IN_PROJECT=true \
VENV_PATH=".venv/" \
APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=DontWarn

ENV PATH="$POETRY_HOME/bin:/home/$USERNAME/.local/bin:$VENV_PATH/bin:$PATH"

RUN apt-get update -qy && \
apt-get install -qy --no-install-recommends \
# libpq-dev and build-essential are necessary for psycopg2, which is required for datacube
build-essential \
libpq-dev

# Adapted from https://github.com/TheKevJames/tools/blob/master/docker-nox/Dockerfile
RUN apt-get update -qy && \
apt-get install -qy --no-install-recommends \
ca-certificates \
curl \
gnupg2 && \
. /etc/os-release && \
echo "deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu ${UBUNTU_CODENAME} main" > /etc/apt/sources.list.d/deadsnakes.list && \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F23C5A6CF475977595C89F51BA6932366A755776 && \
apt-get update -qy && \
apt-get install -qy --fix-missing --no-install-recommends \
git \
openssh-client \
python3.6 \
python3.6-dev \
python3.6-distutils \
python3.6-venv \
python3.7 \
python3.7-dev \
python3.7-distutils \
python3.7-venv \
python3.8 \
python3.8-dev \
python3.8-distutils \
python3.8-venv \
python3.9 \
python3.9-dev \
python3.9-distutils \
python3.9-venv \
python3.10 \
python3.10-dev \
python3.10-distutils \
python3.10-venv \
python3-pip \
python3-distutils

# Install Poetry - respects $POETRY_VERSION & $POETRY_HOME
ENV POETRY_VERSION=1.1.13
RUN curl -sSL https://install.python-poetry.org | python3 -

RUN useradd -ms /bin/bash --uid $USER_UID --user-group $USERNAME \
&& chown -R $USER_UID:$USER_GID /home/$USERNAME
USER $USER_UID

ENV PATH="$POETRY_HOME/bin:/home/$USERNAME/.local/bin:$VENV_PATH/bin:$PATH"

RUN curl -fsS https://bootstrap.pypa.io/get-pip.py --output /tmp/get-pip.py && \
python3.10 /tmp/get-pip.py && \
rm /tmp/get-pip.py

RUN python3.6 -m pip install --user --no-cache-dir --upgrade pip && \
python3.7 -m pip install --user --no-cache-dir --upgrade pip && \
python3.8 -m pip install --user --no-cache-dir --upgrade pip && \
python3.9 -m pip install --user --no-cache-dir --upgrade pip && \
python3.10 -m pip install --user --no-cache-dir --upgrade pip && \
rm -rf /var/cache/apt/lists

RUN python3.10 -m pip install --user --no-cache-dir 'nox-poetry==1.0.0'
29 changes: 29 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"name": "openeo-processes-python",

"dockerFile": "./Dockerfile",

// Sets the run context to one level up instead of the .devcontainer folder.
"context": "..",

// Set *default* container specific settings.json values on container create.
"settings": {
"python.defaultInterpreterPath": ".venv/bin/python",
"python.venvPath": ".venv",
"python.pythonPath": ".venv/bin/python",
"python.testing.pytestArgs": [
"tests"
],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.linting.enabled": true
},

// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance"
],

"postCreateCommand": "poetry install"
}
109 changes: 109 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Inspired by https://github.com/python-poetry/poetry/blob/master/.github/workflows/main.yml

name: CI Pipeline

on:
push:
paths-ignore:
- 'docs/**'
branches:
- master
pull_request:
paths-ignore:
- 'docs/**'
branches:
- '**'
workflow_dispatch:

env:
POETRY_VERSION: 1.1.13

defaults:
run:
shell: bash

jobs:
image-build:
name: Build devcontainer
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3

- name: Set up QEMU
uses: docker/setup-qemu-action@v2

- name: Setup buildx
uses: docker/setup-buildx-action@v2

- name: Downcase IMAGE_NAME
run: |
echo "IMAGE_NAME=${GITHUB_REPOSITORY,,}" >>${GITHUB_ENV}

- name: Add env variables
run: |
echo "USER_UID=$( id --user )" >>${GITHUB_ENV}
echo "USERNAME=$( id --user --name )" >>${GITHUB_ENV}
echo "USER_GID=$( id --group )" >>${GITHUB_ENV}

- name: Build image
id: image-build
uses: docker/build-push-action@v3
with:
file: ./.devcontainer/Dockerfile
context: .
tags: ${{ env.IMAGE_NAME }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
"USER_UID=${{ env.USER_UID }}"
"USERNAME=${{ env.USERNAME }}"
"USER_GID=${{ env.USER_GID }}"

tests:
name: ${{ matrix.python-version }} / ${{ matrix.session }}
needs: [image-build]
runs-on: ubuntu-20.04
strategy:
fail-fast: false # This prevents the entire matrix from stopping early if any job in the matrix fails.
matrix:
python-version: ["3.9", "3.8"]
session: ["tests", "mypy"]
env:
NOXSESSION: ${{ matrix.session }}

steps:
- uses: actions/checkout@v3

- name: Downcase IMAGE_NAME
run: |
echo "IMAGE_NAME=${GITHUB_REPOSITORY,,}" >>${GITHUB_ENV}

- name: Add env variables
run: |
echo "USER_UID=$( id --user )" >>${GITHUB_ENV}
echo "USERNAME=$( id --user --name )" >>${GITHUB_ENV}
echo "USER_GID=$( id --group )" >>${GITHUB_ENV}

- name: Set up QEMU
uses: docker/setup-qemu-action@v2

- name: Setup buildx
uses: docker/setup-buildx-action@v2

- name: Build image
id: image-build
uses: docker/build-push-action@v3
with:
file: ./.devcontainer/Dockerfile
context: .
load: true
tags: ${{ env.IMAGE_NAME }}
cache-from: type=gha
build-args: |
"USER_UID=${{ env.USER_UID }}"
"USERNAME=${{ env.USERNAME }}"
"USER_GID=${{ env.USER_GID }}"

- name: Run nox session
run: |
docker run --rm --mount type=bind,source="$GITHUB_WORKSPACE",target="$GITHUB_WORKSPACE" --workdir="$GITHUB_WORKSPACE" --env NOXSESSION=$NOXSESSION ${IMAGE_NAME} nox --python=${{ matrix.python-version }}
Loading