Update Android App building and runtime #101

rtibbles · 2022-03-18T03:31:34Z

Removes PyEverywhere for building the app
Remove PyEverywhere for controlling the runtime
Updates Python for Android to the latest released version, with minimal patches on top to ensure continued function
Simplifies the P4A bootstrap to control URL loading from Python
Lets Kolibri find unused ports rather than hardcoding
Lints the codebase
Adds a Github Action for creating debug builds of the APK

Fixes #63
Fixes #90
Fixes #93
Fixes #94
Fixes #96
Fixes #97

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1202007641164460

…cal env vars.

.p4a

MBKayro · 2022-03-21T09:20:23Z

Thank you @rtibbles works perfectly :)

DXCanas

I think I've commented on the bits I can be useful on. No functional changes to request, but do have some questions and non-blocking requests for comments here and there 🙃

DXCanas · 2022-03-22T18:23:16Z

.github/workflows/build_apk.yml

+on:
+  workflow_dispatch:
+    # Inputs the workflow accepts.
+    inputs:


DXCanas · 2022-03-22T18:24:29Z

.github/workflows/build_apk.yml

+        options:
+          - 32bit
+          - 64bit
+  workflow_call:


Interesting distinction between call and dispatch

Dispatch is for manually triggering, call is for triggering by a different workflow.

DXCanas · 2022-03-22T18:26:01Z

.github/workflows/build_apk.yml

+    - uses: actions/checkout@v2
+      if: ${{ !inputs.ref }}
+    - uses: actions/checkout@v2
+      if: ${{ inputs.ref }}


This is an irritating wrinkle that we have to do for workflow_call as it gets run in the context of the calling repository, not this repository.

.github/workflows/build_apk.yml

DXCanas · 2022-03-22T18:31:34Z

.github/workflows/pre-commit.yml

@@ -0,0 +1,28 @@
+name: Linting


this'll be fun

DXCanas · 2022-03-22T19:02:48Z

allowlist.txt

+# Ensure that Django's SQLite backend module is included
+sqlite3/*
+lib-dynload/_sqlite3.so
+unittest/*
+wsgiref/*
+lib-dynload/_csv.so
+lib-dynload/_json.so
+# Django REST framework has a dependency on the Django test module
+django/test/*


I take it this is P4A's allowlist, for use during copy?

Yes - this allows things to be copied even if they are in the blocklist.

DXCanas · 2022-03-22T19:03:45Z

blocklist.txt

+# remove some assorted additional plugins
+kolibri/plugins/demo_server/*
+
+# remove python2-only stuff
+kolibri/dist/py2only/*
+
+# Remove cextensions
+kolibri/dist/cext*
+
+# remove source maps
+*.js.map
+
+# remove unused translation files from django and other apps
+kolibri/dist/rest_framework/locale/*
+kolibri/dist/django_filters/locale/*
+kolibri/dist/mptt/locale/*
+
+kolibri/dist/django/contrib/admindocs/locale/*
+kolibri/dist/django/contrib/auth/locale/*
+kolibri/dist/django/contrib/sites/locale/*
+kolibri/dist/django/contrib/contenttypes/locale/*
+kolibri/dist/django/contrib/flatpages/locale/*
+kolibri/dist/django/contrib/sessions/locale/*
+kolibri/dist/django/contrib/humanize/locale/*
+kolibri/dist/django/contrib/admin/locale/*
+
+# remove some django components entirely
+kolibri/dist/django/contrib/gis/*
+kolibri/dist/django/contrib/redirects/*
+kolibri/dist/django/conf/app_template/*
+kolibri/dist/django/conf/project_template/*
+kolibri/dist/django/db/backends/postgresql_psycopg2/*
+kolibri/dist/django/db/backends/postgresql/*
+kolibri/dist/django/db/backends/mysql/*
+kolibri/dist/django/db/backends/oracle/*
+kolibri/dist/django/contrib/postgres/*
+
+# remove bigger chunks of django admin (may not want to do this)
+kolibri/dist/django/contrib/admin/static/*
+kolibri/dist/django/contrib/admin/templates/*
+
+# other assorted testing stuff
+*/test/*
+*/tests/*
+kolibri/dist/tzlocal/test_data/*
+
+# remove some unnecessary apps
+kolibri/dist/redis_cache/*
+kolibri/dist/redis/*


niice. How's the .apk size compare to the old process?

It's a few MB smaller, it seems. I think that's probably mostly from dropping all the PyEverywhere runtime code.

DXCanas · 2022-03-22T19:04:26Z

requirements.txt

+cython
+virtualenv
+git+https://github.com/learningequality/python-for-android@350d7158a0a35f578a95d50de969101579bbdc4f#egg=python-for-android


DXCanas · 2022-03-22T19:05:40Z

scripts/rundocker.sh

  --env P4A_RELEASE_KEYSTORE_PASSWD \
  --env P4A_RELEASE_KEYALIAS_PASSWD \
  --env ARCH \
+  --env ANDROID_HOME=${CONTAINER_HOME}/.local/android \


DXCanas · 2022-03-22T19:06:35Z

scripts/version.py

-        build_number = build_base_number + 2 * int(buildkite_build_number) + increment_for_64bit
-        print(build_number)
+        build_number = (
+            build_base_number + 2 * int(buildkite_build_number) + increment_for_64bit


oof.

I recall some recent discussions around changing this up. We holding off on that?

Also, this still depends on BUILDKITE_BUILD_NUMBER - are we setting that manually via GHA? Or are we skipping this bit altogether? Noting that this is how Android determines whether or not to "upgrade" and installed app.

At the moment I'm only building the debug build in Github Actions - resolving this seemed like a blocker to being able to build the non-debug but develop builds for PRs.

rtibbles added 16 commits March 17, 2022 19:31

Simplify startup checking.

637a143

Remove unneeded flask.

5a5c130

Run server in foreground in service.

2f28d97

Clean up unused imports.

77d7e28

Cleanup unused function.

eff717b

Remove redundant environment variable.

3c81d33

Remove non-android code paths.

3e0ed5c

First pass vendoring of pew runtime components.

f4f7963

Consolidate environment initialization, use service for remoteshell.

048f2df

Remove preseeding from makefile. Rely on kolibri pre-seeding.

08262f5

Simplify main.py code and rely on Java webview updates.

35b8fc5

Vendor runnable. Defer to server for running port.

20a3644

Let server choose free port.

0bf7f5b

Limit number of threads used by servers.

b259de7

Refactor build for latest python for android.

c92b530

Lint codebase.

77b2501

rtibbles requested review from DXCanas and jamalex March 18, 2022 03:31

rtibbles force-pushed the vendor_build branch from 7ddf726 to 290bc71 Compare March 18, 2022 04:20

rtibbles marked this pull request as draft March 18, 2022 15:45

rtibbles added 2 commits March 18, 2022 08:53

Properly pin python for android version and run make setup and set lo…

3ee0815

…cal env vars.

Add pre-commit action.

6c8a333

rtibbles force-pushed the vendor_build branch from 290bc71 to 9ffe527 Compare March 18, 2022 15:56

rtibbles marked this pull request as ready for review March 18, 2022 15:57

Add basic github action for building the APK.

2ff48c2

rtibbles force-pushed the vendor_build branch from 9ffe527 to 2ff48c2 Compare March 18, 2022 16:02

MBKayro reviewed Mar 18, 2022

View reviewed changes

.p4a Show resolved Hide resolved

rtibbles added 3 commits March 19, 2022 16:55

Clean up Python requirements and simplify installation in dockerfile.

bc05533

Update icon to Google Play specifications.

860dba9

Clean up launch splash image. Remove no longer working make run.

37840f1

Run remoteshell service after loading webview to improve load times.

e803908

DXCanas reviewed Mar 22, 2022

View reviewed changes

rtibbles added 2 commits April 4, 2022 09:38

Add comments and be sure to segment action cache by architecture.

34a10e9

Update black.

23565cd

rtibbles merged commit 947fa84 into develop Apr 4, 2022

rtibbles deleted the vendor_build branch April 4, 2022 21:59

Update Android App building and runtime #101

Update Android App building and runtime #101

Uh oh!

Conversation

rtibbles commented Mar 18, 2022 • edited by DXCanas Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MBKayro commented Mar 21, 2022

Uh oh!

DXCanas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rtibbles commented Mar 18, 2022 •

edited by DXCanas

Loading