Assess support for MicroPython

[krzema12]

See https://kotlinlang.slack.com/archives/C0289CS37AS/p1637517359061900?thread_ts=1637517359.061900&cid=C0289CS37AS

https://core-electronics.com.au/tutorials/Python_and_MicroPython.html

Let's check how costly it would be to support both regular Python and Micropython (MP).

Ideas/steps:

• check if there's a regular Linux interpreter for MP
• smoke test: try running a sample output from current Kotlin Python compiler and see how it goes
• run box tests with MP interpreter and see what's the success rate comparing to what we have on Python
• hook MP interpreter into the box tests so that CI runs both
• think of cases where it would be challenging to support both at the same time

[krzema12]

Running on Linux: see

• https://snapcraft.io/micropython
• https://github.com/micropython/micropython/tree/master/ports/unix

[krzema12]

Running on my STM32F429 DISCOVERY board: try the image provided after this change: micropython/micropython#8018, once available. To learn if it's available, check the files on this page: https://micropython.org/download/STM32F429DISC/.

Nightly build already exposes a HEX: https://micropython.org/resources/firmware/STM32F429DISC-20211123-unstable-v1.17-195-gbb7aae557.hex .

[krzema12]

Running example code from here that works with regular Python:

def factorial(n):
    tmp0_subject = n <= 1
    if tmp0_subject == True:
        tmp = 1
    elif tmp0_subject == False:
        tmp = n * factorial((n - 1).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000))
    else:
        noWhenBranchMatchedException()
    
    return tmp

def numberOfCombinations(n, k):
    return (factorial(n) // (factorial(k) * factorial((n - k).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000))).__add__
(0x8000_0000_0000_0000).__and__(0xffff_ffff_ffff_ffff).__sub__(0x8000_0000_0000_0000)).__add__(0x8000_0000_0000_0000).__and__(0xffff_f
fff_ffff_ffff).__sub__(0x8000_0000_0000_0000)

def sumOverflowDemo(a, b):
    return (a + b).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000)

print(isPowerOfTwo(32))
print(isPowerOfTwo(33))

print(factorial(5))

print(numberOfCombinations(4, 3))

print(sumOverflowDemo(2 ** 31 - 1, 2 ** 31 - 10))

gives

$ micropython calculations.py 
Traceback (most recent call last):
  File "calculations.py", line 21, in <module>
  File "calculations.py", line 2, in isPowerOfTwo
AttributeError: 'int' object has no attribute '__add__'

[krzema12]

@SerVB I remember that using these __add__ is a workaround to some issue. Do you remember any details? How hard would it be to do something more casual here that MicroPython would support? Fortunately, it's easy to run MP, at least on Linux. It's available in SnapStore.

[SerVB]

It's for overflow simulation, here is an explanation:

kotlin-python/compiler/ir/backend.py/src/org/jetbrains/kotlin/ir/backend/py/transformers/irToPy/PyIntrinsicTransformers.kt

Lines 132 to 136 in df5531b

	// Simulation of overflow for integer numbers:
	// (number).__add__(HALF_UNSIGNED).__and__(MAX_UNSIGNED).__sub__(HALF_UNSIGNED): From Hacker's Delight (Sign Extension)
	// Example:
	// Overflow of a Byte:
	// (128).__add__(0x80).__and__(0xFF).__sub__(0x80) => -128

We can replace magic methods (like __add__) with ordinary operators (like +) easily and it should work in MP. The initial idea of using these methods was to show that these calls are some internal pieces.

We can, for example, just replace them. Or support a new compilation flag like enum class CompilationMode { PYTHON3_9, MICRO_PYTHON }.

[krzema12]

Thanks! I propose (especially in such seemingly simple cases) is to emit such Python code that works for both Pythons. The compilation flag introduces some extra complexity, maybe we'll need it in the future, though.

My first step towards supporting MicroPython is adding some flag to the box tests that selects Python interpreter (#75). Let's then compare the outputs and see what tests pass in regular Python and fail in MP so far, then we can learn about more differences in practice.

Two good news I see:

• passing https://github.com/krzema12/kotlin-python/blob/python-backend/python/experiments/out_ir.py to micropython yields no parsing issues, which means that the Python generated for the JS IR stdlib is correctly parsed by MicroPython (not yet executed!)
• we're just starting implementing our stdlib, so we'll be able to take into account both pythons already

[krzema12]

228 more failed tests on MIcroPython: 73ed312

Failure causes:

• lots of MemoryError: memory allocation failed, allocating XXX bytes - https://forum.micropython.org/viewtopic.php?t=4912

Differences in failure reasons:

• minor: wording in case of unknown name: is not VS isn't - py/runtime: Adjust message for not defined name micropython/micropython#8024

[LouisCAD]

I think it'd be best to have a MicroPython specific compiler mode, that also doesn't bring the full Kotlin stdlib, given how far apart the RAM quantities with regular Python running devices and even mobile phones.

[LouisCAD]

Particularly, the kotlin.collections package is probably not something you want to bring into a MicroPython running device.
I'd make an embedded systems alternative with typealiases if I want to share some code with other platforms that can use kotlin.collections.

[LouisCAD]

Kotlin to MicroPython for me would be mostly a way to have Kotlin's syntax for MicroPython APIs. Note that you can embed assembly instructions right into MicroPython. This is the kind of thing I'd like to abstract away under an inlined function.

This 30min talk from Damien George, the author of MicroPython himself, is quite informative about optimizations you can do in MicroPython to ensure fastest code: https://www.youtube.com/watch?v=hHec4qL00x0

[krzema12]

I'm afraid it's beyond the current scope of this project. We already struggle with delivering a target for regular Python, and focusing on such optimizations (even if they are necessary sooner or later, I don't deny it) would make the regular Python support way less likely to appear within even 1-2 years.

For now, what I want to do towards MicroPython support, is ensuring correctness. I. e. emitting correct MicroPython code, not really paying attention to the performance and memory. That's the first, mandatory step that I think we can afford. What happens next is in fact optimization that by rule shouldn't happen prematurely. From optimization that we plan to do even for regular Python is dead code elimination, so it would bring only necessary stdlib functions.

Emitting assembly is an entirely different chapter which brings concrete architectures into the picture, and the feature of Python is abstracting them out. Kotlin/Native seems to be closer to this, well, native world. Here we deal with Python and its beauty of being portable.