Negative "size" parameter for memset() via mpz_as_bytes

[gwangmu]

Description

In this crash, the function mpz_as_bytes attempted to feed a negative value to memset. We observed that CPython throws a TypeError saying the "byteorder" parameter to to_bytes() must be str. The parameter is bool in all PoCs instead.

This crash was manifested as heap-buffer-overflow in an older commit (commit 813d559), but it was changed to negative-size-param in the recent commit (commit a3862e7, latest as of 2023-09-26). We've attached five PoCs and their stack traces.

poc.zip

Proof of Concept

$ # build unix port with ASAN, at the root source code directory.
$ export CC=clang
$ export CXX=clang++
$ export CFLAGS="-fsanitize=address -fno-omit-frame-pointer"
$ export CXXFLAGS=$CFLAGS
$ export LDFLAGS=$CFLAGS
$ export DEBUG=1
$ make -C mpy-cross -j
$ make -C ports/unix -j all lib
$
$ # run a poc.
$ export ASAN_OPTIONS="detect_leaks=0"
$ ./ports/unix/build-standard/micropython <poc_file>

Environment

Ubuntu 20.04
Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
Memory: 64 GB

Affected Version

v1.20.0 (commit a3862e7, latest as of 2023-09-26)
v1.20.0 (commit 813d559, 2023-06-19)

[agatti]

This doesn't seem to reproduce anymore on current master:

$ for i in ~/tmp/poc/*.py; do echo $i; ./build-standard/micropython $i; done
/home/agatti/tmp/poc/poc_1.py
Traceback (most recent call last):
  File "/home/agatti/tmp/poc/poc_1.py", line 4, in <module>
OverflowError: buffer too small
/home/agatti/tmp/poc/poc_2.py
Traceback (most recent call last):
  File "/home/agatti/tmp/poc/poc_2.py", line 5, in <module>
OverflowError: buffer too small
/home/agatti/tmp/poc/poc_3.py
Traceback (most recent call last):
  File "/home/agatti/tmp/poc/poc_3.py", line 5, in <module>
OverflowError: buffer too small
/home/agatti/tmp/poc/poc_4.py
Traceback (most recent call last):
  File "/home/agatti/tmp/poc/poc_4.py", line 5, in <module>
OverflowError: buffer too small
/home/agatti/tmp/poc/poc_5.py
Traceback (most recent call last):
  File "/home/agatti/tmp/poc/poc_5.py", line 4, in <module>
OverflowError: buffer too small

$