Possible flakey tests/micro/zephyr/test_zephyr.py "test_relay[mps2_an521]"

[mbs-octoml]

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-8597/38/pipeline

tvm._ffi.base.TVMError: MicroSessionTimeoutError: failed to read reply message after timeout 5s

tests/scripts/task_python_microtvm.sh passes all tests internally.

[gromero]

@areusch It seems the bug we're chasing recently is still there? Unfortunately I was not able to reproduce it on my environment even after ~6 runs in sequence, just being able to reproduce once by chance.

[mbs-octoml]

Here's another one:
https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-8735/2/pipeline/46

[areusch]

we think there are still some lingering bugs, potentially Zephyr-side though not clear now, but we suspect #8761 will address one class of these failures.

[areusch]

@gromero is investigating further

[manupak]

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-8795/14/pipeline
I got this today and have been seeing this intermittently.

[ekalda]

I got a similar micro themed issue as well, possibly related - https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-8900/2/pipeline

[areusch]

the current thinking on these is that it's potentially a QEMU-side issue that is triggered by running a bunch of tests in a row. this test started failing with the #8595 effort. I haven't been able to reproduce it locally in the same way it's failing in the CI. However, at one point during development, we were failing to kill the qemu process and thereby leaving ~10 qemu-system-arm running. If you used a less powerful machine, you could reproduce a similar effect.

When I was able to reproduce this locally, I connected to the emulated ARM system via gdb and reset it. This timeout occurs while TVM is waiting to read an RPC reply from the emulated serial port. I was able to observe that, after resetting the emulated ARM system (while TVM was waiting for a reply), the rebooted ARM system could actually send UART data and TVM could read that data. However, the ARM system did not seem to be able to receive UART data from TVM.

This leads me to suspect one of two root causes:

1. a bug in the Zephyr firmware for MPS2_AN521 that may only manifest itself on QEMU
2. a race condition in the Zephyr qemu UART emulation that fails to transmit data to the ARM system, but can read it back just fine.

@gromero was investigating this before I went on PTO Aug 19. any updates? Perhaps we should consider temporarily disabling mps2_an521 until we can root-cause and fix this.

[asparkhi]

https://ci.tlcpack.ai/job/tvm/job/PR-8833/6/consoleFull I got this a few days back as well.

[gromero]

@gromero was investigating this before I went on PTO Aug 19. any updates? Perhaps we should consider temporarily disabling mps2_an521 until we can root-cause and fix this.

@areusch Hi Andrew. Unfortunately, no meaningful update on it. I've just got back from vacation as well and I'll need more time to dig into it. Yeah I think the best option for now is to just xfail it. I've sent #8927 to do that. I hope that's sufficient on the CI environment. Cheers.