Executor crashed for invalid UTF-8

[werner-matthias]

The executor isn't robust against the output of non-valid UTF-8.
If a student program generates an output that is neither ASCII nor UTF-8, the executor crashes. This is also true is the student program and the runtime generated ANSI control codes to color the error message.
While the validator could use a try-except to handle such cases, the output is lost in any case.

Suggestion: Instead of (utf-8-)strings the output (and probably the input and the standard error) should use byte stings.

Disadvantage: This would be a breaking change.

[troeger]

Using byte strings in the executor output caching solves only half of the problem. What happens on the web server side, when the student output is about to be rendered in HTML? Since we can no longer assume any encoding of the data, you will generate broken renderings in the submission details.

[werner-matthias]

Using byte strings in the executor output caching solves only half of the problem. What happens on the web server side, when the student output is about to be rendered in HTML? Since we can no longer assume any encoding of the data, you will generate broken renderings in the submission details.

Well, this could be handled at validator level by recoding to UTF-8 with the "replace" option or something similar.
Maybe, this could even done within the executor, thus, it doesn't crash but provides still UTF-8. In this way, it wouldn't even be a breaking change, and the output could still be parsed.