[gh ci] less output ( --durations=50)

[sshleifer]

Way too much output in this
This will make it slightly better.
cc @stas00

[stas00]

Thank you for the heads up - I will be working on all these related issues shortly - too much data indeed, but not just that.

[stas00]

I'd say remove them complete for now and also -rA - I need to experiment and see how to make this data available w/o making logs unusable.

[sshleifer]

More context on github actions:

if we can somehow catch the return value of
bash

x= python -m pytest -n 1 --dist=loadfile -s examples --durations=50 | tee test_output.txt
save test_output.txt # always succeeds
sys.exit(x)

or something like that, we can make huge progress on the github actions issue and start making artifacts files.

The reason artifacts files broke was that even in the below code, even if line 1 raises an error, line 2 succeeds so github actions thinks the job succeeded

python -m pytest -n 1 --dist=loadfile -s examples --durations=50 | tee test_output.txt
save test_output.txt # always succeeds

[stas00]

oh, unless I'm missing something, we don't need any of the workarounds.

I already have the first requested component (failures) working, see: #7995

Check out the resulting artifacts:
https://app.circleci.com/pipelines/github/huggingface/transformers/14354/workflows/1ccd616e-218f-4ae1-b413-91d2faa0e942/jobs/104363/artifacts

this is what we want right?

pytest provides hooks for doing this kind of work, so it's just figuring out which hooks to call.

In your example instead of x = cmd what you need to save is $? which is the exit status of the command.

[sshleifer]

That's great, but note that this is all much easier in circleci. My ask is to make it work in github actions.
The failures are already pretty easy to find in circleci.

2. you mean

x= python -m pytest -n 1 --dist=loadfile -s examples --durations=50 | tee test_output.txt
save test_output.txt # always succeeds
sys.exit($x)

?

[sshleifer]

test_failures.txt is really nice!

[stas00]

Ah, good point. let me see what other handy reports I can squeese in circle-ci and then I will move to github actions.

I'm not following your question yet, let me get to github actions and then it'll probably make sense, but yes I'm referring to that example when I said:

In your example instead of x = cmd what you need to save is $? which is the exit status of the command.

i.e. sys.exit($?) but you must save it right away upon pytest completion, since the next command will overwrite it.