Is pytest output guaranteed to contain test file paths to relative to cwd

[karrtikr]

For instance, in my case I had run the following command and got this output

C:\Users\karraj\AppData\Local\Programs\Python\Python37\python.exe c:\Users\karraj\Desktop\vscode-python\pythonFiles\pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir c:\Users\karraj\Desktop\Desktop\test_demo --junit-xml=C:\Users\karraj\AppData\Local\Temp\tmp-41724BpAkWghx9dx.xml Tests/steve

================================== FAILURES ===================================
_______________________ FailingTests.test_failure_still _______________________

self = <hello_test.FailingTests testMethod=test_failure_still>

    def test_failure_still(self):
>       self.assertEqual("I'm right!", "no, I am!")
E       AssertionError: "I'm right!" != 'no, I am!'
E       - I'm right!
E       + no, I am!

Tests\steve\hello_test.py:6: AssertionError
- generated xml file: C:\Users\karraj\AppData\Local\Temp\tmp-4172kPl1yrfyWOkV.xml -
============================== 1 failed in 0.12s ==============================

Tests\steve\hello_test.py is not the absolute path here, but the path relative to my current working directory. Is it guaranteed to contain relative paths always?

[nicoddemus]

Hi @karrtikr,

Those paths are actually related to the rootdir. You can see the rootdir in the header.

The best way to make sure you have the same rootdir always is by creating a pytest.ini in the location you want.

[karrtikr]

Ah I see, but it'll always be relative to the rootdir, correct?
Given this assumption, in case of windows, our extension had made the assumption that the relative path cannot contain ":" - do you think that's fair?
There's some users who I suspect might be seeing absolute paths there. (microsoft/vscode-python#12403)

I will get back here as soon as I can once the user replies, mind keeping this open for a while?

[nicoddemus]

Given this assumption, in case of windows, our extension had made the assumption that the relative path cannot contain ":" - do you think that's fair?

I think it is possible, if the rootdir and the test file reside in different drives.

So you are one of the vscode developres? Cool! You guys are doing great work.

Btw, why are you interested in parsing the terminal output? I ask because for integration usually the best approach are hooks instead.

[karrtikr]

Thanks, Python extension VSCode dev here 👋

I think it is possible, if the rootdir and the test file reside in different drives.

That's fair. But for our extension, we force the rootdir to be the workspace root, and only support running test files inside the workspace root, so they should be on the same drive.

Btw, why are you interested in parsing the terminal output? I ask because for integration usually the best approach are hooks instead.

I am not entirely sure, we were parsing output long before I joined the team. Maybe because hooks were not there earlier, or were not sufficient? I can consult with my team on whether we want to move towards this.

[karrtikr]

@nicoddemus Can you please point to these hooks which we can use to run pytest where we won't need to parse the result?

[nicoddemus]

Thanks, Python extension VSCode dev here 👋

Cool 👋

I am not entirely sure, we were parsing output long before I joined the team

Oh I see. I remember @brettcannon tweeting something along the lines of parsing the output as an early implementation, but that is brittle because we don't treat our output as stable, but he was aware that hooks were the way to go.

Maybe because hooks were not there earlier, or were not sufficient?

They were introduced quite a long time ago, in 2010. I believe they should be more than enough to get any information you need from a pytest run, and they are quite stable.

A quick example on how vscode could integrate with pytest using hooks:

class MyPlugin:

   def pytest_runtest_logreport(report):
       if report.when == "call" and report.outcome == "passed":
           print(f"{report.nodeid} passed")

pytest.main(["path to file.py::test name"], plugins=[MyPlugin()])

The list of hooks can be found in the API Reference, but I bet the more interesting ones for VS integration would be the reporting hooks.

Please feel free to ask any questions, we are more than glad to help with getting the best pytest integration possible in vscode!

[brettcannon]

So we use the hooks for discovery, but we don't for execution (yet; we plan to, but no clue on an ETA).

[Zac-HD]

Hey all 👋

It looks like we've answered this question so I'm closing the issue; feel free to open another if you have more questions.