pytest tests marked as passed or failed if test name contains upper letters

[Galileo-Galilei]

Hello,

I've encountered a little bug recently in the "Tests" tab. I am not 100% sure that the bug is related to vscode or to the python extension. According to the answer on vscode repo, it is due to the extension, that's why I'm reposting here.

Environment data

• VS Code version: 1.34.0
• Extension version (available under the Extensions sidebar): 2019.6.24221
• OS and version: Windows_NT x64 6.1.7601
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.8 64 bit
• Type of virtual environment used : conda
• Relevant/affected Python packages and their versions: user interface of the test tab, pytest
• Jedi or Language Server? "python.jediEnabled" = True

My file settings.json is :

{
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

Bug description :

When a test file contains an upper letter, it is not displayed properly in the test tab. After running if passed or failed, it is displayed with a "grey question mark".

Steps to Reproduce:

1. Create an empty project (note : use a conda environment where pytest is installed)
2. Create a "tests" folder
   2.1 Create a test_working.py file

import pytest
def test_this_one_works():
    assert 1==1

2.2 Create a test_NotWorking.py file with the same code

import pytest
def test_this_one_works():
    assert 1==1

In the Test tab of vscode, run Discover Tests
If everything ran properly, you should see the following architecture :

• (?) tests
  • (?) test_notworking.py (with lower case!!!)
    • (?) test_this_one_does_not_work
  • (?) test_working.py
    • (?) test_this_one_works

In the Test tab of vscode, run Run All Tests.

Actual & expected behaviour

-(green tick ok) tests
- (?) test_notworking.py <- BUG - EXPECTED BEHAVIOUR : should have a green tick and have name in upper case
- (?) test_this_one_does_not_work <- BUG - EXPECTED BEHAVIOUR : should have a green tick and have name in upper case
- (green tick ok) test_working.py (with lower case!!!)
- (green tick ok) test_this_one_works

Logs

The tests seem to be ran whatever is displayed (the bug only concerns the user interface) : indeed in the bottom line (where my python version and my environment are shown), I see an icon "two tests passed". When i run pytest tests in the command line, everything is fine.

Python Test log

python C:\Users\F400411\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear tests
============================= test session starts =============================
platform win32 -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: c:\Users\F400411\Documents\1_Datascience\3_MISSIONS\bug_test_vscode
collected 2 items

tests\test_NotWorking.py .                                               [ 50%]
tests\test_working.py .                                                  [100%]

- generated xml file: C:\Users\F400411\AppData\Local\Temp\tmp-5180zo59mydLbZ4a.xml -
========================== 2 passed in 0.03 seconds ===========================
python C:\Users\F400411\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear tests
============================= test session starts =============================
platform win32 -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: c:\Users\F400411\Documents\1_Datascience\3_MISSIONS\bug_test_vscode
collected 2 items

tests\test_NotWorking.py .                                               [ 50%]
tests\test_working.py .                                                  [100%]

- generated xml file: C:\Users\F400411\AppData\Local\Temp\tmp-5180SM9GGMumKvVy.xml -
========================== 2 passed in 0.02 seconds ===========================

Conclusion :

This bugs is not critical at all since it only concerns the UI, but in my old projects I can't see the tests in the UI (while it used to work perfectly fine few weeks ago). I definitely do not want to change the names of the scripts (the tests scripts are a mirror of the scripts names of my project).

Any hope this would be easy to fix?

Many thanks for reading and any further help !
Galileo

[karrtikr]

Thanks for reporting this @Galileo-Galilei

[PhilMacKay]

This is very likely related to #6410 except in the current issue, upper case letters are in the file name instead of the directory name.

[marsfan]

@PhilMacKay @DonJayamanne I just stumbled across this trying to fix it on my computer. It is definitely (in my opnion) an extension of #6410.

In settings.json if I put
"python.testing.pytestArgs": [ "tests" ],
to set up pytest, it correctly displays the markers as a red x or a green check.
But if I put
"python.testing.pytestArgs": [ "Tests" ],`
in settings .json, the results are all the grey questionmark, even after running.

Hereis the python test log from the a passed test run with all lowercase folder name in settings.json

platform win32 -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: d:\Documents\GitHub\Data-Processing
collected 1 item

tests\test_dataextraction.py .                                           [100%]

- generated xml file: C:\Users\Gabe\AppData\Local\Temp\tmp-143848g0BcuWYtuLN.xml -
========================== 1 passed in 0.07 seconds ===========================

And here is the failed, where the folder is listed as Tests

platform win32 -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: d:\Documents\GitHub\Data-Processing
collected 1 item

Tests\test_dataextraction.py .                                           [100%]

- generated xml file: C:\Users\Gabe\AppData\Local\Temp\tmp-14384m2leEWBHKABE.xml -
========================== 1 passed in 0.08 seconds ===========================

For me it appears that the actual folder name does not matter, just the capitalization in settings.json

[PhilMacKay]

@marsfan I've tried this on my side and it doesn't solve my issue.
Have you tried to see what happens when the python file itself has capital letters in it? What about if it was located inside a sub-folder with a capital letter in it?

Edit: After extensive tests, I have been able to reproduce your issue. Using a Pytest version below 5.0.0 and Python 3.7.3, I get the exact same behaviour as you. It enables me to run tests :D
For a complete fix, we need to change the "NORMCASE" method in ~.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\testing_tools\adapter\pytest_discovery.py in class TestCollector. I changed NORMCASE = staticmethod(os.path.normcase) to NORMCASE = staticmethod(os.path.normpath) and it solved the problem!