isort behavior different on linux vs osx even with config

[ahonnecke]

on OSX:

web on  fox-295 [⇡$!] via 🐍 3.6.4 took 2s
➜ pipenv run isort --recursive --check-only server
ERROR: /Users/ahonnecke/Code/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py Imports are incorrectly sorted.
ERROR: /Users/ahonnecke/Code/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py Imports are incorrectly sorted.


web on  fox-295 [⇡$!] via 🐍 3.6.4
➜ head -n16 /Users/ahonnecke/Code/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py
import json
import os
from time import sleep
from unittest.mock import MagicMock

from django.test import TestCase

from wiremock.constants import Config
from wiremock.resources.mappings import Mapping
from wiremock.resources.mappings.resource import Mappings
from wiremock.server import WireMockServer


from assets.models import Ledger
from dad.configuration import PortfolioManagerConfiguration
from portfolio_manager.models import LedgerAddress, Portfolio

web on  fox-295 [⇡$!] via 🐍 3.6.4
➜ head -n16 /Users/ahonnecke/Code/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py
import json
import os
from time import sleep
from unittest.mock import MagicMock

from django.test import TestCase

from wiremock.constants import Config
from wiremock.resources.mappings import Mapping
from wiremock.resources.mappings.resource import Mappings
from wiremock.server import WireMockServer


from assets.models import Ledger
from dad.configuration import PortfolioManagerConfiguration
from portfolio_manager.wallets.ledger_client import LedgerClient

web on  fox-295 [⇡$!] via 🐍 3.6.4
➜ pipenv run isort --recursive --check-only server
ERROR: /Users/ahonnecke/Code/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py Imports are incorrectly sorted.
ERROR: /Users/ahonnecke/Code/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py Imports are incorrectly sorted.

web on  fox-295 [⇡$!] via 🐍 3.6.4 took 2s
➜ pipenv run isort --recursive server
Fixing /Users/ahonnecke/Code/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py
Fixing /Users/ahonnecke/Code/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py
Skipped 26 files

web on  fox-295 [⇡$!] via 🐍 3.6.4 took 2s
➜ head -n16 /Users/ahonnecke/Code/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py
import json
import os
from time import sleep
from unittest.mock import MagicMock

from django.test import TestCase

from wiremock.constants import Config
from wiremock.resources.mappings import Mapping
from wiremock.resources.mappings.resource import Mappings
from wiremock.server import WireMockServer

from assets.models import Ledger
from dad.configuration import PortfolioManagerConfiguration
from portfolio_manager.wallets.ledger_client import LedgerClient
from util.security import create_user, get_token_for_user

web on  fox-295 [⇡$!] via 🐍 3.6.4
➜ head -n16 /Users/ahonnecke/Code/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py
import json
import os
from time import sleep
from unittest.mock import MagicMock

from django.test import TestCase

from wiremock.constants import Config
from wiremock.resources.mappings import Mapping
from wiremock.resources.mappings.resource import Mappings
from wiremock.server import WireMockServer

from assets.models import Ledger
from dad.configuration import PortfolioManagerConfiguration
from portfolio_manager.models import LedgerAddress, Portfolio
from util.security import create_client_user, get_token_for_user

Cloud based CI platform (Circle CI / ubuntu)

circleci@02354f6b3c74:~/project/repos/web$ pipenv run isort --recursive --check-only server
ERROR: /home/circleci/project/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py Imports are incorrectly sorted.
ERROR: /home/circleci/project/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py Imports are incorrectly sorted.

circleci@02354f6b3c74:~/project/repos/web$ cp /home/circleci/project/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py /home/circleci/project/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py.bak

circleci@02354f6b3c74:~/project/repos/web$ cp /home/circleci/project/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py /home/circleci/project/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py.bak

circleci@02354f6b3c74:~/project/repos/web$ pipenv run isort --recursive server
Fixing /home/circleci/project/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py
Fixing /home/circleci/project/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py
Skipped 26 files
circleci@02354f6b3c74:~/project/repos/web$ diff /home/circleci/project/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py /home/circleci/project/repos/web/server/portfolio_manager/graphql/tests/test_ledger_addresses_query.py.bak
8,12d7
< from wiremock.constants import Config
< from wiremock.resources.mappings import Mapping
< from wiremock.resources.mappings.resource import Mappings
< from wiremock.server import WireMockServer
<
17a13,16
> from wiremock.constants import Config
> from wiremock.resources.mappings import Mapping
> from wiremock.resources.mappings.resource import Mappings
> from wiremock.server import WireMockServer
circleci@02354f6b3c74:~/project/repos/web$ diff /home/circleci/project/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py /home/circleci/project/repos/web/server/portfolio_manager/wallets/test_supported_ledgers.py.bak
8,12d7
< from wiremock.constants import Config
< from wiremock.resources.mappings import Mapping
< from wiremock.resources.mappings.resource import Mappings
< from wiremock.server import WireMockServer
<
17a13,16
> from wiremock.constants import Config
> from wiremock.resources.mappings import Mapping
> from wiremock.resources.mappings.resource import Mappings
> from wiremock.server import WireMockServer
circleci@02354f6b3c74:~/project/repos/web$ pipenv run isort --recursive --check-only server
Skipped 26 files

isort

[settings]
known_django=django
sections=FUTURE,STDLIB,DJANGO,THIRDPARTY,FIRSTPARTY,LOCALFOLDER

TLDR, isort requires a different number of newlines on ubuntu.

[hoylemd]

I'm having a similar issue, except instead of ubuntu, I'm trying to run it in a python:3.6 docker container.

[ahonnecke]

Now I feel silly, I'm actually running it in a circleci/python:3.6.4-node-browsers docker container.

[hoylemd]

Interesting... Now I want to try it on a bunch of other platforms to narrow it down. Maybe it's just docker, or the python images?

[ahonnecke]

I managed to hackily resolve it by ordering the imports in the isort config, so changing it to this:

[settings]
known_django=django
known_wiremock=wiremock
sections=FUTURE,STDLIB,DJANGO,THIRDPARTY,FIRSTPARTY,LOCALFOLDER,WIREMOCK

Normalized the behavior

[hoylemd]

Awesome! I'll try that on my setup tomorrow.

I guess the problem is in how it defaults the sections order, somehow works differently depending on the platform. Sounds like a fun bug hunt :)

[hoylemd]

Alas, this didn't work for me. A lot of the discrepancies appear to be about the space between sections.

When I run isort in a shell on macos, it doesn't seem to be able to tell that a local module (at ./module_name, from the working directory) doesn't belong in the same group as third party libraries like sqlalchemy. But the container understands this distinction perfectly.

(I just added the sections bit - I'm not using django or wiremock)

I wonder if it has anything to do with the egg-info directory in the project? I don't fully understand what it does, but it makes sense to me that it would alter how isort views the local module if there's an egg-info for it. But there's still a big question: why does my container behave differently than local since they both have the egg-info directory...

[ahonnecke]

I think that you need to specify the section order for each module that you're getting errant spaces around. I was getting an extra space above wiremock, adding wiremock to the ordered sections resolved that so I think you'd need to add each package that's getting extra spaces above it to the config file.

[hoylemd]

I solved it! the general problem was that somehow some local modules were appearing to import from site-packages, marking them as third party. (probably had something to do with the egg-info directory, but I don't really know).

Solution was to add the local packages to known_firstparty. For some reason it also had trouble identifying flask as a third party (it would default to first party), but I just added it to known_thirdparty, and it's all good.

[scorphus]

Make sure that all packages and specially Python are in the exact same version on both environments. Check #724 for reference. Please remember to close this issue once you manage to fix it.

[ahonnecke]

They seemingly are, from what I can tell. Is there a further check that I can perform? @scorphus

In the docker container

circleci@21868dbbaf66:~/project/repos/web$ pipenv run isort --version

/#######################################################################\

     `sMMy`
     .yyyy-                                                      `
    ##soos##                                                    ./o.
          `     ``..-..`         ``...`.``         `   ```` ``-ssso```
     .s:-y-   .+osssssso/.     ./ossss+:so+:`    :+o-`/osso:+sssssssso/
     .s::y-   osss+.``.``     -ssss+-.`-ossso`   ssssso/::..::+ssss:::.
     .s::y-   /ssss+//:-.`   `ssss+     `ssss+   sssso`       :ssss`
     .s::y-   `-/+oossssso/  `ssss/      sssso   ssss/        :ssss`
     .y-/y-       ````:ssss`  ossso.    :ssss:   ssss/        :ssss.
     `/so:`    `-//::/osss+   `+ssss+-/ossso:    /sso-        `osssso/.
       \/      `-/oooo++/-      .:/++:/++/-`      ..           `://++/.


         isort your Python imports for you so you don't have to

                            VERSION 4.3.4

\########################################################################/

circleci@21868dbbaf66:~/project/repos/web$ pipenv run python --version
Python 3.6.4

Locally

➜ pipenv run isort --version

/#######################################################################\

     `sMMy`
     .yyyy-                                                      `
    ##soos##                                                    ./o.
          `     ``..-..`         ``...`.``         `   ```` ``-ssso```
     .s:-y-   .+osssssso/.     ./ossss+:so+:`    :+o-`/osso:+sssssssso/
     .s::y-   osss+.``.``     -ssss+-.`-ossso`   ssssso/::..::+ssss:::.
     .s::y-   /ssss+//:-.`   `ssss+     `ssss+   sssso`       :ssss`
     .s::y-   `-/+oossssso/  `ssss/      sssso   ssss/        :ssss`
     .y-/y-       ````:ssss`  ossso.    :ssss:   ssss/        :ssss.
     `/so:`    `-//::/osss+   `+ssss+-/ossso:    /sso-        `osssso/.
       \/      `-/oooo++/-      .:/++:/++/-`      ..           `://++/.


         isort your Python imports for you so you don't have to

                            VERSION 4.3.4

\########################################################################/


web on  handle-soft-timeout [⇡$] via 🐍 3.6.4
➜ pipenv run python --version
Python 3.6.4

[scorphus]

One thing I can think of, is there any cache being saved and restored? That may leave the container virtualenv in a different state than your local one.

Or may I assume you're building and running the container locally? I did that with the following Dockerfile and got exact same results from both the local container's virtualenv and my local virtualenv. See:

FROM circleci/python:3.6.4

USER root

RUN mkdir /src /venv
COPY . /src/
RUN chown -R circleci:circleci /src /venv

USER circleci
WORKDIR /src

RUN python3 -m venv ../venv && \
    . ../venv/bin/activate && \
    pip install -r requirements.txt

CMD python main.py

From container's virtualenv:

(venv) circleci@850827ac06ca:/src$ python --version
Python 3.6.4
(venv) circleci@850827ac06ca:/src$ isort -c -vb

/#######################################################################\

     `sMMy`
     .yyyy-                                                      `
    ##soos##                                                    ./o.
          `     ``..-..`         ``...`.``         `   ```` ``-ssso```
     .s:-y-   .+osssssso/.     ./ossss+:so+:`    :+o-`/osso:+sssssssso/
     .s::y-   osss+.``.``     -ssss+-.`-ossso`   ssssso/::..::+ssss:::.
     .s::y-   /ssss+//:-.`   `ssss+     `ssss+   sssso`       :ssss`
     .s::y-   `-/+oossssso/  `ssss/      sssso   ssss/        :ssss`
     .y-/y-       ````:ssss`  ossso.    :ssss:   ssss/        :ssss.
     `/so:`    `-//::/osss+   `+ssss+-/ossso:    /sso-        `osssso/.
       \/      `-/oooo++/-      .:/++:/++/-`      ..           `://++/.


         isort your Python imports for you so you don't have to

                            VERSION 4.3.4

\########################################################################/

else-type place_module for json returned STDLIB
else-type place_module for logging returned STDLIB
else-type place_module for os returned STDLIB
else-type place_module for sys returned STDLIB
else-type place_module for traceback returned STDLIB
from-type place_module for collections returned STDLIB
from-type place_module for datetime returned STDLIB
from-type place_module for logging.config returned STDLIB
from-type place_module for XPTO returned FIRSTPARTY
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
else-type place_module for datetime returned STDLIB
else-type place_module for json returned STDLIB
else-type place_module for logging returned STDLIB
else-type place_module for os returned STDLIB
else-type place_module for smtplib returned STDLIB
else-type place_module for ssl returned STDLIB
else-type place_module for time returned STDLIB
else-type place_module for uuid returned STDLIB
from-type place_module for email.mime.text returned STDLIB
from-type place_module for typing returned STDLIB
else-type place_module for etcd3 returned THIRDPARTY
else-type place_module for pytz returned THIRDPARTY
from-type place_module for jinja2 returned THIRDPARTY
from-type place_module for tornado.ioloop returned THIRDPARTY
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
from-type place_module for .allocation_agent returned LOCALFOLDER
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
else-type place_module for datetime returned STDLIB
from-type place_module for os returned STDLIB
from-type place_module for time returned STDLIB
from-type place_module for unittest returned STDLIB
from-type place_module for unittest.mock returned STDLIB
from-type place_module for xml.etree returned STDLIB
else-type place_module for pytz returned THIRDPARTY
from-type place_module for XPTO FIRSTPARTY
from-type place_module for XPTO FIRSTPARTY
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
SUCCESS: /src/somedir/somefile.py Everything Looks Good!
(venv) circleci@850827ac06ca:/src$ cat .isort.cfg 
[settings]
line_length = 100
lines_after_imports = 2
not_skip =
    __init__.py
(venv) circleci@850827ac06ca:/src$

From local virtualenv:

(venv) scorphus@mac:~/repo$ python --version
Python 3.6.4
(venv) scorphus@mac:~/repo$ isort -c -vb

/#######################################################################\

     `sMMy`
     .yyyy-                                                      `
    ##soos##                                                    ./o.
          `     ``..-..`         ``...`.``         `   ```` ``-ssso```
     .s:-y-   .+osssssso/.     ./ossss+:so+:`    :+o-`/osso:+sssssssso/
     .s::y-   osss+.``.``     -ssss+-.`-ossso`   ssssso/::..::+ssss:::.
     .s::y-   /ssss+//:-.`   `ssss+     `ssss+   sssso`       :ssss`
     .s::y-   `-/+oossssso/  `ssss/      sssso   ssss/        :ssss`
     .y-/y-       ````:ssss`  ossso.    :ssss:   ssss/        :ssss.
     `/so:`    `-//::/osss+   `+ssss+-/ossso:    /sso-        `osssso/.
       \/      `-/oooo++/-      .:/++:/++/-`      ..           `://++/.


         isort your Python imports for you so you don't have to

                            VERSION 4.3.4

\########################################################################/

else-type place_module for json returned STDLIB
else-type place_module for logging returned STDLIB
else-type place_module for os returned STDLIB
else-type place_module for sys returned STDLIB
else-type place_module for traceback returned STDLIB
from-type place_module for collections returned STDLIB
from-type place_module for datetime returned STDLIB
from-type place_module for logging.config returned STDLIB
from-type place_module for tmh.controller.allocation_agent returned FIRSTPARTY
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/main.py Everything Looks Good!
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tests/__init__.py Everything Looks Good!
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tests/unit/placeholder.py Everything Looks Good!
else-type place_module for datetime returned STDLIB
from-type place_module for os returned STDLIB
from-type place_module for time returned STDLIB
from-type place_module for unittest returned STDLIB
from-type place_module for unittest.mock returned STDLIB
from-type place_module for xml.etree returned STDLIB
else-type place_module for pytz returned THIRDPARTY
from-type place_module for tmh.controller returned FIRSTPARTY
from-type place_module for tmh.controller.allocation_agent returned FIRSTPARTY
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tests/unit/test_allocation_agent.py Everything Looks Good!
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tests/func/placeholder.py Everything Looks Good!
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tmh/__init__.py Everything Looks Good!
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tmh/controller/__init__.py Everything Looks Good!
from-type place_module for .allocation_agent returned LOCALFOLDER
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tmh/controller/allocation_agent/__init__.py Everything Looks Good!
else-type place_module for datetime returned STDLIB
else-type place_module for json returned STDLIB
else-type place_module for logging returned STDLIB
else-type place_module for os returned STDLIB
else-type place_module for smtplib returned STDLIB
else-type place_module for ssl returned STDLIB
else-type place_module for time returned STDLIB
else-type place_module for uuid returned STDLIB
from-type place_module for email.mime.text returned STDLIB
from-type place_module for typing returned STDLIB
else-type place_module for etcd3 returned THIRDPARTY
else-type place_module for pytz returned THIRDPARTY
from-type place_module for jinja2 returned THIRDPARTY
from-type place_module for tornado.ioloop returned THIRDPARTY
SUCCESS: /Users/pablo/Workspace/mobilityhouse/tmh-nl-tennet-allocation-agent/tmh/controller/allocation_agent/allocation_agent.py Everything Looks Good!
(venv) scorphus@mac:~/repo$ cat .isort.cfg 
[settings]
line_length = 100
lines_after_imports = 2
not_skip =
    __init__.py
(venv) scorphus@mac:~/repo$

Try running all from local and see where you get from there. I had the exact same issue, BTW.

I hope this helps you solve your issue as well.

[ahonnecke]

Not running it locally, but actually in the CI pipeline. I'll try again with -c -vb thanks!

[ahonnecke]

isort.version.local.txt

[aarcro]

Having this issue, There's actually different behavior between pipenv run isort (wrong) and pipenv shell then isort inside the subshell (correct).