ImportError: No module named pluggy

[sschuberth]

Using ScanCode version 2.0.1.post135.94d8aaa8 I'm running into ImportError: No module named pluggy errors on CI, but not locally. Still investigating why. Stacktrace:

Traceback (most recent call last):
File "/workspace/tools/scancode/bin/scancode", line 11, in <module>
  load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')()
File "/workspace/tools/scancode/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 561, in load_entry_point
  return get_distribution(dist).load_entry_point(group, name)
File "/workspace/tools/scancode/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2649, in load_entry_point
  return ep.load()
File "/workspace/tools/scancode/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2303, in load
  return self.resolve()
File "/workspace/tools/scancode/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2309, in resolve
  module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/workspace/tools/scancode/src/scancode/cli.py", line 52, in <module>
  import plugincode.output
File "/workspace/tools/scancode/src/plugincode/output.py", line 32, in <module>
  from pluggy import HookimplMarker

[sschuberth]

I don't see

Configuring ScanCode for first use...

in the CI logs. We're running ScanCode from a Git checkout that is being fetched to incrementally. Can it be that the "first use" check does not properly work if new dependencies like pluggy are being introduced?

[sschuberth]

Interestingly, I also don't see the SCANCODE_DEV_MODE file being created. Shouldn't that be the case when running from a Git checkout?

[pombredanne]

@sschuberth is this a fresh checkout?

[pombredanne]

@sschuberth if this is a incremental checkout and this was already "configured" or run before, then you should run ./configure on this after each incremental pull

[sschuberth]

is this a fresh checkout?

No, it's not.

if this is a incremental checkout and this was already "configured" or run before, then you should run ./configure on this after each incremental pull

Ah, then this is a misunderstanding on my side. I though the logic would be smart enough to know for which commit ./configure was run in case of running from a checkout.

Couldn't this be easily implemented by writing the commit SHA1 into SCANCODE_DEV_MODE?

[pombredanne]

For instance if I start with a master checkout and run ./scancode -h then I switch to develop and run ./scancode -h there again, I get the same error as yours.
Running ./configure fixes this and is needed when working from "incremental" checkouts

Configure is pretty dumb and does not try to keep track of anything as it could be run from a plain tarbal or checkout. SCANCODE_DEV_MODE at this stage only deals with checking the license index cache and reindexing if needed... I guess we could use it for something else too, but there is no logic that I can fathom that could be applied as we cannot rely on Git being there and this being a checkout, short of applying the same logic that we have in the license cache invalidation... but this would slow down every development mode run quite a bit.

[sschuberth]

Looks like I sort of misunderstood all of what SCAN_DEV_MODE is about, then. In the code it says to create a "development mode tag file", but I could not find any comment on what "development mode" actually is, and how it differs from "normal mode". Could you maybe amend that code comment accordingly?

WRT automatic re-configuring when running from Git checkouts, I was thinking about a simply logic like:

• check if a .git directory is present
• if so, check a file or something for which Git commit configure was run last
• if that commit matches the current working tree, and the tree is clean, do not re-configure
• otherwise re-configure

Do you think that would work?

[pombredanne]

@sschuberth your logic could work alright and we could use SCAN_DEV_MODE for this. The problem I see is one of a chicken and egg: configure is dumb and therefore the logic to do all this would have to be external to configure itself which would make the machinery a tad complicated.
On the other hand simply rerun ./configure after a pull is all that's needed. Do you think this could work out for you?

[sschuberth]

On the other hand simply rerun ./configure after a pull is all that's needed. Do you think this could work out for you?

Sure, that's not an issue for us to do. I was just thinking about making the logic more convenient along the way 😉

I would have implemented the logic simply as part of scancode / scancode.bat. Let me try if I can get something to work today or tomorrow.

In any case, I'll close this issue as it turned out to be invalid. Thanks for your support!

[pombredanne]

@sschuberth no need to close especially if you are working on a possible fix :P And I surely need need to add doc for the SCANCODE_DEV_MODE....

[sschuberth]

And I surely need need to add doc for the SCANCODE_DEV_MODE....

Just did that in #737 :-)

[pombredanne]

@sschuberth so I guess this could be closed now? I merged #737

[sschuberth]

Well, #737 was only about improving the docs. If you don't mind, I wanted to leave this open until I actually improved the reconfiguration logic.