CodeQL GitHub Action works on upstream but not on fork

[Vampire]

The upstream https://github.com/spockframework/spock recently enabled the CodeQL GitHub action.
I made a new PR and the analysis was run.
The problem is, on upstream it checked Java and was successful: https://github.com/spockframework/spock/runs/784895802
While in the fork on the same commit it tried to extract JavaScript and failed: https://github.com/Vampire/spock/runs/784926819

Why do these behave different and how to fix it besides restricting the action to only run on upstream?

[Vampire]

Somehow in upstream it detects java while in the fork it detects java, javascript while the api request for languages returns the same JSON.
And it seems on upstream version 1.0.0 is used, whereas in the fork 0.0.0-20200601 is used. :-/
Of course upstream could explicitly define the languages I guess, but it seems there is something else borked.

[chrisgavin]

It looks like the detected languages in Vampire/spock were out-of-date and incorrectly included JavaScript.

I've re-run the language detection on this repository and now the languages seem to be correctly detected.

Please try re-running your workflow and see if this has fixed it.

If the languages being detected for a repository are incorrect, it is also possible to manually specify which languages to analyze when running the CodeQL Action by using the languages input to the init Action.

      - name: Initialize CodeQL
        uses: github/codeql-action/init@v1
        with:
          languages: java

[Vampire]

Yep, I also found out all this in the meantime.
The situation that caused this was, that the master branch of my fork was ages old on a commit that included several JavaScript files.
I then pushed the master branch to the current state and the CodeQL analysis again failed as the language detection needed a bit to update.
I now rerun the same analysis and now the languages are recognized correctly and it is working.

But I still see two flaws in the CodeQL action:

1. If JavaScript is configured as language (either automatically detected, or setup manually) but there are not JavaScript files found, the anaylsis fails with an error instead of simply finding no issues which I would expect.

2. The language detection is based on the files in the default branch. But this might not be optimal. If the CodeQL analysis runs on a branch or PR, then it should detect in that branch what languages are present.

   For example in this case I had, the master branch was ages old as I usually make a new branch per PR and don't really care about the master branch in my fork, so the languages were detected as Java and JavaScript, while the branches that were checked did not contain any JavaScript files.

   The same is also true the other way around, if you get a PR that adds a language, like for example some JavaScript files, they are not checked. As soon as you then merge that PR to master, the language is detected, the files are checked and can suddenly fail the check. This is even worse if you follow Gitflow workflow where a merge into master is practically the same as releasing and suddenly checks start failing that didn't fail before in the other branches.

[chrisgavin]

If JavaScript is configured as language (either automatically detected, or setup manually) but there are not JavaScript files found, the anaylsis fails with an error instead of simply finding no issues which I would expect.

This is a good observation, and something we have discussed in the past. The main reason we have chosen for it to fail rather than reporting no results is to prevent cases where some may have configured Code Scanning incorrectly and are not actually analyzing any code. This is particularly a problem with compiled languages like Java where things like a checked in build directory can mean you accidentally don't build any code in your build step.

The language detection is based on the files in the default branch. But this might not be optimal. If the CodeQL analysis runs on a branch or PR, then it should detect in that branch what languages are present.

Yes, you are correct that this is currently an issue when the languages in a repository change. This is hopefully fairly rare, but I agree it would be more correct for the language information to come from the branch being analyzed. I've created an internal issue for us to investigate other options for detecting which languages to analyze which may provide better results.

Thank you for your feedback about this.