Where are the published CI build logs, specifically for fsharpqa?

[abelbraaksma]

Since it's very hard to run tests locally (I've figured it out and will write up the results soon, but also since HostedCompilerService.exe keeps running after closing VS, keeping files locked), and since running them takes a lot of time, it would be great if we could download the logs with the errors from the CI results.

The logs are published (see, for instance, this: https://dev.azure.com/dnceng/public/_build/results?buildId=755330&view=logs&j=f63a89e5-f92e-5427-3260-3ba3a585220c&t=a75259a2-1178-52d5-2540-928aa6fdbf8d)

But when you go to a CI node, click "artifacts" and then click the necessary download, nothing happens: it is not downloaded. See https://dev.azure.com/dnceng/public/_build/results?buildId=755330&view=artifacts&type=publishedArtifacts

The regular log does not contain the error output, it only shows "failed" for fsharpqa tests, which, in my case, leads to "guessing what the test expects"... (add confused smiley here).

#9779

[cartermp]

You can download logs from the top-left of the CI portal:

However there are no diagnostic logs published for the fsharpqa tests as far as I can tell. This is one of the reasons why they're so awful :)

Perhaps there's a way to publish them but since it's a bespoke framework instead of a standardized one like NUnit or xUnit, it's unlikely that most systems know how to properly display logs unless it's just some plaintext.

[abelbraaksma]

However there are no diagnostic logs published for the fsharpqa tests as far as I can tell. This is one of the reasons why they're so awful :)

@cartermp, Indeed ;). I thought this was the key:

And then this:

But nothing gets downloaded when I click that.

Locally, an error log is created, though, but a bunch of tests don't run, or fail horribly with assert-popup windows, leading to many false negatives.

[abelbraaksma]

In the build.ps1 it has this:

        $resultsLog = "test-net40-fsharpqa-results.log"
        $errorLog = "test-net40-fsharpqa-errors.log"
        $failLog = "test-net40-fsharpqa-errors"

[abelbraaksma]

Looks like the following command (and actions) gets this running locally:

• Close all VS instances
• Close, from Task Mgr, or kill, HostedCompilerService.exe and/or VBSCompiler.exe. You may also kill any hanging watson processes
• Run this from the root of the repo: build -testFSharpQA -ci -bl -c Release
  • Note that without -c Release, you'll get a bunch of more errors, false negatives, and assertion-popups
  • Without closing processes, it may not compile, or errors/crashes and popups half way
  • Without -ci -bl the tests won't run properly at all
• It may appear as if the process hangs after HexCharEncode.fs, but the next test just takes 4-6 min to complete

After this, the log file [basedir]artifacts\TestResults\Release contains the log information you need. It may contain a lot of empty lines, but just scroll through...

Many of the tests here will have the following lines in the error-log, which you should ignore and not try to fix:

*** The following necessary lines were incorrectly matched:

Unexpected Compiler Output 
FAIL

[dsyme]

I've always struggled to find the logs for fsharpqa too

[abelbraaksma]

@dsyme I don't know how long it may take to get these converted to the new test system, but if it is still several months away, a simple tweak to CI would be to just upload these auxiliary artifacts, even though they aren't binary logs. They are there, but cannot be accessed right now.

[abelbraaksma]

@dsyme, @cartermp, I suddenly found out where these logs are (after I found out that the azure-pipelines.fsy were correct). It's one of those things that once you know it, it's easy to find. I made an edited gif-video tutorial, we can add it to the contributors instruction:

@pblasucci, you struggled with this in your most recent PR as well. Now you know where to access them, even if you cannot build them locally.

[cartermp]

Thanks! Could you add a note about this to the TESTGUIDE.md file? This will eventually not be an issue as these tests get migrated, but in the meantim eit would be good to point people to that.

[abelbraaksma]

@cartermp, pr is here: #10018 (still some things to work out, but almost finished).