[BUG] NRE in OutliersAnalyser

[ig-sinicyn]

BDN, master branch, OutliersAnalyser.cs

            var actual = report.AllMeasurements.Where(m => m.IterationMode == IterationMode.MainTarget).ToArray();
            var result = report.AllMeasurements.Where(m => m.IterationMode == IterationMode.Result).ToArray();
            var actualOutliers = actual.GetStatistics().Outliers;

the report contains only measurements with m.IterationMode == IterationMode.Result
(dont know why, will investigate and update the issue).

the line

var actualOutliers = actual.GetStatistics().Outliers;

fails with NRE as .GetStatistics() returns null.

I'd never return null from any .GetSomeComplexObject() method. These should either return empty value or be prefixed with Try and marked with [CanBeNull], this issue shows why :)

[ig-sinicyn]

UPD: missing reports was my fault, sorry :)

Forgot to dump the Target/Warmup times from my custom Engine.

[AndreyAkinshin]

Ok, but it's still a bug, I will fix it today.

[ig-sinicyn]

UPD2 Okay, it's kinda complicated now.:)

My code uses same approach as in BenchmarkProgram.txt.
results are reported via RunResults.Print();

and (RunResults.cs)

       // TODO: rewrite without allocations
        public IEnumerable<Measurement> GetMeasurements()
        {
            foreach (var measurement in Main)
            {
...
                yield return new Measurement(
                    measurement.LaunchIndex,
                    IterationMode.Result,
                    ++resultIndex,
                    measurement.Operations,
                    value);
            }
        }

        public void Print()
        {
            foreach (var measurement in GetMeasurements())
                WriteLine(measurement.ToOutputLine());
            WriteLine();
        }

So it turns out the Print() does only a half of work.

Rest of timings are dumped at Engine.RunIteration():

            if (!IsDiagnoserAttached) WriteLine(measurement.ToOutputLine());

Summary:

1. bug is still there, with IsDiagnoserAttached it'll reproduce on standard engine too.
2. engine should not take responsibility of submitting the results as it's the thing very easy to break and very hard to debug.

[adamsitnik]

So it turns out the Print() does only a half of work.
Rest of timings are dumped at Engine.RunIteration():

I am not sure about that. IMHO this:

if (!IsDiagnoserAttached) WriteLine(measurement.ToOutputLine());

prints the results during execution to show that Engine is working. This is ommited when Diagnoser is attached in order to avoid heap allocations with creating string and printing.

results.Print() prints all results one more time at the end. The diagnoser is not catching this allocations because it's already detached

[ig-sinicyn]

@adamsitnik
As it is for now, results.Print() do not print anything but IterationMode.Result timings. Proof.

If the measurements will be printed twice it'll screw the results as the Executor parses console output and fills the Summary.Reports from it. Proofs: capture from console, Parsing.

I'll create separate issue for that.