Matching failed to compile

[felixhekhorn]

Running eko resulted in

2022-05-19 17:58:25,999 eko.evolution_operator/INFO: Evolution: computing operators 2.722500e+00 -> 1.000000e+01, nf=4
2022-05-19 17:58:25,999 eko.evolution_operator/INFO: Evolution: µ_R^2 distance: 6.806250e-01 -> 2.500000e+00
2022-05-19 17:58:25,999 eko.evolution_operator/INFO: Scale Variation: (µ_F/µ_R)^2 = 4.000000e+00, mode: expanded
2022-05-19 17:58:27,223 eko.evolution_operator/INFO: Evolution: a_s distance: 2.494975e-02 -> 1.910111e-02
2022-05-19 17:58:27,223 eko.evolution_operator/INFO: Evolution: order: 1, solution strategy: truncated
2022-05-19 18:00:52,979 eko.evolution_operator/INFO: Evolution: computing operators: - 9/50 took: 145.698243 s
2022-05-19 18:00:53,748 eko.evolution_operator/INFO: Evolution: computing operators: - 15/50 took: 146.446427 s
2022-05-19 18:00:54,008 eko.evolution_operator/INFO: Evolution: computing operators: - 5/50 took: 146.740913 s
2022-05-19 18:00:54,355 eko.evolution_operator/INFO: Evolution: computing operators: - 3/50 took: 147.093093 s
2022-05-19 18:00:54,446 eko.evolution_operator/INFO: Evolution: computing operators: - 13/50 took: 147.152215 s
2022-05-19 18:00:54,620 eko.evolution_operator/INFO: Evolution: computing operators: - 1/50 took: 147.362054 s
2022-05-19 18:00:54,905 eko.evolution_operator/INFO: Evolution: computing operators: - 11/50 took: 147.619450 s
2022-05-19 18:00:56,091 eko.evolution_operator/INFO: Evolution: computing operators: - 7/50 took: 148.816877 s
2022-05-19 18:00:56,770 eko.evolution_operator/INFO: Evolution: computing operators: - 10/50 took: 3.791091 s
2022-05-19 18:00:57,047 eko.evolution_operator/INFO: Evolution: computing operators: - 16/50 took: 3.298314 s
2022-05-19 18:00:58,106 eko.evolution_operator/INFO: Evolution: computing operators: - 14/50 took: 3.660176 s
2022-05-19 18:00:58,106 eko.evolution_operator/INFO: Evolution: computing operators: - 6/50 took: 4.098238 s
...
2022-05-19 18:01:07,163 eko.evolution_operator/INFO: Evolution: Total time 159.940010 s

which is fine - because he needs to compile on each core after that its just normal; after that it is quick:

2022-05-19 18:01:07,234 eko.evolution_operator/INFO: Evolution: computing operators 2.722500e+00 -> 2.000000e+01, nf=4
...
2022-05-19 18:01:25,740 eko.evolution_operator/INFO: Evolution: Total time 18.505267 s

but then I reached the threshold and I get

2022-05-19 18:01:25,809 eko.evolution_operator/INFO: Evolution: computing operators 2.722500e+00 -> 2.420640e+01, nf=4
...
2022-05-19 18:01:44,208 eko.evolution_operator/INFO: Evolution: Total time 18.397738 s

and after that nothing is happening for 45min (but the CPU running at full power) when I decided to kill the process

I haven't investigated yet - @giacomomagni @alecandido any ideas?

[alecandido]

Nope, no real idea, I'm lacking details. Try to accumulate more information, if possible (like where is actually stopped, and things like this).

Just a stupid observation @giacomomagni can quickly reply to: we don't have any shared resources, do we?
Simply because a typical problem of parallel computation is a deadlock, that would have exactly this phenomenology, but I consider it unlikely.

In any case @felixhekhorn have you tried to disable parallel mode, and run on a single core? If it's a feature of parallelism this way we can detect, otherwise it's a simple check that will rule out this annoying chance.

[felixhekhorn]

Wait a second - I killed the process and I believe I killed the parent one, but apparently no! the log continued just now:

2022-05-19 18:56:31,499 eko.evolution_operator/INFO: Matching: computing operators: - 3/50 took: 3287.242057 s
2022-05-19 18:56:34,868 eko.evolution_operator/INFO: Matching: computing operators: - 1/50 took: 3290.608216 s
2022-05-19 18:56:38,921 eko.evolution_operator/INFO: Matching: computing operators: - 4/50 took: 7.420076 s
2022-05-19 18:56:40,460 eko.evolution_operator/INFO: Matching: computing operators: - 7/50 took: 3296.180899 s
2022-05-19 18:56:42,673 eko.evolution_operator/INFO: Matching: computing operators: - 2/50 took: 7.805148 s
2022-05-19 18:56:43,680 eko.evolution_operator/INFO: Matching: computing operators: - 17/50 took: 4.726941 s
2022-05-19 18:56:46,674 eko.evolution_operator/INFO: Matching: computing operators: - 19/50 took: 3.959325 s
2022-05-19 18:56:47,718 eko.evolution_operator/INFO: Matching: computing operators: - 8/50 took: 7.257330 s
2022-05-19 18:56:48,084 eko.evolution_operator/INFO: Matching: computing operators: - 18/50 took: 4.403091 s
2022-05-19 18:56:50,439 eko.evolution_operator/INFO: Matching: computing operators: - 20/50 took: 3.765395 s
2022-05-19 18:56:51,472 eko.evolution_operator/INFO: Matching: computing operators: - 23/50 took: 3.370410 s
2022-05-19 18:56:51,659 eko.evolution_operator/INFO: Matching: computing operators: - 21/50 took: 3.901861 s
2022-05-19 18:56:53,108 eko.evolution_operator/INFO: Matching: computing operators: - 11/50 took: 3308.826739 s
...
2022-05-19 18:57:11,571 eko.evolution_operator/INFO: Matching: Total time 3327.356147 s
2022-05-19 18:57:11,583 eko.evolution_operator/INFO: Evolution: computing operators 2.420640e+01 -> 2.800000e+01, nf=5
2022-05-19 18:57:11,583 eko.evolution_operator/INFO: Evolution: µ_R^2 distance: 6.051600e+00 -> 7.000000e+00
2022-05-19 18:57:11,584 eko.evolution_operator/INFO: Scale Variation: (µ_F/µ_R)^2 = 4.000000e+00, mode: expanded
2022-05-19 18:57:11,586 eko.evolution_operator/INFO: Evolution: a_s distance: 1.692281e-02 -> 1.658675e-02
2022-05-19 18:57:11,586 eko.evolution_operator/INFO: Evolution: order: 1, solution strategy: truncated
2022-05-19 18:57:15,074 eko.evolution_operator/INFO: Evolution: computing operators: - 15/50 took: 3.410720 s
2022-05-19 18:57:15,165 eko.evolution_operator/INFO: Evolution: computing operators: - 13/50 took: 3.507786 s
2022-05-19 18:57:15,390 eko.evolution_operator/INFO: Evolution: computing operators: - 11/50 took: 3.737253 s

there is something fishy ...

[alecandido]

Now I'm more concerned about parallel execution

[giacomomagni]

Hi @felixhekhorn , I don't have any clue why this happened... do you have something we can use to reproduce the bug?

[felixhekhorn]

• Do we really want this for 0.9.2? meaning blocking the referee reply?
• I haven't much of a clue yet - I'm even more puzzled
• I computed some ekos together with @andreab1997 forgetting about this - at it worked; I now suspect that in my first debug attempt I did revert to 0.8 and we were using that one (so @andreab1997 I think what we saw was not using the fixed version)
• I also have a suspicion about numba - because he tried to compile N3LO matching although I'm asking for an NLO eko, so he's not doing JIT ... moreover numba produced a bunch of strange messages (some of which contained the string "error"), but since it is quite tough to debug, I can only guess at the moment; and my guess is that something with the automatic argument type detection is going wrong
• tomorrow, I'll try to cut some stuff out of eko ...

just to share some of the fun with you 🙃

call void @PyErr_SetString(i8* nonnull @PyExc_RuntimeError, i8* getelementptr inbounds ([254 x i8], [254 x i8]* @".const.missing Environment: _ZN08NumbaEnv3eko18evolution_operator11QuadKerBase12__init___243B42c8tJTIeFCjyCbUFRqqOAK_2f6h0kCng1maAA_3d_3dEN8instance8jitclass96QuadKerBase_237f635aea4820_3cis_singlet_3abool_2cis_log_3abool_2clogx_3afloat64_2cu_3afloat64_3eEdbdx", i64 0, i64 0))
  ret i8* null
...
call void @PyErr_SetString(i8* nonnull @PyExc_SystemError, i8* getelementptr inbounds ([43 x i8], [43 x i8]* @".const.unknown error when calling native function", i64 0, i64 0))
  br label %entry.if
...
.const.pickledata.140064608552896:
	.ascii	"\200\004\225;\000\000\000\000\000\000\000\214\bbuiltins\224\214\nIndexError\224\223\224\214\030tuple index out of range\224\205\224N\207\224."
	.size	.const.pickledata.140064608552896, 70

[andreab1997]

• Do we really want this for 0.9.2? meaning blocking the referee reply?
• I haven't much of a clue yet - I'm even more puzzled
• I computed some ekos together with @andreab1997 forgetting about this - at it worked; I now suspect that in my first debug attempt I did revert to 0.8 and we were using that one (so @andreab1997 I think what we saw was not using the fixed version)
• I also have a suspicion about numba - because he tried to compile N3LO matching although I'm asking for an NLO eko, so he's not doing JIT ... moreover numba produced a bunch of strange messages (some of which contained the string "error"), but since it is quite tough to debug, I can only guess at the moment; and my guess is that something with the automatic argument type detection is going wrong
• tomorrow, I'll try to cut some stuff out of eko ...

just to share some of the fun with you 🙃

So is still possible that the solution we tried was correct and today we were looking at the wrong results?

[alecandido]

.const.pickledata.140064608552896:
	.ascii	"\200\004\225;\000\000\000\000\000\000\000\214\bbuiltins\224\214\nIndexError\224\223\224\214\030tuple index out of range\224\205\224N\207\224."
	.size	.const.pickledata.140064608552896, 70

This is clearly numba pickling some compiled routine.

[giacomomagni]

Hi @felixhekhorn, thanks for taking care of this.
I believe that something that can fix these issues about the N3LO matching would be to produce a parametrised expression of A_qqNS which is the only one including the analytical weight=5 harmonics sums and the functions contained in the eko.harmonics.f_functions module.
see here

For sure, it will speed up the compilation a lot.
I will work on it as soon as I have time.

Edit: compiling the weight=3 and weight=4 harmonics shouldn't take (g_functions) that long

[alecandido]

For sure, it will speed up the compilation a lot.

This would be wonderful.

I will work on it as soon as I have time.

No need to rush: matching have been useful for IC, but that's done, the next step it is only N3LO evolution.
So, you can take the time it is needed to complete the anomalous dimensions interpolation. Once that is done (completed or to be reviewed together), then if you can dedicate some time to speed matching would be really appreciated.

(Still keep in mind that for N3LO there is still yadism to be implemented, and the error propagation at this point)