MPI spinning consumes excessive battery and interferes with other apps

[m8pple]

I'd noticed that trying to do any work with the orchestrator killed my laptop battery, to
the point that I'd mostly stopped trying. Then I accidentally left the orchestrator open,
and realised (after it had burnt half my battery), that it was the spinning threads
that were running permanently even if the orchestrator was idle.

Many users of the orchestrator are going to be developing applications and spending
most of their time getting them compiling and parsing (and eventually simulating), and
they are quite likely to be doing this on laptops. Every student we have had working
with POETS so far has worked on their laptop, and only connected to a hardware server once
it was all debugged and working in simulation.

So it would be nice if the spinner adapted if nothing is happening, e.g. some kind of
exponential backoff down to 10ms if there is repeatedly nothing there. I can see
something mentioned in the Mothership spinner, but this seems to be related to
debugging rather than efficiency.

Presumably this would be added here:

Orchestrator/Source/Common/CommonBase.cpp

Lines 101 to 109 in 372ccc2

	for (;;) {
	// See if there are any MPI packets coming down the pipe
	MPI_Status status; // Note the multi-threaded MPI probe
	int flag;
	MPI_Iprobe(MPI_ANY_SOURCE,MPI_ANY_TAG,MPI_COMM_WORLD,&flag,&status);
	if (flag==0) { // Nothing there....
	OnIdle(); // Guess
	continue; // And try again
	}

[m8pple]

There seems to be an API issue in that the OnIdle method cannot tell the MPISpinner code that it has idle
code to do, or that even if it did do something that it would like the opportunity to do something again.
So I guess there is no way for MPISpinner to know whether it is wasting its time or if the OnIdle method
is actually driving something.

[m8pple]

Suggested fix for two threads that spin is in dt10_idle_work_reduction

[mvousden]

I like this - something I was meaning to get around to at some point, but never designed properly.