explicitly call lifecycle stop

[nishantmonu51]

found realtime tasks getting stuck on shutdown even after status being shown as SUCCESS.
On further investigation found that the vm was waiting for all the daemon threads to shutdown.
was able to reproduce this locally, calling lifecycle.shutdown explicitly and not relying on the shutdown hook fixes the issue.

[nishantmonu51]

@drcrallen , as you are most familiar with this, can you also double check this change ?

[guobingkun]

👍
I was able to reproduce the problem on our build pipeline. Specifically ITRealtimeIndexTaskTest was not shut down after firehose shuts down. I can still see the peon process from ps -ef

[guobingkun]

Here is the task log https://gist.github.com/guobingkun/6025d9b0afa3edfe8e6e for the realtime task. At 2015-08-13T22:34:22,824, TimedShutoffFirehose was shutting down.

[drcrallen]

@guobingkun Thanks for the log.

@nishantmonu51 Can this add a test in ITRealtimeIndexTaskTest to look for this error case?

[drcrallen]

Before merging this, I would much rather get a thread dump of the offending tasks and see if they can be unstuck.

[fjy]

👍

[himanshug]

+1
bcoz, I think this adds more surety to shutdown process

[himanshug]

also, @guobingkun is going to try this out on our test cluster to see if it fixes the problem we noticed

[drcrallen]

Originally introduced in #1524