Lock continuation before calling event handler.

[shinrich]

After further deploying builds with the asserts in handleEvent we have seen several variants of the following stack. In the cases I've dug into the continuation called from reply_to_cont is a HttpSM. And indeed it is unlocked.

Grabbing the lock at the top of probeEvent and rescheduling if either lock cannot be obtained should solve the problem. To simply the lock scope problem, we get the action.mutex lock again if the continuation lock is not present. Should only be a compare and increment overhead.

 0  ] libc-2.17.so       __GI_raise                                                   ( :undefined         ) 
[ 1  ] libc-2.17.so       __GI_abort                                                   ( :undefined         ) 
[ 2  ] libtsutil.so.7     ink_abort                                                    ( ink_error.cc:99    ) 
[ 3  ] libtsutil.so.7     _ink_assert                                                  ( ink_assert.cc:37   ) 
[ 4  ] traffic_server     Continuation::handleEvent(int, void*)                        ( Continuation.cc:32 ) 
[ 5  ] traffic_server     reply_to_cont                                                ( HostDB.cc:491      ) 
[ 6  ] traffic_server     HostDBContinuation::probeEvent(int, Event*)                  ( HostDB.cc:1736     ) 
[ 7  ] traffic_server     HostDBContinuation::dnsPendingEvent(int, Event*)             ( HostDB.cc:1193     ) 
[ 8  ] traffic_server     Continuation::handleEvent(int, void*)                        ( Continuation.cc:33 ) 
[ 9  ] traffic_server     HostDBContinuation::remove_trigger_pending_dns()             ( HostDB.cc:1793     ) 
[ 10 ] traffic_server     HostDBContinuation::dnsEvent(int, HostEnt*)                  ( HostDB.cc:1487     ) 
[ 11 ] traffic_server     Continuation::handleEvent(int, void*)                        ( Continuation.cc:33 ) 
[ 12 ] traffic_server     DNSEntry::postEvent(int, Event*)                             ( DNS.cc:1282        ) 
[ 13 ] traffic_server     Continuation::handleEvent(int, void*)                        ( Continuation.cc:33 ) 
[ 14 ] traffic_server     EThread::process_event(Event*, int)                          ( UnixEThread.cc:132 ) 
[ 15 ] traffic_server     EThread::process_queue(Queue<Event, Event::Link_link>*, i... ( UnixEThread.cc:171 ) 
[ 16 ] traffic_server     EThread::execute_regular()                                   ( UnixEThread.cc:231 ) 
[ 17 ] traffic_server     EThread::execute()                                           ( UnixEThread.cc:326 ) 
[ 18 ] traffic_server     spawn_thread_internal                                        ( Thread.cc:85       ) 
[ 19 ] libpthread-2.17.so start_thread                                                 ( :undefined         )

[vmamidi]

when EThread::process_event(Event*, int) is called, it is not acquiring the HttpSM mutex? DNS/HostDB mutex should be same as that of HttpSM, right ?

[SolidWallOfCode]

No. The lock in HostDBContinuation should be the lock on the bucket in which the HostDB record resides, so that the continuation can updated it safely.

[shinrich]

In any case, this stack has the HttpSM mutex not held because the assert went off.

[vmamidi]

it makes sense to hold the bucket mutex. If the theory is right, we should be able to reproduce the scenario as it should crash if DNS takes longer than transaction time.

[bryancall]

Does this only happen with the global session pool? I was talking to @zwoop about this earlier today.

[shinrich]

I have only tested with the global pool environment.

[bryancall]

Cherry picked to 8.1.0