SIGSEGV on Vol::force_evacuate_head

[masaori335]

Backtrace

[ 00 ] libpthread-2.17.so  waitpid                                                              
[ 01 ] traffic_server      crash_logger_invoke(int, siginfo_t*, void*)                          ( Crash.cc:168 )
[ 02 ] libpthread-2.17.so                                                                       
[ 03 ] traffic_server      Vol::force_evacuate_head(Dir*, int)                                  ( P_CacheVol.h:461 )
[ 04 ] traffic_server      CacheVC::openReadClose(int, Event*)                                  ( CacheRead.cc:542 )
[ 05 ] traffic_server      CacheVC::calluser(int)                                               ( P_CacheInternal.h:705 )
[ 06 ] traffic_server      CacheVC::openReadMain(int, Event*)                                   ( CacheRead.cc )
[ 07 ] traffic_server      CacheVC::openReadReadDone(int, Event*)                               ( CacheRead.cc:639 )
[ 08 ] traffic_server      CacheVC::handleReadDone(int, Event*)                                 ( eventsystem/I_Continuation.h:219 )
[ 09 ] traffic_server      CacheVC::openReadMain(int, Event*)                                   ( eventsystem/I_Continuation.h:219 )
[ 10 ] traffic_server      EThread::process_event(Event*, int)                                  ( I_Continuation.h:219 )
[ 11 ] traffic_server      EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*)  ( UnixEThread.cc:162 )
[ 12 ] traffic_server      EThread::execute_regular()                                           ( UnixEThread.cc:222 )
[ 13 ] traffic_server      EThread::execute()                                                   ( UnixEThread.cc:327 )
[ 14 ] traffic_server      spawn_thread_internal(void*) [clone .llvm.4297563414356737819]       ( Thread.cc:92 )
[ 15 ] libpthread-2.17.so  start_thread                                                         

Affected Versions

We faced this crash with 8.0.x, 8.1.x, and 9.0.x intermittently.

[cmcfarlen]

Looking at the core file:

(gdb) thread 38
[Switching to thread 38 (Thread 0x7fdde0796700 (LWP 19126))]
#0  0x00007fdde76121d9 in waitpid () from /lib64/libpthread.so.0
(gdb) fr 3
#3  evacuation_block_exists (dir=0x7f9d8edaa9e6, p=0x7fdddb2fa000) at ./P_CacheVol.h:461
461         if (dir_offset(&b->dir) == dir_offset(dir))
(gdb) info locals
b = 0xea326b5d0d1ef237
(gdb) p *b
Cannot access memory at address 0xea326b5d0d1ef237
(gdb) li
456     TS_INLINE EvacuationBlock *
457     evacuation_block_exists(Dir *dir, Vol *p)
458     {
459       EvacuationBlock *b = p->evacuate[dir_evac_bucket(dir)].head;
460       for (; b; b = b->link.next)
461         if (dir_offset(&b->dir) == dir_offset(dir))
462           return b;
463       return nullptr;
464     }
465

The local variable b holds an invalid pointer so this crashes when dereferencing b on line 161.

line 459 assumes dir_evac_bucket returns a valid index into p->evacuate. This update adds a bounds check around this.

Further analysis on this scenario:

(gdb) p p->evacuate_size
$1 = 6442
(gdb) x/5xh dir
0x7f9d8edaa9e6: 0x0412  0x3d77  0x338f  0x0000  0x0018

The Dir type is an array of 5 uint16_t. Plugging these values into a unit test and calling the dir_evac_bucket macro yields 12526 which is outside of the bounds of p->evacuate_size.