C++ Mutex Deadlock Detection with Callstack

February 18, 2014

There are tons of ways of going about debugging deadlocks in C++. This is an exercise and example of implementing a solution at the code level. This implements a special mutex type that detects deadlock based on a timeout and proceeds to print out the callstack of the offending threads. It allows you to troubleshoot and pinpoint where a deadlock occurs. The source can be found on GitHub.

$ ./callstack 
running thread 1 running thread 2 140602880931584140602872538880
 
Failed to lock in Failed to lock in 140602872538880140602880931584 locked by 140602880931584 locked by 140602880931584
Backtrace for 140602880931584
 
Backtrace for 140602880931584
_callstack_signal_handler, self: 140602880931584, target: 140602880931584, caller: 140602872538880
./callstack(_Z15print_backtracem+0x5e) [0x401f80]
./callstack(_ZN6CMutex9timedlockERK8timespec+0x10d) [0x402421]
./callstack(_Z7thread1Pv+0x66) [0x4019ea]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd]
 
./callstack() [0x401c7f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fe0a9327cb0]
/lib/x86_64-linux-gnu/libgcc_s.so.1(+0xed6b) [0x7fe0a9543d6b]
/lib/x86_64-linux-gnu/libgcc_s.so.1(+0xf490) [0x7fe0a9544490]
/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Backtrace+0x2e) [0x7fe0a9544d3e]
/lib/x86_64-linux-gnu/libc.so.6(backtrace+0x58) [0x7fe0a9063108]
./callstack(_Z12GetCallstackmPPvi+0x4f) [0x401de4]
./callstack(_Z15print_backtracem+0x5e) [0x401f80]
./callstack(_ZN6CMutex9timedlockERK8timespec+0x10d) [0x402421]
./callstack(_Z7thread1Pv+0x66) [0x4019ea]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd]
 
Failed to lock in 140602880931584 locked by 140602880931584
Backtrace for 140602880931584
./callstack(_Z15print_backtracem+0x5e) [0x401f80]
./callstack(_ZN6CMutex9timedlockERK8timespec+0x10d) [0x402421]
./callstack(_Z7thread1Pv+0x66) [0x4019ea]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd]
 
Failed to lock in 140602872538880 locked by 140602880931584
Backtrace for 140602880931584
_callstack_signal_handler, self: 140602880931584, target: 140602880931584, caller: 140602872538880
./callstack() [0x401c7f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fe0a9327cb0]
[0x7fff7d9ff707]
[0x7fff7d9ff7e5]
/lib/x86_64-linux-gnu/librt.so.1(clock_gettime+0x1d) [0x7fe0a9a4f15d]
./callstack(_ZN6CMutex9timedlockERK8timespec+0x5a) [0x40236e]
./callstack(_Z7thread1Pv+0x66) [0x4019ea]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd]

Related Posts