Comparative Analysis of Source Based vs Binary Based Race Detection in Multi- Threaded Applications
Date Issued
2026-07-07
Author(s)
Shirinbayli, Vusal
ADA University
Abstract
Concurrent software is now standard in modern systems, but data races remain one of the most difficult bugs to detect and debug. In practice, two main approaches are used for dynamic race detection in C and C++: compile-time source instrumentation (as in ThreadSanitizer) and dynamic binary instrumentation (as in Valgrind Helgrind). Although both are widely used, there is little controlled empirical evidence comparing their behaviors across different workloads, and tool choice is often based on convention rather than analysis. To address this gap, this thesis empirically compares source-based and binary-based dynamic race detection through a controlled evaluation of ThreadSanitizer and Valgrind Helgrind. The evaluation pairs 208 DataRaceBench benchmarks with a purpose-built suite of 38 synthetic programs designed to isolate specific algorithmic vulnerabilities. This dual-suite methodology is then analyzed using statistical tests, effect size measures, and regression modeling to characterize overhead scaling under varying conditions. Across both suites, a consistent trade-off emerges between precision and recall. ThreadSanitizer shows higher precision, while Helgrind generally achieves higher recall. On DataRaceBench, ThreadSanitizer reaches 57.8% precision and 81.0% recall, while Helgrind
reaches 50.1% precision and 92.0% recall. In the synthetic benchmarks, the differences become more pronounced: Helgrind frequently reports false positives in cases involving non-mutex synchronization, while ThreadSanitizer misses some schedule-dependent races that align with
known theoretical limitations. Category D results also exposed a previously unreported Helgrind false negative case not predicted by standard lockset theory, where consistently unprotected variables fail to trigger the expected lockset inconsistency signal.
Runtime overhead analysis showed opposite scaling behavior for the two tools. ThreadSanitizer stayed stable across the full range of 2 to 32 threads, increasing by less than 50 milliseconds overall. Helgrind, in contrast, tripled its runtime, with performance crossing over around eight threads. Memory usage remained clearly separated in all cases, with a rank- biserial correlation of 1.000 and Helgrind consistently using about 3 to 4 times more memory than ThreadSanitizer. The study also identifies failure cases where each tool breaks down. ThreadSanitizer can time out or degrade under heavy instrumentation in some parallel workloads. Helgrind, in contrast, may fail to report issues in certain cases without clear indication of error, which can make silent failures difficult to detect in automated testing. Overall, the findings form a failure taxonomy organized by detection algorithm, along with a practical decision framework based on experimental evidence. The framework identifies ThreadSanitizer as the appropriate default for lock-free codebases and continuous integration pipelines, and Helgrind as the better choice for safety-critical auditing on mutex-based systems at lower thread counts. In high-stakes environments where missing a race condition is unacceptable, it recommends a dual-tool strategy that uses both tools together.
reaches 50.1% precision and 92.0% recall. In the synthetic benchmarks, the differences become more pronounced: Helgrind frequently reports false positives in cases involving non-mutex synchronization, while ThreadSanitizer misses some schedule-dependent races that align with
known theoretical limitations. Category D results also exposed a previously unreported Helgrind false negative case not predicted by standard lockset theory, where consistently unprotected variables fail to trigger the expected lockset inconsistency signal.
Runtime overhead analysis showed opposite scaling behavior for the two tools. ThreadSanitizer stayed stable across the full range of 2 to 32 threads, increasing by less than 50 milliseconds overall. Helgrind, in contrast, tripled its runtime, with performance crossing over around eight threads. Memory usage remained clearly separated in all cases, with a rank- biserial correlation of 1.000 and Helgrind consistently using about 3 to 4 times more memory than ThreadSanitizer. The study also identifies failure cases where each tool breaks down. ThreadSanitizer can time out or degrade under heavy instrumentation in some parallel workloads. Helgrind, in contrast, may fail to report issues in certain cases without clear indication of error, which can make silent failures difficult to detect in automated testing. Overall, the findings form a failure taxonomy organized by detection algorithm, along with a practical decision framework based on experimental evidence. The framework identifies ThreadSanitizer as the appropriate default for lock-free codebases and continuous integration pipelines, and Helgrind as the better choice for safety-critical auditing on mutex-based systems at lower thread counts. In high-stakes environments where missing a race condition is unacceptable, it recommends a dual-tool strategy that uses both tools together.