Full Report
Many security bugs are race conditions, where multi-threaded execution has to occur with the right interleaving for a negative effect to appear. This creates challenges for several use cases: Confirming bug candidates that have been discovered manually or through static analysis. Regression tests: After fixing a race condition bug, there is often no good way to write a regression test that reliably triggers the bug as part of a test suite. Automatic bug discovery, such as fuzzing: It is hard for a fuzzer to exercise all interesting interleavings of concurrent operations, or reach code paths that are only exercised when operations are racing.
Analysis Summary
# Research: Testing race conditions with memory access tracing and stack-based delay injection
## Metadata
- **Authors:** Jann Horn
- **Institution:** Google Project Zero
- **Publication:** Project Zero Technical Blog
- **Date:** September 8, 2026 (Projected/Document Date)
## Abstract
This research addresses the inherent difficulty of identifying, reproducing, and fixing race condition vulnerabilities in multi-threaded environments, specifically the Linux kernel. The author introduces **MAccConc** (Memory Access Concurrency), a suite of tools designed to explore execution interleavings by tracing memory accesses and using stack-based delay injections. Unlike traditional methods that rely on manual `mdelay()` calls or specific hypervisor modifications, MAccConc utilizes compiler-level instrumentation (KASAN) and kernel coverage (KCOV) to identify "communication points" (overlapping memory accesses between threads) and systematically force specific interleavings to expose bugs.
## Research Objective
The research aims to solve three primary challenges in concurrency security:
1. **Bug Confirmation:** Reliably proving a suspected race condition exists.
2. **Regression Testing:** Creating stable tests that consistently trigger a race condition after a fix is applied.
3. **Automatic Discovery:** Enhancing fuzzers to explore code paths that only execute during specific racing conditions.
## Methodology
### Approach
The methodology relies on identifying **Communication Points**—memory accesses in different threads that overlap where at least one is a write.
- **Tracing:** Uses KASAN (Kernel Address Sanitizer) instrumentation in "outline" mode to intercept memory accesses.
- **Identifier Stability:** Addresses the "instruction pointer problem" by using **Count-Augmented Stack Traces**. This uniquely identifies a memory access not just by the code location, but by its call stack and the number of times that specific stack has been seen in the current execution.
- **Delay Injection:** Implements a mechanism to "pause" threads at specific communication points to force A-B-A interleavings or other specific sequences.
### Dataset/Environment
- **Target:** Linux Kernel.
- **Runtime:** Tested using KVM/QEMU environments.
- **Instrumentation:** Integrated with KCOV (Kernel Coverage) for data collection.
### Tools & Technologies
- **MAccConc:** The custom toolset including a terminal UI and GUI for interleaving exploration.
- **KASAN & KCOV:** Linux kernel subsystems used for memory monitoring and coverage reporting.
- **BPF (Berkeley Packet Filter):** Used for efficient data collection in the kernel.
## Key Findings
### Primary Results
1. **Deterministic Reproduction:** The tool successfully automates the testing of A-B-A interleavings, making previously "flaky" race conditions reliably reproducible.
2. **Stable Access Identification:** Count-augmented stack traces proved effective at identifying specific memory accesses across multiple runs, even when absolute memory addresses changed.
3. **Visualization:** Manual exploration via TUI/GUI allows developers to "step through" races similarly to how they draw manual ASCII race diagrams.
### Supporting Evidence
- The research demonstrates the tool's ability to recreate complex Linux kernel vulnerabilities, such as the `rt_spin_unlock` Use-After-Free (UAF) and `jbd2` deadlock, which previously required complex manual diagrams to explain.
### Novel Contributions
- **Count-Augmented Stack Traces:** A novel way to create stable "IDs" for memory accesses that survive across different executions and address space layouts.
- **Software-Only Interleaving Control:** Unlike SKI (which requires QEMU patches), MAccConc works within the standard Linux kernel infrastructure using KASAN/KCOV.
## Technical Details
The core innovation is the **Stack-Based Delay Injection**. When the kernel hits a designated "communication point," the tool checks the current stack trace and a per-thread counter. If it matches a pre-defined target, the thread enters a controlled delay. This allows a second thread to "overtake" the first at a precise moment in the execution flow. The identification string looks like: `[hash_of_stack_trace]:[occurrence_count]`.
## Practical Implications
### For Security Practitioners
- Provides a structured way to verify "won't fix" or "could not reproduce" bug reports.
- Reduces the reliance on "spray and pray" techniques for triggering races during exploit development.
### For Defenders
- Enables the creation of high-fidelity regression tests. If a race is fixed, a MAccConc script can ensure that the specific interleaving no longer results in a crash or inconsistent state.
### For Researchers
- Offers a blueprint for building "concurrency-aware" fuzzers that don't just hunt for new code paths, but for new *interleavings* of existing paths.
## Limitations
- **Overhead:** Heavy instrumentation (KASAN outline mode) significantly slows down execution.
- **Complexity:** Requires deep knowledge of the target code to identify which memory accesses are relevant for manual exploration.
- **Kernel Panics:** Current KCOV implementation loses data if the kernel crashes before the buffer is flushed (a potential fix involves virtiofs with DAX).
## Comparison to Prior Work
- **vs. SKI:** SKI uses QEMU snapshots and TCG mode, making it powerful but harder to set up. MAccConc is more "native" to the Linux kernel.
- **vs. DTrace/chill():** DTrace is limited to function boundaries. MAccConc can trigger on *any* instrumented memory access (instruction-level granularity).
## Real-world Applications
- **Kernel Patch Validation:** Ensuring that lock primitives actually protect the intended critical sections.
- **CI/CD Integration:** Running race-prone test cases in a specialized CI environment that cycles through different interleavings.
## Future Work
- **Deadlock Detection Integration:** Speeding up tests by detecting when a forced interleaving has caused a logical deadlock.
- **Fuzzing Integration:** Building a "Snowboard-like" system to identify syscalls that share communication points and fuzzing them in parallel.
- **Host-Shared KCOV:** Moving the trace buffer to host memory to preserve data during a hard kernel crash.
## References
- SKI: *Exposing Kernel Concurrency Bugs through Systematic Schedule Exploration* (OSDI '14).
- Snowboard: *Finding Kernel Concurrency Bugs via Systematic Inter-thread Communication Analysis* (SOSP '21).
- GitHub Repository: `googleprojectzero/MAccConc` (Defanged: hxxps://github[.]com/googleprojectzero/MAccConc)