Inside Tachyon

Each sample begins with a read of another process’s memory. The operating system decides who may do that, so the first step depends on the platform. Tachyon does not stop the process to read it. On Linux it uses process_vm_readv, which copies a list of remote memory ranges into local buffers in a single system call.

Linux

process_vm_readvNeeds root, CAP_SYS_PTRACE, or a relaxed Yama ptrace_scope. The default scope of 1 only lets a parent trace its own children, which is why run works out of the box while attach often does not.

macOS

task_for_pid(), mach_vm readsNeeds root or the com.apple.security.cs.debugger entitlement. Getting the task port for the target is the part that needs privilege.

Windows

OpenProcess, ReadProcessMemoryNeeds administrator rights or SeDebugPrivilege to open the target with read access.

There is one more constraint, and it is about Python itself. The profiler and the target must run the same minor version of Python. Pre-releases must match exactly. Free-threaded and default builds cannot profile each other. The next section explains why: the memory layouts Tachyon decodes are only described for one exact build.

Every CPython process has one global, _PyRuntime. Every interpreter, thread and frame can be reached from it. A debugger would find it through debug symbols, but release builds are stripped. CPython solves this by placing the runtime in a named binary section, PyRuntime, and putting a description of its own memory layout at the start of that section.

The description is a struct called _Py_DebugOffsets. It begins with the 8-byte cookie "xdebugpy", then the version and a free-threading flag. After those come about a hundred byte offsets: where threads_head lives in an interpreter state, where current_frame lives in a thread state, where previous lives in a frame, and so on. Tachyon hardcodes none of these offsets. It reads the table and uses its values.

/* simplified, from Include/internal/pycore_debug_offsets.h */
typedef struct _Py_DebugOffsets {
    char     cookie[8];       // "xdebugpy"
    uint64_t version;         // 0x030f.... for 3.15
    uint64_t free_threaded;
    // each field below is a uint64_t byte offset
    struct { size; finalizing; interpreters_head; } runtime_state;
    struct { ... threads_head; ... } interpreter_state;
    struct { ... current_frame; base_frame; ...
             native_thread_id; datastack_chunk; ... } thread_state;
    struct { ... previous; executable; instr_ptr; ... } interpreter_frame;
    struct { ... linetable; ... co_code_adaptive; ... } code_object;
    // ... and more
} _Py_DebugOffsets;

/* simplified, from
   pycore_debug_offsets.h */
typedef struct _Py_DebugOffsets {
  char     cookie[8];  // "xdebugpy"
  uint64_t version;    // 0x030f....
  uint64_t free_threaded;
  // the rest are uint64_t byte offsets
  struct {
    size; finalizing; interpreters_head;
  } runtime_state;
  struct {
    ... threads_head; ...
  } interpreter_state;
  struct {
    ... current_frame; base_frame;
    native_thread_id; datastack_chunk; ...
  } thread_state;
  struct {
    ... previous; executable;
    instr_ptr; ...
  } interpreter_frame;
  struct {
    ... linetable; ...
    co_code_adaptive; ...
  } code_object;
  // ... and more
} _Py_DebugOffsets;

The cookie check matters more than it seems. One process can contain two copies of libpython. For example, ctypes can dlopen a second one whose runtime was never initialized. An early version took the first matching mapping. The fix (gh-144563) reads the first eight bytes of each candidate and accepts only one that starts with xdebugpy.

The offsets table comes from the process being profiled, so Tachyon treats it as untrusted input. A shared validator (gh-148178) checks every offset and size when the table is loaded, and malformed tables are rejected. After that the sampling loop can use the offsets directly, with no bounds checks in the hot path.

This table is also why the versions must match. The offsets describe one layout of PyThreadState and _PyInterpreterFrame, and those structs change between minor releases.

From the runtime, Tachyon follows interpreters_head to an interpreter state. From there it follows threads_head into a singly linked list of thread states. For each PyThreadState it reads native_thread_id and then follows next. The walk stops at a null pointer, or after 8,192 threads as a guard against a corrupted list.

Each thread state already records most of what the profiling modes need. Tachyon reads those fields into a small bitfield of status flags for every thread on every sample.

• GIL. On default builds, holds_gil says whether the thread owns the GIL. If it does not and gil_requested is set, the thread is waiting for it. Free-threaded builds have no GIL, so Tachyon reads the active bit of the thread’s status instead.
• Exception. The thread is handling an exception if current_exception is set. Tachyon also follows the exc_info chain to the innermost handler, which is how sys.exception() finds it.
• On CPU. The interpreter does not record this, so Tachyon asks the kernel. On Linux it reads the state letter from /proc/<pid>/task/<tid>/stat, where R means running. This costs a file read, so Tachyon only does it when the mode needs it.
• Main thread. Tachyon compares each thread state with the runtime’s main thread state.

A profiling mode is a rule about which samples to keep. wall keeps every sample, including time spent sleeping or waiting on I/O. cpu keeps samples where the kernel reports the thread as running. gil keeps samples where the thread holds the GIL. exception keeps samples taken while an exception is being raised or handled. Try each one on four simulated threads below.

Comparing modes is often more useful than any one of them. A function that is large under wall and missing under cpu is waiting, not computing. A thread that is high under cpu and low under gil is spending its time in C code that released the GIL. The flame graph’s sidebar shows the same split as GIL held, released and waiting percentages.

Since Python 3.11, most frames are not heap objects. A function call pushes a _PyInterpreterFrame onto a per-thread data stack. The data stack is a linked list of large chunks, so a call is mostly a pointer bump. Each frame stores a previous pointer to its caller. The thread state’s current_frame points at the innermost frame.

A naive unwinder would read one frame, follow previous, read the next frame, and so on. That is one system call per frame. Tachyon first copies whole data-stack chunks (following datastack_chunk and each chunk’s previous) and then walks the frame chain in its own memory. A remote frame address becomes an offset into a local buffer.

The walk checks each frame. It must end at the thread’s base_frame, which acts as a sentinel. If it ends anywhere else, the walk fails with “Incomplete sample: did not reach base frame” and the sample is dropped. A frame with a null executable is being torn down, so the walk skips it. The owner field separates frames owned by a thread from generator frames and from FRAME_OWNED_BY_INTERPRETER entry frames, which mark where C code called back into Python. A depth limit protects against cycles, and a separate check reports a “Broken frame chain” if an address does not match the previous pointer that led to it.

Two kinds of frame do not come from Python code. With --native, an entry frame between two Python frames becomes a <native> marker, so you can see where C code was running. When the garbage collector is active, a <GC> frame shows its cost. Both markers are stored with address 0, a detail the frame cache relies on later.

A frame does not record a function name. It holds executable, a tagged reference to a code object, and instr_ptr, a pointer into that code object’s bytecode. Tachyon removes the tag bits, reads the PyCodeObject, and pulls out the file name, qualified name, first line number and line table. Code objects rarely change, so the parsed result is cached by address. A hot function is decoded once, not on every sample.

The line number comes from instr_ptr. Subtract the start of the bytecode and divide by two bytes per code unit to get an instruction index. Then decode co_linetable, the compact location table from PEP 626 and PEP 657, which maps ranges of instructions to line deltas. On free-threaded builds a thread may run its own copy of the bytecode, so tlbc_index selects the right one.

Most of a deep stack does not change between samples. If a server is deep in request handling, most of its frames will still be the same one sampling interval later. Reading them again is wasted work. The frame cache (gh-138122) avoids it with a small change to the interpreter.

Each thread state has a field called last_profiled_frame. After each sample, Tachyon writes the innermost frame’s address into it. When the interpreter pops that frame, it moves the marker to the caller. Before the next sample, every frame above the marker has stayed in place, frozen at its call site. Tachyon walks from the current frame down to the marker, re-reads the marker frame because its line may have moved, and takes the callers from its cache.

The figure below plays five samples through the cache. Back and Next step through them by hand.

Why an address alone isn’t enough

Data-stack slots are reused. When a function returns, the next call made from that depth gets the same slot and the same address, as in sample 4 above. A matching address therefore does not prove that the frame is the one the cache remembers. The first version of the cache keyed only on the anchor address, which left it open to this kind of ABA problem.

The fix (gh-151613) adds last_profiled_frame_seq next to the anchor. When Tachyon stores a stack, it also stores the sequence value. On lookup, it counts the real frames that sat above the anchor in the cached stack and checks that stored_seq + pops == seq. In other words, the anchor must have moved exactly as many times as the cached frames above it were popped. Synthetic <native> and <GC> entries have address 0 and are not counted. The anchor and the counter sit next to each other in PyThreadState, so one remote read returns both. A second read of the anchor confirms it did not change during the lookup. On a cache miss, Tachyon copies the stack chunks before storing anything, so the cache is filled from one consistent snapshot.

The cache has limits. It holds a fixed number of thread slots and a fixed number of frames per thread. Only complete stacks that reach the base frame are stored. Entries for threads that no longer exist are removed. A later change (gh-149584) reads the interpreter, thread and frame structs at their exact sizes instead of whole pages. It also predicts the next thread state and top frame addresses so the reads can be batched into one process_vm_readv call.

By default Tachyon does not pause the target. The program keeps calling and returning while its memory is read, so a read can catch a stack in the middle of a change. Chunk copying makes this window much smaller, because one system call captures a whole region. The window still exists.

Tachyon reports these failures instead of hiding them. At the end of a run it prints its sampling efficiency (the share of reads that succeeded) and its missed samples (the share of expected samples it did not collect). Programs with many generators and coroutines are the most likely to produce occasional torn stacks.

If you need exact snapshots, --blocking stops every thread for each sample, using ptrace on Linux and thread_suspend on macOS. That costs the target real time. The command line warns when it is combined with an interval under 100 µs and suggests a rate of 1 kHz or lower.

An asyncio program’s real call stack mostly shows the event loop: run_forever, then _run_once, then Handle._run, then whichever coroutine step is running. The question you want answered is a different one: which await chain led to this code? With --async-aware, Tachyon reads asyncio’s own debug offsets (a second table, also validated when loaded). It finds the task running on each thread and follows the graph of which tasks await which. It then builds a logical stack from that graph.

--async-mode running (the default for run and attach) records only the task that is executing. --async-mode all records suspended tasks too, which shows where work is stuck waiting. Asyncio must already be imported in the target. This mode cannot be combined with --native, --all-threads, --no-gc, --binary or any --mode other than wall.

Each sample is a list of frames for each thread, with the leaf frame first. A collector receives the samples and aggregates them. The simplest output is the collapsed format: one line per distinct stack, frames joined with semicolons, followed by a count. A flame graph draws those lines. Stacks with the same prefix merge, each box is as wide as its share of samples, and the order along the x-axis carries no meaning. The figure below folds twelve samples on its own.

Tachyon includes several collectors, and you choose one with a flag:

A sampling profiler estimates. If a function shows up in k of n samples, its estimated share of time is k/n, and its estimated time is that share multiplied by the duration. The error shrinks with the square root of the sample count. Collecting 100 times more samples makes the estimate about 10 times tighter. At the default 1 kHz, a function that takes 5% of the time is pinned to about ±1.4 percentage points after one second (1,000 samples), and to about ±0.14 after 100 seconds. Samples taken close together are correlated, so the real error is somewhat larger. That is why a higher -r rate, or a longer -d duration, matters more than it might seem.

The same arithmetic shows a limit. Functions that run for much less than one sampling interval are seen only in proportion to how often they run. A function called once for 20 µs during a 10 s run will probably not appear at all. That doesn’t skew the profile as a whole, because short functions are counted in proportion to the time they take. But a sampling profiler cannot count calls, so use the tracing profiler when you need call counts.