Inside Tachyon

Each sample begins with a read of another process’s memory. Tachyon uses the operating system’s debugging APIs to copy ranges of the target’s memory into its own buffers, without stopping the process. On Linux a single system call copies a whole list of ranges at once. These APIs are privileged, so the operating system decides who may profile whom, and each platform has its own rules:

• Linux: process_vm_readv. It needs root, CAP_SYS_PTRACE or a ptrace_scope of 0. The default, 1, only lets a parent trace its child, so run works where attach may not.
• macOS: task_for_pid() and mach_vm reads. They need root or the com.apple.security.cs.debugger entitlement.
• Windows: ReadProcessMemory. It needs administrator rights or SeDebugPrivilege.

Before it can read a single frame, the unwinder needs two things. It needs a place to start: every interpreter, thread and frame is reached by following pointers from one global, _PyRuntime, whose address can change from run to run. And it needs to know how to read what it finds: the byte offset of each field in each struct, which changes between Python versions and builds. CPython puts both in one place. The first thing inside _PyRuntime is a table, _Py_DebugOffsets, that lists those offsets. It does not give addresses; it says where each field sits inside its struct. So finding the runtime also finds the instructions for reading it.

A debugger would find _PyRuntime through debug symbols, but release builds are stripped. CPython solves this by placing the runtime in a named binary section, PyRuntime. To find it, the unwinder looks up libpython (or the python binary) in the target’s memory mappings, reads the section table from that file on disk, and adds the address the file was loaded at.

The table begins with the 8-byte cookie "xdebugpy", then the version and a free-threading flag. After those come about a hundred entries, mostly byte offsets: where threads_head lives in an interpreter state, where current_frame lives in a thread state, where previous lives in a frame, and so on. The unwinder hardcodes none of these offsets. It reads the table and uses its values. The table first shipped in Python 3.13, before Tachyon. Python 3.14 extended it for PEP 768, adding the fields a debugger needs to run code in the target (sys.remote_exec). New fields arrive the same way: when CPython added last_profiled_frame for the frame cache in 3.15, its offset went into this table too.

/* simplified, from Include/internal/pycore_debug_offsets.h */
typedef struct _Py_DebugOffsets {
    char     cookie[8];       // "xdebugpy"
    uint64_t version;         // 0x030f.... for 3.15
    uint64_t free_threaded;
    // each field below is a uint64_t: a byte offset, or a struct's size
    struct { size; finalizing; interpreters_head; } runtime_state;
    struct { ... threads_head; ... } interpreter_state;
    struct { ... current_frame; base_frame; ...
             native_thread_id; datastack_chunk; ... } thread_state;
    struct { ... previous; executable; instr_ptr; ... } interpreter_frame;
    struct { ... linetable; ... co_code_adaptive; ... } code_object;
    // ... and more
} _Py_DebugOffsets;

/* simplified, from
   pycore_debug_offsets.h */
typedef struct _Py_DebugOffsets {
  char     cookie[8];  // "xdebugpy"
  uint64_t version;    // 0x030f....
  uint64_t free_threaded;
  // the rest are uint64_t offsets and sizes
  struct {
    size; finalizing; interpreters_head;
  } runtime_state;
  struct {
    ... threads_head; ...
  } interpreter_state;
  struct {
    ... current_frame; base_frame;
    native_thread_id; datastack_chunk; ...
  } thread_state;
  struct {
    ... previous; executable;
    instr_ptr; ...
  } interpreter_frame;
  struct {
    ... linetable; ...
    co_code_adaptive; ...
  } code_object;
  // ... and more
} _Py_DebugOffsets;

The cookie check matters more than it seems. One process can contain two copies of libpython. For example, ctypes can dlopen a second one whose runtime was never initialized. An early version took the first matching mapping. The fix (gh-144563) reads the first eight bytes of each candidate and accepts only one that starts with xdebugpy.

The offsets table comes from the process being profiled, so the unwinder treats it as untrusted input. A shared validator (gh-148178) checks every offset and size when the table is loaded, and malformed tables are rejected. After that the sampling loop can use the offsets directly, with no bounds checks in the hot path.

This table is also why the versions must match. The offsets describe one layout of PyThreadState and _PyInterpreterFrame, and those structs change between minor releases. So the profiler and the target must run the same minor version of Python, pre-releases must match exactly, and free-threaded and default builds cannot profile each other.

From the runtime, the unwinder follows interpreters_head to an interpreter state. By default Tachyon samples only the main thread, which the unwinder finds through threads_main. With -a (--all-threads) the unwinder follows threads_head instead, into a singly linked list of thread states. For each PyThreadState it reads native_thread_id and then follows next until it reaches a null pointer.

A profiling mode is a rule about which samples to keep. wall keeps every sample, including time spent sleeping or waiting on I/O. cpu keeps samples where the kernel reports the thread as running. gil keeps samples where the thread holds the GIL. exception keeps samples taken while an exception is being raised or handled. The test runs before the stack walk, so a thread that fails it is skipped without reading its frames.

Each mode tests one status flag, and the unwinder builds the flags from what the thread state already records:

• GIL. On default builds, holds_gil says whether the thread owns the GIL. If it does not and gil_requested is set, the thread is waiting for it. Free-threaded builds have no GIL, so the unwinder reads the active bit of the thread’s status instead.
• Exception. The thread is handling an exception if current_exception is set. The unwinder also follows the exc_info chain to the innermost handler, which is how sys.exception() finds it.
• On CPU. The interpreter does not record this, so the unwinder asks the kernel. On Linux it reads the state letter from /proc/<pid>/task/<tid>/stat, where R means running. This costs a file read, so the unwinder only does it when the mode needs it.

Try each mode on four simulated threads below, as if sampled with -a.

Comparing modes is often more useful than any one of them. A function that is large under wall and missing under cpu is waiting, not computing. A thread that is high under cpu and low under gil is spending its time in C code that released the GIL. The flame graph’s sidebar shows the same split as GIL held, released and waiting percentages.

Since Python 3.11, most frames are not heap objects. A function call pushes a _PyInterpreterFrame onto a per-thread data stack. The data stack is a linked list of large chunks, so a call is mostly a pointer bump. Each frame stores a previous pointer to its caller. The thread state’s current_frame points at the innermost frame.

Reading one frame, following previous, then reading the next frame would cost one system call per frame. The unwinder first copies whole data-stack chunks (following datastack_chunk and each chunk’s previous) and then walks the frame chain in its own memory. A remote frame address becomes an offset into a local buffer.

The walk checks each frame. It must end at the thread’s base_frame, which acts as a sentinel. If it ends anywhere else, the walk fails with “Incomplete sample: did not reach base frame” and the sample is dropped. A frame with a null executable is being torn down, so the walk skips it. The owner field separates frames owned by a thread from generator frames and from FRAME_OWNED_BY_INTERPRETER entry frames, which mark where C code called back into Python. A depth limit protects against cycles, and a separate check reports a “Broken frame chain” if an address does not match the previous pointer that led to it.

Two kinds of frame do not come from Python code. With --native, an entry frame between two Python frames becomes a <native> marker, so you can see where C code was running. The interpreter’s GC state records the frame that started a collection; when the walk meets that frame, it adds a <GC> frame above it, so the cost of collection shows up (--no-gc turns this off). Both markers are stored with address 0, a detail the frame cache relies on later.

A frame does not record a function name or a line number. It holds two pointers: executable, which points at the code object being run, and instr_ptr, which points at a position in that code object’s bytecode. On free-threaded builds the code object pointer carries a flag in its low bit, which the unwinder masks off. Everything a person wants to see comes from the code object.

The first time the unwinder meets a code object, it reads the object in one go, then the three things it points to: the qualified name (qualname), the file name (filename) and the line table (co_linetable). It keeps them, together with firstlineno and the start of the bytecode, in a cache keyed by the code object’s address, so a hot function is decoded once rather than on every sample. A failed read does not cost the whole sample. The frame is recorded as <unreadable frame>, or with <unknown function> or <unknown file> in place of the part that could not be read.

The line comes from instr_ptr. Subtracting the start of the bytecode (co_code_adaptive) and counting in two-byte code units gives an instruction index. parse_linetable then walks co_linetable, the compact location table from PEP 626 and PEP 657, to the entry that covers that index. On free-threaded builds a thread may run its own copy of the bytecode, so tlbc_index selects the copy to measure from. One more shortcut helps tight loops: each cached code object remembers the last instruction index it saw and the result it built, and a frame stopped at the same instruction reuses that result without decoding anything.

Keeping the cache honest

This cache has the same weakness as the frame cache in the next section: addresses are reused. When a code object is freed, a new one can be allocated at the same address, and a cache keyed by address would then report the wrong function. CPython closes the gap with a counter. Every time any code object is freed, code_dealloc adds one to a counter on the interpreter, and the debug table publishes its offset as interpreter_state.code_object_generation. The unwinder reads the counter on every sample and clears its whole code object cache when the value has changed. Free-threaded builds keep a second counter, tlbc_generation, which changes when threads are created or destroyed and invalidates the cached bytecode copies.

Which instruction was running

With --opcodes, the unwinder also reads the two-byte code unit at instr_ptr and keeps its opcode. The interpreter rewrites hot instructions into specialized forms as it runs, so this shows, for example, LOAD_ATTR_INSTANCE_VALUE where the bytecode started as a plain LOAD_ATTR, and the output shows both names. The documentation lists where it appears: flame graph tooltips, per-line panels in the heatmap, a panel in the live view and markers in the Firefox Profiler output. Each sample then needs the instruction itself, so the same-instruction shortcut is turned off.

With names and lines in place, each sample is now a real stack: function, file and line for every frame of every sampled thread. The next section is about reading less of it.

The frame cache rests on a small change to the interpreter. PR 142137 (under gh-138122) added a field to every thread state that exists only for profilers: last_profiled_frame. After each sample, the unwinder writes the address of the innermost frame into it, if it has changed. From then on the interpreter keeps it valid: when that frame returns, the interpreter moves the marker to its caller. It does this only when the marker matches the frame being popped, which, in the words of CPython’s internal docs, avoids any overhead when profiling is inactive. Those docs call the marker a high-water mark: the frame it points to is still on the stack, and its callers have not moved since the last sample. A later fix (gh-151436) made every place that removes a frame, including generators and coroutines, update the marker.

At the default rate, a new sample is due every millisecond, and most of a deep stack does not change in that time. If a server is deep in request handling, most of its frames will still be the same one sampling interval later. The marker tells the unwinder which ones it can skip. It walks from the current frame down to the marker, re-reads the marker frame because its line may have moved, and takes the callers from its cache. If the write ever fails, only the cache suffers.

The figure below plays five samples through the cache. Back and Next step through them by hand.

Why an address alone isn’t enough

Data-stack slots are reused. When a function returns, the next call made from that depth gets the same slot and the same address, as in sample 4 above. A matching address therefore does not prove that the frame is the one the cache remembers. The first version of the cache keyed only on the anchor address, which left it open to this kind of ABA problem.

The fix (gh-151613) adds last_profiled_frame_seq next to the anchor. When the unwinder stores a stack, it also stores the sequence value. On lookup, it counts the real frames that sat above the anchor in the cached stack and checks that stored_seq + pops == seq. In other words, the anchor must have moved exactly as many times as the cached frames above it were popped. Synthetic <native> and <GC> entries have address 0 and are not counted. The anchor and the counter sit next to each other in PyThreadState, so one remote read returns both. A second read of the anchor confirms it did not change during the lookup. On a cache miss, the unwinder copies the stack chunks before storing anything, so the cache is filled from one consistent snapshot.

The cache has limits. It holds a fixed number of thread slots and a fixed number of frames per thread. Only complete stacks that reach the base frame are stored. Entries for threads that no longer exist are removed. A later change (gh-149584) reads the interpreter, thread and frame structs at their exact sizes instead of whole pages. It also predicts the next thread state and top frame addresses so the reads can be batched into one process_vm_readv call.

By default Tachyon does not pause the target. The program keeps calling and returning while its memory is read, so a read can catch a stack in the middle of a change. Chunk copying makes this window much smaller, because one system call captures a whole region. The window still exists.

Tachyon counts these failures instead of hiding them; the next section shows where. The documentation names programs with many generators and coroutines, which switch rapidly between yield points, as the most likely to produce torn stacks.

If you need exact snapshots, --blocking pauses the target around each sample: on Linux it stops each thread with ptrace, on macOS it suspends the whole task with task_suspend, and on Windows it calls NtSuspendProcess. That costs the target real time. The command line warns when it is combined with an interval under 100 µs and suggests a rate of 1 kHz or lower.

Back to the loop from the start of the post. -r sets the rate, and the default 1khz gives an interval of 1,000 µs. -d sets a duration in seconds. Without it, the run lasts until the target exits, which the loop sees as a ProcessLookupError. Simplified, the loop looks like this:

# simplified, from Lib/profiling/sampling/sample.py
next_time = time.perf_counter()
while running:
    now = time.perf_counter()
    if next_time > now:                     # early: wait
        wait = (next_time - now) * 0.9
        if wait > 0.0001:
            time.sleep(wait)
    elif next_time < now:                   # due: sample
        try:
            stack = unwinder.get_stack_trace()
            collector.collect(stack)
        except (RuntimeError, OSError, ...):
            collector.collect_failed_sample()
            errors += 1
        num_samples += 1
        next_time += interval

# simplified, from
# profiling/sampling/sample.py
next_time = time.perf_counter()
while running:
  now = time.perf_counter()
  if next_time > now:   # early
    wait = (next_time - now) * 0.9
    if wait > 0.0001:
      time.sleep(wait)
  elif next_time < now: # due
    try:
      stack = unwinder.get_stack_trace()
      collector.collect(stack)
    except (RuntimeError, OSError, ...):
      collector.collect_failed_sample()
      errors += 1
    num_samples += 1
    next_time += interval

The schedule is fixed in advance. next_time moves forward by exactly one interval per sample, however long the sample took, so a slow sample does not push the later ones back. When the loop is early, it sleeps for 90% of the time remaining if that is more than 100 µs, then checks the clock again; shorter waits are spent spinning. When it is late, it samples at once. Because next_time still advances one interval per sample, a profiler that has fallen behind takes samples back to back until it catches up.

Two kinds of loss come out of this loop, and they mean different things. A failed attempt, such as a torn stack from section 07, raises an exception inside the C module. The loop counts it, tells the collector with collect_failed_sample(), and carries on. Missed samples are counted differently. At the end of the run, the loop divides the running time by the interval to get the number of samples it should have attempted. Failed attempts count as attempted, so the gap measures only the time when the profiler could not keep up. The run ends by printing the number of samples, the achieved rate and the error rate (failed attempts as a share of all attempts), with a warning if any samples were missed. The flame graph shows both figures in its sidebar, and --realtime-stats prints the achieved rate while the run is going.

One more detail keeps the collectors cheap. A program that stays in one place produces the same stack many times in a row. Collectors that only count stacks, such as the ones behind --collapsed and --flamegraph, set aggregating = True. For them, the loop holds back identical consecutive samples and delivers the stack once, with a list of their timestamps, up to 8,192 at a time. The collector weights the stack by the number of timestamps.

An asyncio program’s real call stack mostly shows the event loop: run_forever, then _run_once, then Handle._run, then whichever coroutine step is running. The question you want answered is a different one: which await chain led to this code? With --async-aware, the unwinder also reads asyncio’s own debug offsets (a second table, also validated when loaded). It finds the task running on each thread and reads the graph of which tasks await which. Tachyon then builds a logical stack from that graph.

--async-mode running (the default for run and attach) records only the task that is executing. --async-mode all records suspended tasks too, which shows where work is stuck waiting. Asyncio must already be imported in the target. This mode cannot be combined with --native, --all-threads, --no-gc, --binary or any --mode other than wall.

Each sample is a list of frames for each thread, with the leaf frame first. A collector receives the samples and aggregates them. The simplest output is the collapsed format: one line per distinct stack, frames joined with semicolons, followed by a count. A flame graph draws those lines. Stacks with the same prefix merge, each box is as wide as its share of samples, and the order along the x-axis carries no meaning. The figure below folds twelve samples on its own.

Tachyon includes several collectors, and you choose one with a flag:

A sampling profiler estimates. If a function shows up in k of n samples, its estimated share of time is k/n, and its estimated time is that share multiplied by the duration. The error shrinks with the square root of the sample count. Collecting 100 times more samples makes the estimate about 10 times tighter. At the default 1 kHz, a function that takes 5% of the time is pinned to about ±1.4 percentage points after one second (1,000 samples), and to about ±0.14 after 100 seconds. Samples taken close together are correlated, so the real error is somewhat larger. That is why a higher -r rate, or a longer -d duration, matters more than it might seem.

The same arithmetic shows a limit. Functions that run for much less than one sampling interval are seen only in proportion to how often they run. A function called once for 20 µs during a 10 s run will probably not appear at all. That doesn’t skew the profile as a whole, because short functions are counted in proportion to the time they take. But a sampling profiler cannot count calls, so use the tracing profiler when you need call counts.