Inside Tachyon

Each sample begins with a read of another process’s memory. Tachyon uses the operating system’s debugging APIs to copy ranges of the target’s memory into its own buffers, without stopping the process. On Linux a single system call copies a whole list of ranges at once. These APIs are privileged, so the operating system decides who may profile whom, and each platform has its own rules:

• Linux: process_vm_readv. It needs root, CAP_SYS_PTRACE or a ptrace_scope of 0. The default, 1, only lets a parent trace its child, so run works where attach may not.
• macOS: task_for_pid() and mach_vm reads. They need root or the com.apple.security.cs.debugger entitlement.
• Windows: ReadProcessMemory. It needs administrator rights or SeDebugPrivilege.

Before it can read a single frame, the unwinder needs two things. It needs a place to start: every interpreter, thread and frame is reached by following pointers from one global, _PyRuntime, whose address can change from run to run. And it needs to know how to read what it finds: the byte offset of each field in each struct, which changes between Python versions and builds. CPython puts both in one place. The first thing inside _PyRuntime is a table, _Py_DebugOffsets, that lists those offsets. It does not give addresses; it says where each field sits inside its struct. So finding the runtime also finds the instructions for reading it.

A debugger would find _PyRuntime through debug symbols, but release builds are stripped. CPython solves this by placing the runtime in a named binary section, PyRuntime. To find it, the unwinder looks up libpython (or the python binary) in the target’s memory mappings, reads the section table from that file on disk, and adds the address the file was loaded at.

The debug offsets table

The table begins with the 8-byte cookie "xdebugpy", then the version and a free-threading flag. After those come about a hundred entries, mostly byte offsets: where threads_head lives in an interpreter state, where current_frame lives in a thread state, where previous lives in a frame, and so on.⁴ The unwinder hardcodes none of these offsets. It reads the table and uses its values. The table first shipped in Python 3.13, before Tachyon. Python 3.14 extended it for PEP 768, adding the fields a debugger needs to run code in the target (sys.remote_exec).² New fields arrive the same way: when CPython added last_profiled_frame for the frame cache in 3.15, its offset went into this table too.

/* simplified, from Include/internal/pycore_debug_offsets.h */
typedef struct _Py_DebugOffsets {
    char     cookie[8];       // "xdebugpy"
    uint64_t version;         // 0x030f.... for 3.15
    uint64_t free_threaded;
    // each field below is a uint64_t: a byte offset, or a struct's size
    struct { size; finalizing; interpreters_head; } runtime_state;
    struct { ... threads_head; ... } interpreter_state;
    struct { ... current_frame; base_frame; ...
             native_thread_id; datastack_chunk; ... } thread_state;
    struct { ... previous; executable; instr_ptr; ... } interpreter_frame;
    struct { ... linetable; ... co_code_adaptive; ... } code_object;
    // ... and more
} _Py_DebugOffsets;

/* simplified, from
   pycore_debug_offsets.h */
typedef struct _Py_DebugOffsets {
  char     cookie[8];  // "xdebugpy"
  uint64_t version;    // 0x030f....
  uint64_t free_threaded;
  // the rest are uint64_t offsets and sizes
  struct {
    size; finalizing; interpreters_head;
  } runtime_state;
  struct {
    ... threads_head; ...
  } interpreter_state;
  struct {
    ... current_frame; base_frame;
    native_thread_id; datastack_chunk; ...
  } thread_state;
  struct {
    ... previous; executable;
    instr_ptr; ...
  } interpreter_frame;
  struct {
    ... linetable; ...
    co_code_adaptive; ...
  } code_object;
  // ... and more
} _Py_DebugOffsets;

The cookie check matters more than it seems. One process can contain two copies of libpython. For example, ctypes can dlopen a second one whose runtime was never initialized. An early version took the first matching mapping. The fix (gh-144563) reads the first eight bytes of each candidate and accepts only one that starts with xdebugpy.

The offsets table comes from the process being profiled, so the unwinder treats it as untrusted input. A shared validator (gh-148178) checks every offset and size when the table is loaded, and malformed tables are rejected. After that the sampling loop can use the offsets directly, with no bounds checks in the hot path.

This table is also why the versions must match. The offsets describe one layout of PyThreadState and _PyInterpreterFrame, and those structs change between minor releases. So the profiler and the target must run the same minor version of Python, pre-releases must match exactly, and free-threaded and default builds cannot profile each other.¹²

From the runtime, the unwinder follows interpreters_head to an interpreter state. By default Tachyon samples only the main thread, which the unwinder finds through threads_main. With -a (--all-threads) the unwinder follows threads_head instead, into a singly linked list of thread states. For each PyThreadState it reads native_thread_id and then follows next until it reaches a null pointer.

A profiling mode is a rule about which samples to keep. wall keeps every sample, including time spent sleeping or waiting on I/O. cpu keeps samples where the kernel reports the thread as running. gil keeps samples where the thread holds the GIL. exception keeps samples taken while an exception is being raised or handled. The test runs before the stack walk, so a thread that fails it is skipped without reading its frames.

Each mode tests one status flag, and the unwinder builds the flags from what the thread state already records:

• GIL. On default builds, holds_gil says whether the thread owns the GIL. If it does not and gil_requested is set, the thread is waiting for it. Free-threaded builds have no GIL, so the unwinder reads the active bit of the thread’s status instead.
• Exception. The thread is handling an exception if current_exception is set. The unwinder also follows the exc_info chain to the innermost handler, which is how sys.exception() finds it.
• On CPU. The interpreter does not record this, so the unwinder asks the kernel. On Linux it reads the state letter from /proc/<pid>/task/<tid>/stat, where R means running. This costs a file read, so the unwinder only does it when the mode needs it.

Try each mode on four simulated threads below, as if sampled with -a.

Comparing modes is often more useful than any one of them. A function that is large under wall and missing under cpu is waiting, not computing. A thread that is high under cpu and low under gil is spending its time in C code that released the GIL. The flame graph’s sidebar shows the same split as GIL held, released and waiting percentages.

Since Python 3.11, most frames are not heap objects. A function call pushes a _PyInterpreterFrame onto a per-thread data stack. The data stack is a linked list of large chunks, so a call is mostly a pointer bump. Each frame stores a previous pointer to its caller. The thread state’s current_frame points at the innermost frame.

Reading one frame, following previous, then reading the next frame would cost one system call per frame. Instead, the unwinder copies whole data-stack chunks (following datastack_chunk and each chunk’s previous) and then walks the frame chain in its own memory. A remote frame address becomes an offset into a local buffer. A frame outside the chunks, such as a generator’s, is read on its own. When the stack has not changed since the last sample, the frame cache (§06) answers without copying anything.

The walk checks each frame. It must end at the thread’s base_frame, which acts as a sentinel. If it ends anywhere else, the walk fails with “Incomplete sample: did not reach base frame” and the sample is dropped. A frame with a null executable is being torn down, so the walk skips it. The owner field separates frames owned by a thread from generator frames and from FRAME_OWNED_BY_INTERPRETER entry frames, which mark where C code called back into Python. A depth limit protects against cycles, and a separate check reports a “Broken frame chain” if an address does not match the previous pointer that led to it.

Two kinds of frame do not come from Python code. With --native, an entry frame between two Python frames becomes a <native> marker, so you can see where C code was running. The interpreter’s GC state records the frame that started a collection; when the walk meets that frame, it adds a <GC> frame above it, so the cost of collection shows up (--no-gc turns this off). Both markers are stored with address 0, a detail the frame cache relies on later.

A frame does not record a function name or a line number. It holds two pointers: executable, which points at the code object being run, and instr_ptr, which points at a position in that code object’s bytecode. On any build, the code object pointer can carry a flag in its low bit, which the unwinder masks off. Everything a person wants to see comes from the code object.

The first time the unwinder meets a code object, it reads the object in one go, then the three things it points to: the qualified name (qualname), the file name (filename) and the line table (co_linetable). It keeps them, together with firstlineno and the start of the bytecode, in a cache keyed by the code object’s address (the code-object cache), so a hot function is decoded once rather than on every sample. A failed read does not cost the whole sample. The frame is recorded as <unreadable frame>, or with <unknown function> or <unknown file> in place of the part that could not be read.

The line comes from instr_ptr. Subtracting the start of the bytecode (co_code_adaptive) and counting in two-byte code units gives an instruction index. parse_linetable then walks co_linetable, the compact location table from PEP 626 and PEP 657, to the entry that covers that index. On free-threaded builds a thread may run its own copy of the bytecode, so tlbc_index selects the copy to measure from. One more shortcut helps tight loops: each cached code object remembers the last instruction index it saw and the result it built, and a frame stopped at the same instruction reuses that result without decoding anything.

Keeping the code-object cache honest

The code-object cache has the same weakness as the frame cache in the next section: addresses are reused. When a code object is freed, a new one can be allocated at the same address, and a cache keyed by address would then report the wrong function. CPython closes the gap with a counter. Every time any code object is freed, code_dealloc adds one to a counter on the interpreter, and the debug table publishes its offset as interpreter_state.code_object_generation. The unwinder reads the counter on every sample and clears its whole code-object cache when the value has changed. Free-threaded builds keep a second counter, tlbc_generation, which changes when threads are created or destroyed and invalidates the cached bytecode copies.

Which instruction was running

With --opcodes, the unwinder also reads the two-byte code unit at instr_ptr and keeps its opcode. The interpreter rewrites hot instructions into specialized forms as it runs, so this shows, for example, LOAD_ATTR_INSTANCE_VALUE where the bytecode started as a plain LOAD_ATTR, and the output shows both names. The documentation¹ lists where it appears: flame graph tooltips, per-line panels in the heatmap, a panel in the live view and markers in the Firefox Profiler output. Each sample then needs the instruction itself, so the same-instruction shortcut is turned off.

With names and lines in place, each sample is now a real stack: function, file and line for every frame of every sampled thread. The next section is about reading less of it.

At the default rate, a new sample is due every millisecond, and most of a deep stack does not change in that time. If a server is deep in request handling, most of its frames will still be the same one sampling interval later. The frame cache lets the unwinder skip them: it keeps each thread’s last stack and reads only the frames that changed. It is an option of the unwinder, cache_frames, and Tachyon always turns it on.

To know which frames changed, the unwinder needs help from the interpreter. PR 142137 (under gh-138122), the change that added the cache, also added a field to every thread state: last_profiled_frame. Only profilers that read the stack from another process use it. After each sample, the unwinder writes the address of the innermost frame into it, if it has changed. From then on the interpreter keeps it valid: when that frame returns, the interpreter moves the marker to its caller. It does this only when the marker matches the frame being popped, which, in the words of CPython’s internal docs⁹, avoids any overhead when profiling is inactive. Those docs call the marker a high-water mark: the frame it points to is still on the stack, and its callers have not moved since the last sample. A later fix (gh-151436) made every place that removes a frame, including generators and coroutines, update the marker.

On the next sample, the unwinder walks from the current frame down to the marker, re-reads the marker frame because its line may have moved, and takes the callers from its cache. If the write ever fails, only the cache suffers.

The figure below plays five samples through the cache. Back and Next step through them by hand.

Why an address alone isn’t enough

Data-stack slots are reused. When a function returns, the next call made from that depth gets the same slot and the same address, as in sample 4 above. A matching address therefore does not prove that the frame is the one the cache remembers. The first version of the cache keyed only on the anchor address, which left it open to this kind of ABA problem.

The fix (gh-151613) adds last_profiled_frame_seq next to the anchor. When the unwinder stores a stack, it also stores the sequence value. On lookup, it counts the real frames that sat above the anchor in the cached stack and checks that stored_seq + pops == seq. In other words, the anchor must have moved exactly as many times as the cached frames above it were popped. Synthetic <native> and <GC> entries have address 0 and are not counted. The anchor and the counter sit next to each other in PyThreadState, so one remote read returns both. A second read of the anchor confirms it did not change during the lookup. On a cache miss, the unwinder copies the stack chunks before storing anything, so the cache is filled from one consistent snapshot.

The cache has limits. It holds a fixed number of thread slots and a fixed number of frames per thread. Only complete stacks that reach the base frame are stored. Entries for threads that no longer exist are removed. A later change (gh-149584) reads the interpreter, thread and frame structs at their exact sizes instead of whole pages. It also predicts the next thread state and top frame addresses so the reads can be batched into one process_vm_readv call.

By default Tachyon does not pause the target. The program keeps calling and returning while its memory is read, so a read can catch a stack in the middle of a change. Chunk copying makes this window much smaller, because one system call captures a whole region. The window still exists.

The walk in §04 Walking the frames catches most of these. A torn chain usually does not end at the thread’s base_frame, so the walk fails with “Incomplete sample” and the sample is dropped. Tachyon counts these failures instead of hiding them; the next section shows where. The check is not complete, though. A tear can also leave a chain that still ends at the base frame, and then the stack is recorded. The documentation¹ warns that such stacks can “mix frames from different execution states or that never actually existed”. It names programs with many generators and coroutines, which switch rapidly between yield points, as the most likely to produce them.

If you need exact snapshots, --blocking pauses the target around each sample: on Linux it stops each thread with ptrace, on macOS it suspends the whole task with task_suspend, and on Windows it calls NtSuspendProcess. That costs the target real time. The command line warns when it is combined with an interval under 100 µs and suggests a rate of 1 kHz or lower.

Tachyon samples on a fixed schedule. next_time is the moment the next sample is due. It starts at the moment sampling begins, so the first sample is taken at once, and every later sample is due a whole number of intervals after that start. Each turn of the loop checks the clock against next_time, then either waits or takes a sample, and moves next_time forward.

When the loop is early, it waits. It sleeps for 90% of the time left and checks the clock again; once that 90% would be under 100 µs, it spins instead.

When a sample is due, the loop calls the unwinder and hands the stack to the collector.

Either way, next_time then moves forward by exactly one interval, however long the attempt took. The interval comes from the sampling rate, -r: at the default 1khz, due times are 1,000 µs apart. The schedule is fixed in advance, so a slow sample does not push the later ones back. A loop that has fallen behind samples back to back until it catches up.

Two kinds of loss come out of this loop, and they mean different things. A failed attempt, such as a torn stack from §07 Torn reads, raises an exception inside the unwinder. The loop counts it, tells the collector with collect_failed_sample(), and carries on. Missed samples are counted differently. At the end of the run, the loop divides the running time by the interval to get the number of samples it should have attempted. Failed attempts count as attempted, so the gap measures only the time when the profiler could not keep up.

The run ends when -d seconds have passed or, without -d, when the target exits, which the loop sees as a ProcessLookupError. It prints the number of samples, the achieved rate and the error rate (failed attempts as a share of all attempts), with a warning if any samples were missed. The flame graph shows both figures in its sidebar, and --realtime-stats prints the achieved rate while the run is going.

One more detail keeps the collectors cheap. A program that stays in one place produces the same stack many times in a row. The collectors behind --pstats, --collapsed, --flamegraph and --gecko set aggregating = True. For them, the loop holds back identical consecutive samples and delivers the stack once, with a list of their timestamps, up to 8,192 at a time. A counting collector weights the stack by the number of timestamps; the Gecko collector keeps each timestamp for its timeline.

An asyncio program’s real call stack mostly shows the event loop: run_forever, then _run_once, then Handle._run, then whichever coroutine step is running. The question you want answered is a different one: which await chain led to this code? With --async-aware, the unwinder also reads asyncio’s own debug offsets (a second table, also validated when loaded). It finds the task running on each thread and reads the graph of which tasks await which. Tachyon then builds a logical stack from that graph.

--async-mode running (the default for run and attach) records only the task that is executing. --async-mode all records suspended tasks too, which shows where work is stuck waiting. Asyncio must already be imported in the target. This mode cannot be combined with --native, --all-threads, --no-gc, --binary or any --mode other than wall.

Each sample is a list of frames for each thread, with the leaf frame first. A collector receives the samples and aggregates them. The simplest output is the collapsed format: one line per distinct stack and thread. A line starts with the thread id (tid: and the number), then lists the frames from the outermost in, each as file.py:function:line, joined with semicolons, and ends with the count. A flame graph draws those lines. Stacks with the same prefix merge, each box is as wide as its share of samples, and the order along the x-axis carries no meaning. The figure below folds twelve samples on its own.

Tachyon includes several collectors, and you choose one with a flag:

A sampling profiler estimates. If a function shows up in k of n samples, its estimated share of time is k/n, and Tachyon reports its time as k sampling intervals. The error shrinks with the square root of the sample count: collecting 100 times more samples makes the estimate about 10 times tighter. Results also vary a little between runs, and the documentation¹ suggests a different tool for scripts that finish in under a second, or for differences of only a percent or two. That is why a higher -r rate, or a longer -d duration, matters.

The same arithmetic shows a limit. A call much shorter than one interval is caught only by chance, with a probability proportional to its length, so a function called once for 20 µs during a 10 s run will probably not appear at all. Over many calls this evens out: a function’s sample count still tracks its total time, so the profile as a whole is not skewed. But a sampling profiler cannot count calls, so use the tracing profiler when you need call counts.