From a782c6b36dbf411f03c11a0e9a21d4c4fe7c14bf Mon Sep 17 00:00:00 2001 From: Sam Gross Date: Fri, 5 Dec 2025 16:09:36 -0500 Subject: [PATCH] gh-124379: Document _PyStackRef --- Include/internal/pycore_stackref.h | 7 --- InternalDocs/README.md | 2 + InternalDocs/stackrefs.md | 79 ++++++++++++++++++++++++++++++ 3 files changed, 81 insertions(+), 7 deletions(-) create mode 100644 InternalDocs/stackrefs.md diff --git a/Include/internal/pycore_stackref.h b/Include/internal/pycore_stackref.h index e59611c07fa793..c86beebe6554c3 100644 --- a/Include/internal/pycore_stackref.h +++ b/Include/internal/pycore_stackref.h @@ -479,13 +479,6 @@ PyStackRef_AsPyObjectBorrow(_PyStackRef stackref) #define PyStackRef_IsDeferred(ref) (((ref).bits & Py_TAG_BITS) == Py_TAG_DEFERRED) -static inline PyObject * -PyStackRef_NotDeferred_AsPyObject(_PyStackRef stackref) -{ - assert(!PyStackRef_IsDeferred(stackref)); - return (PyObject *)stackref.bits; -} - static inline PyObject * PyStackRef_AsPyObjectSteal(_PyStackRef stackref) { diff --git a/InternalDocs/README.md b/InternalDocs/README.md index a6e2df5ae4a9c3..06f67b3cfc124a 100644 --- a/InternalDocs/README.md +++ b/InternalDocs/README.md @@ -36,6 +36,8 @@ Program Execution - [The Bytecode Interpreter](interpreter.md) +- [Stack references (_PyStackRef)](stackrefs.md) + - [The JIT](jit.md) - [Garbage Collector Design](garbage_collector.md) diff --git a/InternalDocs/stackrefs.md b/InternalDocs/stackrefs.md new file mode 100644 index 00000000000000..8290937c059316 --- /dev/null +++ b/InternalDocs/stackrefs.md @@ -0,0 +1,79 @@ +# Stack references (`_PyStackRef`) + +Stack references are the interpreter's tagged representation of values on the evaluation stack. +They carry metadata to track ownership and support optimizations such as tagged small ints. + +## Shape and tagging + +- A `_PyStackRef` is a tagged pointer-sized value (see `Include/internal/pycore_stackref.h`). +- Tag bits distinguish three cases: + - `Py_TAG_REFCNT` clear - reference count lives on the pointed-to object. + - `Py_TAG_REFCNT` set - ownership is "borrowed" (no refcount to drop on close) or the object is immortal. + - `Py_INT_TAG` - tagged small integer stored directly in the stackref (no heap allocation). +- Special constants: `PyStackRef_NULL`, `PyStackRef_ERROR`, and embedded `None`/`True`/`False`. + +In GIL builds, most objects carry their refcount; tagged borrowed refs skip decref on close. In free +threading builds, the tag is also used to mark deferred refcounted objects so the GC can see them and +to avoid refcount contention for short-lived stack values. + +## Converting to and from PyObject* + +Three conversions control ownership: + +- `PyStackRef_FromPyObjectNew(obj)` - create a new reference (INCREF if mortal). +- `PyStackRef_FromPyObjectSteal(obj)` - take over ownership without changing the count unless the + object is immortal. +- `PyStackRef_FromPyObjectBorrow(obj)` - create a borrowed stackref (never decref on close). + +The `obj` argument must not be `NULL`. + +Going back to `PyObject*` mirrors this: + +- `PyStackRef_AsPyObjectBorrow(ref)` - borrow the underlying pointer +- `PyStackRef_AsPyObjectSteal(ref)` - transfer ownership from the stackref +- `PyStackRef_AsPyObjectNew(ref)` - create a new owning reference + +Only `PyStackRef_AsPyObjectBorrow` allows ref to be `PyStackRef_NULL`. + +## Operations on stackrefs + +The interpreter treats `_PyStackRef` as the unit of stack storage. Ownership must be managed with +the stackref primitives: + +- `PyStackRef_DUP` - like `Py_NewRef` for stackrefs; preserves the original. +- `PyStackRef_Borrow` - create a borrowed stackref from another stackref. +- `PyStackRef_CLOSE` / `PyStackRef_XCLOSE` - like `Py_DECREF`; invalidates the stackref. +- `PyStackRef_CLEAR` - like `Py_CLEAR`; closes and sets the stackref to `PyStackRef_NULL` +- `PyStackRef_MakeHeapSafe` - converts borrowed reference to owning reference + +Borrow tracking (for debug builds with `Py_STACKREF_DEBUG`) records who you borrowed from and reports +double-close, leaked borrows, or use-after-close via fatal errors. + +## Borrow-friendly opcodes + +The interpreter can push borrowed references directly. For example, `LOAD_FAST_BORROW` loads a local +variable as a borrowed `_PyStackRef`, avoiding both INCREF and DECREF for the temporary lifetime on +the evaluation stack. + +## Tagged integers on the stack + +Small ints can be stored inline with `Py_INT_TAG`, so no heap object is involved. Helpers like +`PyStackRef_TagInt`, `PyStackRef_UntagInt`, and `PyStackRef_IncrementTaggedIntNoOverflow` operate on +these values. Type checks use `PyStackRef_IsTaggedInt` and `PyStackRef_LongCheck`. + +## Free threading considerations + +With `Py_GIL_DISABLED`, `Py_TAG_DEFERRED` is an alias for `Py_TAG_REFCNT`. +Objects that support deferred reference counting can be pushed to the evaluation +stack and stored in local variables without directly incrementing the reference +count because they are only freed during cyclic garbage collection. This avoids +reference count contention on short-lived values such as methods and types. The GC +scans each thread's locals and evaluation stack to keep objects that use +deferred reference counting alive. + +## Debugging support + +`Py_STACKREF_DEBUG` builds replace the inline tags with table-backed IDs so the runtime can track +creation sites, borrows, closes, and leaks. Enabling `Py_STACKREF_CLOSE_DEBUG` additionally records +double closes. The tables live on `PyInterpreterState` and are initialized in `pystate.c`; helper +routines reside in `Python/stackrefs.c`.