gh-110812: Isolating Extension Modules HOWTO: List GC-related gotchas (GH-111504)

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

Doc/howto/isolating-extensions.rst

@@ -339,12 +339,44 @@ That is, heap types should:
 - Define a traverse function using ``Py_tp_traverse``, which
   visits the type (e.g. using :c:expr:`Py_VISIT(Py_TYPE(self))`).
 
-Please refer to the :ref:`the documentation <type-structs>` of
+Please refer to the the documentation of
 :c:macro:`Py_TPFLAGS_HAVE_GC` and :c:member:`~PyTypeObject.tp_traverse`
 for additional considerations.
 
-If your traverse function delegates to the ``tp_traverse`` of its base class
-(or another type), ensure that ``Py_TYPE(self)`` is visited only once.
+The API for defining heap types grew organically, leaving it
+somewhat awkward to use in its current state.
+The following sections will guide you through common issues.
+
+
+``tp_traverse`` in Python 3.8 and lower
+.......................................
+
+The requirement to visit the type from ``tp_traverse`` was added in Python 3.9.
+If you support Python 3.8 and lower, the traverse function must *not*
+visit the type, so it must be more complicated::
+
+   static int my_traverse(PyObject *self, visitproc visit, void *arg)
+   {
+       if (Py_Version >= 0x03090000) {
+           Py_VISIT(Py_TYPE(self));
+       }
+       return 0;
+   }
+
+Unfortunately, :c:data:`Py_Version` was only added in Python 3.11.
+As a replacement, use:
+
+* :c:macro:`PY_VERSION_HEX`, if not using the stable ABI, or
+* :py:data:`sys.version_info` (via :c:func:`PySys_GetObject` and
+  :c:func:`PyArg_ParseTuple`).
+
+
+Delegating ``tp_traverse``
+..........................
+
+If your traverse function delegates to the :c:member:`~PyTypeObject.tp_traverse`
+of its base class (or another type), ensure that ``Py_TYPE(self)`` is visited
+only once.
 Note that only heap type are expected to visit the type in ``tp_traverse``.
 
 For example, if your traverse function includes::
@@ -356,11 +388,70 @@ For example, if your traverse function includes::
     if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) {
         // a heap type's tp_traverse already visited Py_TYPE(self)
     } else {
+        if (Py_Version >= 0x03090000) {
             Py_VISIT(Py_TYPE(self));
         }
+    }
 
-It is not necessary to handle the type's reference count in ``tp_new``
-and ``tp_clear``.
+It is not necessary to handle the type's reference count in
+:c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_clear`.
+
+
+Defining ``tp_dealloc``
+.......................
+
+If your type has a custom :c:member:`~PyTypeObject.tp_dealloc` function,
+it needs to:
+
+- call :c:func:`PyObject_GC_UnTrack` before any fields are invalidated, and
+- decrement the reference count of the type.
+
+To keep the type valid while ``tp_free`` is called, the type's refcount needs
+to be decremented *after* the instance is deallocated. For example::
+
+   static void my_dealloc(PyObject *self)
+   {
+       PyObject_GC_UnTrack(self);
+       ...
+       PyTypeObject *type = Py_TYPE(self);
+       type->tp_free(self);
+       Py_DECREF(type);
+   }
+
+The default ``tp_dealloc`` function does this, so
+if your type does *not* override
+``tp_dealloc`` you don't need to add it.
+
+
+Not overriding ``tp_free``
+..........................
+
+The :c:member:`~PyTypeObject.tp_free` slot of a heap type must be set to
+:c:func:`PyObject_GC_Del`.
+This is the default; do not override it.
+
+
+Avoiding ``PyObject_New``
+.........................
+
+GC-tracked objects need to be allocated using GC-aware functions.
+
+If you use use :c:func:`PyObject_New` or :c:func:`PyObject_NewVar`:
+
+- Get and call type's :c:member:`~PyTypeObject.tp_alloc` slot, if possible.
+  That is, replace ``TYPE *o = PyObject_New(TYPE, typeobj)`` with::
+
+      TYPE *o = typeobj->tp_alloc(typeobj, 0);
+
+  Replace ``o = PyObject_NewVar(TYPE, typeobj, size)`` with the same,
+  but use size instead of the 0.
+
+- If the above is not possible (e.g. inside a custom ``tp_alloc``),
+  call :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`::
+
+      TYPE *o = PyObject_GC_New(TYPE, typeobj);
+
+      TYPE *o = PyObject_GC_NewVar(TYPE, typeobj, size);
 
 
 Module State Access from Classes