languages/cpython
languages/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython synced 2024-09-05 00:05:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

bpo-43795: PEP 652 user documentation (GH-25668)

Browse source 
- Reformat the C API and ABI Versioning page (and extend/clarify a bit)
- Rewrite the stable ABI docs into a general text on C API Compatibility
- Add a list of Limited API contents, and notes for the individual items. 
- Replace `Include/README.rst` with a link to a devguide page with the same info
This commit is contained in:
Petr Viktorin 2021-05-11 16:04:33 +02:00 committed by GitHub
parent d1b81574ed
commit b05955d6f5
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 1180 additions and 1097 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

75
Doc/c-api/apiabiversion.rst
View file

@ -6,34 +6,57 @@
API and ABI Versioning
***********************
``PY_VERSION_HEX`` is the Python version number encoded in a single integer.
CPython exposes its version number in the following macros.
Note that these correspond to the version code is **built** with,
not necessarily the version used at **run time**.
For example if the ``PY_VERSION_HEX`` is set to ``0x030401a2``, the underlying
version information can be found by treating it as a 32 bit number in
the following manner:
See :ref:`stable` for a discussion of API and ABI stability across versions.
+-------+-------------------------+------------------------------------------------+
| Bytes | Bits (big endian order) | Meaning |
+=======+=========================+================================================+
| ``1`` | ``1-8`` | ``PY_MAJOR_VERSION`` (the ``3`` in |
| | | ``3.4.1a2``) |
+-------+-------------------------+------------------------------------------------+
| ``2`` | ``9-16`` | ``PY_MINOR_VERSION`` (the ``4`` in |
| | | ``3.4.1a2``) |
+-------+-------------------------+------------------------------------------------+
| ``3`` | ``17-24`` | ``PY_MICRO_VERSION`` (the ``1`` in |
| | | ``3.4.1a2``) |
+-------+-------------------------+------------------------------------------------+
| ``4`` | ``25-28`` | ``PY_RELEASE_LEVEL`` (``0xA`` for alpha, |
| | | ``0xB`` for beta, ``0xC`` for release |
| | | candidate and ``0xF`` for final), in this |
| | | case it is alpha. |
+-------+-------------------------+------------------------------------------------+
| | ``29-32`` | ``PY_RELEASE_SERIAL`` (the ``2`` in |
| | | ``3.4.1a2``, zero for final releases) |
+-------+-------------------------+------------------------------------------------+
.. c:macro:: PY_MAJOR_VERSION
The ``3`` in ``3.4.1a2``.
.. c:macro:: PY_MINOR_VERSION
The ``4`` in ``3.4.1a2``.
.. c:macro:: PY_MICRO_VERSION
The ``1`` in ``3.4.1a2``.
.. c:macro:: PY_RELEASE_LEVEL
The ``a`` in ``3.4.1a2``.
This can be ``0xA`` for alpha, ``0xB`` for beta, ``0xC`` for release
candidate or ``0xF`` for final.
.. c:macro:: PY_RELEASE_SERIAL
The ``2`` in ``3.4.1a2``. Zero for final releases.
.. c:macro:: PY_VERSION_HEX
The Python version number encoded in a single integer.
The underlying version information can be found by treating it as a 32 bit
number in the following manner:
+-------+-------------------------+-------------------------+--------------------------+
| Bytes | Bits (big endian order) | Meaning | Value for ``3.4.1a2`` |
+=======+=========================+=========================+==========================+
| 1 | 1-8 | ``PY_MAJOR_VERSION`` | ``0x03`` |
+-------+-------------------------+-------------------------+--------------------------+
| 2 | 9-16 | ``PY_MINOR_VERSION`` | ``0x04`` |
+-------+-------------------------+-------------------------+--------------------------+
| 3 | 17-24 | ``PY_MICRO_VERSION`` | ``0x01`` |
+-------+-------------------------+-------------------------+--------------------------+
| 4 | 25-28 | ``PY_RELEASE_LEVEL`` | ``0xA`` |
+ +-------------------------+-------------------------+--------------------------+
| | 29-32 | ``PY_RELEASE_SERIAL`` | ``0x2`` |
+-------+-------------------------+-------------------------+--------------------------+
Thus ``3.4.1a2`` is hexversion ``0x030401a2`` and ``3.10.0`` is
hexversion ``0x030a00f0``.
Thus ``3.4.1a2`` is hexversion ``0x030401a2``.
All the given macros are defined in :source:`Include/patchlevel.h`.

176
Doc/c-api/stable.rst
View file

@ -2,37 +2,157 @@
.. _stable:
***********************************
***************
C API Stability
***************
Python's C API is covered by the Backwards Compatibility Policy, :pep:`387`.
While the C API will change with every minor release (e.g. from 3.9 to 3.10),
most changes will be source-compatible, typically by only adding new API.
Changing existing API or removing API is only done after a deprecation period
or to fix serious issues.
CPython's Application Binary Interface (ABI) is forward- and
backwards-compatible across a minor release (if these are compiled the same
way; see :ref:`stable-abi-platform` below).
So, code compiled for Python 3.10.0 will work on 3.10.8 and vice versa,
but will need to be compiled separately for 3.9.x and 3.10.x.
Names prefixed by an underscore, such as ``_Py_InternalState``,
are private API that can change without notice even in patch releases.
Stable Application Binary Interface
***********************************
===================================
Traditionally, the C API of Python will change with every release. Most changes
will be source-compatible, typically by only adding API, rather than changing
existing API or removing API (although some interfaces do get removed after
being deprecated first).
Python 3.2 introduced the *Limited API*, a subset of Python's C API.
Extensions that only use the Limited API can be
compiled once and work with multiple versions of Python.
Contents of the Limited API are :ref:`listed below <stable-abi-list>`.
Unfortunately, the API compatibility does not extend to binary compatibility
(the ABI). The reason is primarily the evolution of struct definitions, where
addition of a new field, or changing the type of a field, might not break the
API, but can break the ABI. As a consequence, extension modules need to be
recompiled for every Python release (although an exception is possible on Unix
when none of the affected interfaces are used). In addition, on Windows,
extension modules link with a specific pythonXY.dll and need to be recompiled to
link with a newer one.
To enable this, Python provides a *Stable ABI*: a set of symbols that will
remain compatible across Python 3.x versions. The Stable ABI contains symbols
exposed in the Limited API, but also other ones – for example, functions
necessary to support older versions of the Limited API.
Since Python 3.2, a subset of the API has been declared to guarantee a stable
ABI. Extension modules wishing to use this API (called "limited API") need to
define ``Py_LIMITED_API``. A number of interpreter details then become hidden
from the extension module; in return, a module is built that works on any 3.x
version (x>=2) without recompilation.
(For simplicity, this document talks about *extensions*, but the Limited API
and Stable ABI work the same way for all uses of the API – for example,
embedding Python.)
In some cases, the stable ABI needs to be extended with new functions.
Extension modules wishing to use these new APIs need to set ``Py_LIMITED_API``
to the ``PY_VERSION_HEX`` value (see :ref:`apiabiversion`) of the minimum Python
version they want to support (e.g. ``0x03030000`` for Python 3.3). Such modules
will work on all subsequent Python releases, but fail to load (because of
missing symbols) on the older releases.
.. c:macro:: Py_LIMITED_API
As of Python 3.2, the set of functions available to the limited API is
documented in :pep:`384`. In the C API documentation, API elements that are not
part of the limited API are marked as "Not part of the limited API."
Define this macro ``Py_LIMITED_API`` before including ``Python.h`` to
opt in to only use the Limited API.
Defining ``Py_LIMITED_API`` to ``3`` will limit the available API so that
the extension will work without recompilation with all Python 3.x releases
(x>=2) on the particular :ref:`platform <stable-abi-platform>`.
Defining ``Py_LIMITED_API`` to a value of :c:data:`PY_VERSION_HEX` will
limit the available API so that the extension will work without
recompilation with all Python 3 releases from the specified one.
This will allow using additional API introduced up to this version,
but the extension will lose compatibility with earlier Python versions.
Rather than using the ``PY_VERSION_HEX`` macro directly, hardcode a minimum
minor version (e.g. ``0x030A0000`` for Python 3.10) for stability when
compiling with future Python versions.
On Windows, extensions that use the Stable ABI should be linked against
``python3.dll`` rather than a version-specific library such as
``python39.dll``.
On some platforms, Python will look for and load shared library files named
with the ``abi3`` tag (e.g. ``mymodule.abi3.so``).
It does not check if such extensions conform to a Stable ABI.
The user (or their packaging tools) need to ensure that, for example,
extensions built with the 3.10+ Limited API are not installed for lower
versions of Python.
All functions in the Stable ABI are present as functions in Python's shared
library, not solely as macros. This makes them usable from languages that don't
use the C preprocessor.
Limited API Scope and Performance
---------------------------------
The goal for the Limited API is to allow everything that is possible with the
full C API, but possibly with a performance penalty.
For example, while :c:func:`PyList_GetItem` is available, its “unsafe” macro
variant :c:func:`PyList_GET_ITEM` is not.
The macro can be faster because it can rely on version-specific implementation
details of the list object.
Without ``Py_LIMITED_API`` defined, some C API functions are inlined or
replaced by macros.
Defining ``Py_LIMITED_API`` disables this inlining, allowing stability as
Python's data structures are improved, but possibly reducing performance.
By leaving out the ``Py_LIMITED_API`` definition, it is possible to compile
a Limited API extension with a version-specific ABI. This can improve
performance for that Python version, but will limit compatibility.
Compiling with ``Py_LIMITED_API`` will then yield an extension that can be
distributed where a version-specific one is not available – for example,
for prereleases of an upcoming Python version.
Limited API Caveats
-------------------
Note that compiling with ``Py_LIMITED_API`` is *not* a complete guarantee that
code conforms to the Limited API or the Stable ABI. ``Py_LIMITED_API`` only
covers definitions, but an API also includes other issues, such as expected
semantics.
One issue that ``Py_LIMITED_API`` does not guard against is calling a function
with arguments that are invalid in a lower Python version.
For example, consider a function that starts accepting ``NULL`` for an
argument. In Python 3.9, ``NULL`` now selects a default behavior, but in
Python 3.8, the argument will be used directly, causing a ``NULL`` dereference
and crash. A similar argument works for fields of structs.
Another issue is that some struct fields are currently not hidden when
``Py_LIMITED_API`` is defined, even though they're part of the Limited API.
For these reasons, we recommend testing an extension with *all* minor Python
versions it supports, and preferably to build with the *lowest* such version.
We also recommend reviewing documentation of all used API to check
if it is explicitly part of the Limited API. Even with ``Py_LIMITED_API``
defined, a few private declarations are exposed for technical reasons (or
even unintentionally, as bugs).
Also note that the Limited API is not necessarily stable: compiling with
``Py_LIMITED_API`` with Python 3.8 means that the extension will
run with Python 3.12, but it will not necessarily *compile* with Python 3.12.
In particular, parts of the Limited API may be deprecated and removed,
provided that the Stable ABI stays stable.
.. _stable-abi-platform:
Platform Considerations
=======================
ABI stability depends not only on Python, but also on the compiler used,
lower-level libraries and compiler options. For the purposes of the Stable ABI,
these details define a “platform”. They usually depend on the OS
type and processor architecture
It is the responsibility of each particular distributor of Python
to ensure that all Python versions on a particular platform are built
in a way that does not break the Stable ABI.
This is the case with Windows and macOS releases from ``python.org`` and many
third-party distributors.
.. _stable-abi-list:
Contents of Limited API
=======================
Currently, the Limited API includes the following items:
.. limited-api-list::

3
Doc/conf.py
View file

@ -225,8 +225,9 @@
# Options for extensions
# ----------------------
# Relative filename of the reference count data file.
# Relative filename of the data files
refcount_file = 'data/refcounts.dat'
stable_abi_file = 'data/stable_abi.dat'
# Sphinx 2 and Sphinx 3 compatibility
# -----------------------------------

1814
Doc/data/stable_abi.dat
View file

File diff suppressed because it is too large Load diff

110
Doc/tools/extensions/c_annotations.py
View file

@ -10,8 +10,10 @@
* stable API annotations
Usage: Set the `refcount_file` config value to the path to the reference
Usage:
* Set the `refcount_file` config value to the path to the reference
count data file.
* Set the `stable_abi_file` config value to the path to stable ABI list.
:copyright: Copyright 2007-2014 by Georg Brandl.
:license: Python license.
@ -20,11 +22,23 @@
from os import path
from docutils import nodes
from docutils.parsers.rst import directives
from docutils.parsers.rst import Directive
from docutils.statemachine import StringList
import csv
from sphinx import addnodes
from sphinx.domains.c import CObject
REST_ROLE_MAP = {
'function': 'func',
'var': 'data',
'type': 'type',
'macro': 'macro',
'type': 'type',
}
class RCEntry:
def __init__(self, name):
self.name = name
@ -33,12 +47,10 @@ def __init__(self, name):
self.result_refs = None
class Annotations(dict):
@classmethod
def fromfile(cls, filename):
d = cls()
fp = open(filename, 'r')
try:
class Annotations:
def __init__(self, refcount_filename, stable_abi_file):
self.refcount_data = {}
with open(refcount_filename, 'r') as fp:
for line in fp:
line = line.strip()
if line[:1] in ("", "#"):
@ -50,9 +62,9 @@ def fromfile(cls, filename):
function, type, arg, refcount, comment = parts
# Get the entry, creating it if needed:
try:
entry = d[function]
entry = self.refcount_data[function]
except KeyError:
entry = d[function] = RCEntry(function)
entry = self.refcount_data[function] = RCEntry(function)
if not refcount or refcount == "null":
refcount = None
else:
@ -64,27 +76,58 @@ def fromfile(cls, filename):
else:
entry.result_type = type
entry.result_refs = refcount
finally:
fp.close()
return d
self.stable_abi_data = {}
with open(stable_abi_file, 'r') as fp:
for record in csv.DictReader(fp):
role = record['role']
name = record['name']
self.stable_abi_data[name] = record
def add_annotations(self, app, doctree):
for node in doctree.traverse(addnodes.desc_content):
par = node.parent
if par['domain'] != 'c':
continue
if par['stableabi']:
node.insert(0, nodes.emphasis(' Part of the stable ABI.',
' Part of the stable ABI.',
classes=['stableabi']))
if par['objtype'] != 'function':
continue
if not par[0].has_key('ids') or not par[0]['ids']:
continue
name = par[0]['ids'][0]
if name.startswith("c."):
name = name[2:]
entry = self.get(name)
objtype = par['objtype']
# Stable ABI annotation. These have two forms:
# Part of the [Stable ABI](link).
# Part of the [Stable ABI](link) since version X.Y.
record = self.stable_abi_data.get(name)
if record:
if record['role'] != objtype:
raise ValueError(
f"Object type mismatch in limited API annotation "
f"for {name}: {record['role']!r} != {objtype!r}")
stable_added = record['added']
message = ' Part of the '
emph_node = nodes.emphasis(message, message,
classes=['stableabi'])
ref_node = addnodes.pending_xref(
'Stable ABI', refdomain="std", reftarget='stable',
reftype='ref', refexplicit="False")
ref_node += nodes.Text('Stable ABI')
emph_node += ref_node
if record['ifdef_note']:
emph_node += nodes.Text(' ' + record['ifdef_note'])
if stable_added == '3.2':
# Stable ABI was introduced in 3.2.
emph_node += nodes.Text('.')
else:
emph_node += nodes.Text(f' since version {stable_added}.')
node.insert(0, emph_node)
# Return value annotation
if objtype != 'function':
continue
entry = self.refcount_data.get(name)
if not entry:
continue
elif not entry.result_type.endswith("Object*"):
@ -99,13 +142,36 @@ def add_annotations(self, app, doctree):
def init_annotations(app):
refcounts = Annotations.fromfile(
path.join(app.srcdir, app.config.refcount_file))
app.connect('doctree-read', refcounts.add_annotations)
annotations = Annotations(
path.join(app.srcdir, app.config.refcount_file),
path.join(app.srcdir, app.config.stable_abi_file),
)
app.connect('doctree-read', annotations.add_annotations)
class LimitedAPIList(Directive):
has_content = False
required_arguments = 0
optional_arguments = 0
final_argument_whitespace = True
def run(self):
content = []
for record in annotations.stable_abi_data.values():
role = REST_ROLE_MAP[record['role']]
name = record['name']
content.append(f'* :c:{role}:`{name}`')
pnode = nodes.paragraph()
self.state.nested_parse(StringList(content), 0, pnode)
return [pnode]
app.add_directive('limited-api-list', LimitedAPIList)
def setup(app):
app.add_config_value('refcount_file', '', True)
app.add_config_value('stable_abi_file', '', True)
app.connect('builder-inited', init_annotations)
# monkey-patch C object...

66
Include/README.rst
View file

@ -3,66 +3,10 @@ The Python C API
The C API is divided into three sections:
1. ``Include/``
2. ``Include/cpython/``
3. ``Include/internal/``
1. ``Include/``: Limited API
2. ``Include/cpython/``: CPython implementation details
3. ``Include/internal/``: The internal API
Information on changing the C API is available `in the developer guide`_
Include: Limited API
====================
``Include/``, excluding the ``cpython`` and ``internal`` subdirectories,
contains the public Limited API (Application Programming Interface).
The Limited API is a subset of the C API, designed to guarantee ABI
stability across Python 3 versions, and is defined in :pep:`384`.
Guidelines for expanding the Limited API:
- Functions *must not* steal references
- Functions *must not* return borrowed references
- Functions returning references *must* return a strong reference
- Macros should not expose implementation details
- Please start a public discussion before expanding the API
- Functions or macros with a ``_Py`` prefix do not belong in ``Include/``.
It is possible to add a function or macro to the Limited API from a
given Python version. For example, to add a function to the Limited API
from Python 3.10 and onwards, wrap it with
``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000``.
Include/cpython: CPython implementation details
===============================================
``Include/cpython/`` contains the public API that is excluded from the
Limited API and the Stable ABI.
Guidelines for expanding the public API:
- Functions *must not* steal references
- Functions *must not* return borrowed references
- Functions returning references *must* return a strong reference
Include/internal: The internal API
==================================
With PyAPI_FUNC or PyAPI_DATA
-----------------------------
Functions or structures in ``Include/internal/`` defined with
``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are
exposed only for specific use cases like debuggers and profilers.
With the extern keyword
-----------------------
Functions in ``Include/internal/`` defined with the ``extern`` keyword
*must not and can not* be used outside the CPython code base. Only
built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN``
macro defined) can use such functions.
When in doubt, new internal C functions should be defined in
``Include/internal`` using the ``extern`` keyword.
.. _in the developer guide: https://devguide.python.org/c-api/

33
Tools/scripts/stable_abi.py
View file

@ -21,6 +21,7 @@
import os.path
import io
import re
import csv
MISSING = object()
@ -45,6 +46,11 @@
MACOS = (sys.platform == "darwin")
UNIXY = MACOS or (sys.platform == "linux") # XXX should this be "not Windows"?
IFDEF_DOC_NOTES = {
'MS_WINDOWS': 'on Windows',
'HAVE_FORK': 'on platforms with fork()',
'USE_STACKCHECK': 'on platforms with USE_STACKCHECK',
}
# The stable ABI manifest (Misc/stable_abi.txt) exists only to fill the
# following dataclasses.
@ -227,16 +233,31 @@ def sort_key(item):
key=sort_key):
write(f'EXPORT_DATA({item.name})')
REST_ROLES = {
'function': 'function',
'data': 'var',
'struct': 'type',
'macro': 'macro',
# 'const': 'const', # all undocumented
'typedef': 'type',
}
@generator("doc_list", 'Doc/data/stable_abi.dat')
def gen_doc_annotations(manifest, args, outfile):
"""Generate/check the stable ABI list for documentation annotations"""
write = partial(print, file=outfile)
write("# Generated by Tools/scripts/stable_abi.py")
write()
for item in manifest.select(ABIItem.KINDS, include_abi_only=False):
write(item.name)
writer = csv.DictWriter(
outfile, ['role', 'name', 'added', 'ifdef_note'], lineterminator='\n')
writer.writeheader()
for item in manifest.select(REST_ROLES.keys(), include_abi_only=False):
if item.ifdef:
ifdef_note = IFDEF_DOC_NOTES[item.ifdef]
else:
ifdef_note = None
writer.writerow({
'role': REST_ROLES[item.kind],
'name': item.name,
'added': item.added,
'ifdef_note': ifdef_note})
def generate_or_check(manifest, args, path, func):
"""Generate/check a file with a single generator