mirror of
https://github.com/python/cpython
synced 2024-10-05 06:17:28 +00:00
[3.12] gh-114785: Remove content from Porting from Python2
how-to (GH-114805) (GH-115327)
Keep the page though, because people might still rely on it (the traffic shows that they do).
Instead of our own manual we now give links to the 3rd-party ones.
(cherry picked from commit 705c76d4a2
)
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
This commit is contained in:
parent
cab8d07d17
commit
2ea2d25cc6
|
@ -13,7 +13,6 @@ Currently, the HOWTOs are:
|
|||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
pyporting.rst
|
||||
cporting.rst
|
||||
curses.rst
|
||||
descriptor.rst
|
||||
|
|
|
@ -1,3 +1,5 @@
|
|||
:orphan:
|
||||
|
||||
.. _pyporting-howto:
|
||||
|
||||
*************************************
|
||||
|
@ -6,423 +8,30 @@ How to port Python 2 Code to Python 3
|
|||
|
||||
:author: Brett Cannon
|
||||
|
||||
.. topic:: Abstract
|
||||
Python 2 reached its official end-of-life at the start of 2020. This means
|
||||
that no new bug reports, fixes, or changes will be made to Python 2 - it's
|
||||
no longer supported: see :pep:`373` and
|
||||
`status of Python versions <https://devguide.python.org/versions>`_.
|
||||
|
||||
Python 2 reached its official end-of-life at the start of 2020. This means
|
||||
that no new bug reports, fixes, or changes will be made to Python 2 - it's
|
||||
no longer supported.
|
||||
If you are looking to port an extension module instead of pure Python code,
|
||||
please see :ref:`cporting-howto`.
|
||||
|
||||
This guide is intended to provide you with a path to Python 3 for your
|
||||
code, that includes compatibility with Python 2 as a first step.
|
||||
The archived python-porting_ mailing list may contain some useful guidance.
|
||||
|
||||
If you are looking to port an extension module instead of pure Python code,
|
||||
please see :ref:`cporting-howto`.
|
||||
Since Python 3.13 the original porting guide was discontinued.
|
||||
You can find the old guide in the
|
||||
`archive <https://docs.python.org/3.12/howto/pyporting.html>`_.
|
||||
|
||||
The archived python-porting_ mailing list may contain some useful guidance.
|
||||
|
||||
Third-party guides
|
||||
==================
|
||||
|
||||
The Short Explanation
|
||||
=====================
|
||||
There are also multiple third-party guides that might be useful:
|
||||
|
||||
To achieve Python 2/3 compatibility in a single code base, the basic steps
|
||||
are:
|
||||
- `Guide by Fedora <https://portingguide.readthedocs.io>`_
|
||||
- `PyCon 2020 tutorial <https://www.youtube.com/watch?v=JgIgEjASOlk>`_
|
||||
- `Guide by DigitalOcean <https://www.digitalocean.com/community/tutorials/how-to-port-python-2-code-to-python-3>`_
|
||||
- `Guide by ActiveState <https://www.activestate.com/blog/how-to-migrate-python-2-applications-to-python-3>`_
|
||||
|
||||
#. Only worry about supporting Python 2.7
|
||||
#. Make sure you have good test coverage (coverage.py_ can help;
|
||||
``python -m pip install coverage``)
|
||||
#. Learn the differences between Python 2 and 3
|
||||
#. Use Futurize_ (or Modernize_) to update your code (e.g. ``python -m pip install future``)
|
||||
#. Use Pylint_ to help make sure you don't regress on your Python 3 support
|
||||
(``python -m pip install pylint``)
|
||||
#. Use caniusepython3_ to find out which of your dependencies are blocking your
|
||||
use of Python 3 (``python -m pip install caniusepython3``)
|
||||
#. Once your dependencies are no longer blocking you, use continuous integration
|
||||
to make sure you stay compatible with Python 2 and 3 (tox_ can help test
|
||||
against multiple versions of Python; ``python -m pip install tox``)
|
||||
#. Consider using optional :term:`static type checking <static type checker>`
|
||||
to make sure your type usage
|
||||
works in both Python 2 and 3 (e.g. use mypy_ to check your typing under both
|
||||
Python 2 and Python 3; ``python -m pip install mypy``).
|
||||
|
||||
.. note::
|
||||
|
||||
Note: Using ``python -m pip install`` guarantees that the ``pip`` you invoke
|
||||
is the one installed for the Python currently in use, whether it be
|
||||
a system-wide ``pip`` or one installed within a
|
||||
:ref:`virtual environment <tut-venv>`.
|
||||
|
||||
Details
|
||||
=======
|
||||
|
||||
Even if other factors - say, dependencies over which you have no control -
|
||||
still require you to support Python 2, that does not prevent you taking the
|
||||
step of including Python 3 support.
|
||||
|
||||
Most changes required to support Python 3 lead to cleaner code using newer
|
||||
practices even in Python 2 code.
|
||||
|
||||
|
||||
Different versions of Python 2
|
||||
------------------------------
|
||||
|
||||
Ideally, your code should be compatible with Python 2.7, which was the
|
||||
last supported version of Python 2.
|
||||
|
||||
Some of the tools mentioned in this guide will not work with Python 2.6.
|
||||
|
||||
If absolutely necessary, the six_ project can help you support Python 2.5 and
|
||||
3 simultaneously. Do realize, though, that nearly all the projects listed in
|
||||
this guide will not be available to you.
|
||||
|
||||
If you are able to skip Python 2.5 and older, the required changes to your
|
||||
code will be minimal. At worst you will have to use a function instead of a
|
||||
method in some instances or have to import a function instead of using a
|
||||
built-in one.
|
||||
|
||||
|
||||
Make sure you specify the proper version support in your ``setup.py`` file
|
||||
--------------------------------------------------------------------------
|
||||
|
||||
In your ``setup.py`` file you should have the proper `trove classifier`_
|
||||
specifying what versions of Python you support. As your project does not support
|
||||
Python 3 yet you should at least have
|
||||
``Programming Language :: Python :: 2 :: Only`` specified. Ideally you should
|
||||
also specify each major/minor version of Python that you do support, e.g.
|
||||
``Programming Language :: Python :: 2.7``.
|
||||
|
||||
|
||||
Have good test coverage
|
||||
-----------------------
|
||||
|
||||
Once you have your code supporting the oldest version of Python 2 you want it
|
||||
to, you will want to make sure your test suite has good coverage. A good rule of
|
||||
thumb is that if you want to be confident enough in your test suite that any
|
||||
failures that appear after having tools rewrite your code are actual bugs in the
|
||||
tools and not in your code. If you want a number to aim for, try to get over 80%
|
||||
coverage (and don't feel bad if you find it hard to get better than 90%
|
||||
coverage). If you don't already have a tool to measure test coverage then
|
||||
coverage.py_ is recommended.
|
||||
|
||||
|
||||
Be aware of the differences between Python 2 and 3
|
||||
--------------------------------------------------
|
||||
|
||||
Once you have your code well-tested you are ready to begin porting your code to
|
||||
Python 3! But to fully understand how your code is going to change and what
|
||||
you want to look out for while you code, you will want to learn what changes
|
||||
Python 3 makes in terms of Python 2.
|
||||
|
||||
Some resources for understanding the differences and their implications for you
|
||||
code:
|
||||
|
||||
* the :ref:`"What's New" <whatsnew-index>` doc for each release of Python 3
|
||||
* the `Porting to Python 3`_ book (which is free online)
|
||||
* the handy `cheat sheet`_ from the Python-Future project.
|
||||
|
||||
|
||||
Update your code
|
||||
----------------
|
||||
|
||||
There are tools available that can port your code automatically.
|
||||
|
||||
Futurize_ does its best to make Python 3 idioms and practices exist in Python
|
||||
2, e.g. backporting the ``bytes`` type from Python 3 so that you have
|
||||
semantic parity between the major versions of Python. This is the better
|
||||
approach for most cases.
|
||||
|
||||
Modernize_, on the other hand, is more conservative and targets a Python 2/3
|
||||
subset of Python, directly relying on six_ to help provide compatibility.
|
||||
|
||||
A good approach is to run the tool over your test suite first and visually
|
||||
inspect the diff to make sure the transformation is accurate. After you have
|
||||
transformed your test suite and verified that all the tests still pass as
|
||||
expected, then you can transform your application code knowing that any tests
|
||||
which fail is a translation failure.
|
||||
|
||||
Unfortunately the tools can't automate everything to make your code work under
|
||||
Python 3, and you will also need to read the tools' documentation in case some
|
||||
options you need are turned off by default.
|
||||
|
||||
Key issues to be aware of and check for:
|
||||
|
||||
Division
|
||||
++++++++
|
||||
|
||||
In Python 3, ``5 / 2 == 2.5`` and not ``2`` as it was in Python 2; all
|
||||
division between ``int`` values result in a ``float``. This change has
|
||||
actually been planned since Python 2.2 which was released in 2002. Since then
|
||||
users have been encouraged to add ``from __future__ import division`` to any
|
||||
and all files which use the ``/`` and ``//`` operators or to be running the
|
||||
interpreter with the ``-Q`` flag. If you have not been doing this then you
|
||||
will need to go through your code and do two things:
|
||||
|
||||
#. Add ``from __future__ import division`` to your files
|
||||
#. Update any division operator as necessary to either use ``//`` to use floor
|
||||
division or continue using ``/`` and expect a float
|
||||
|
||||
The reason that ``/`` isn't simply translated to ``//`` automatically is that if
|
||||
an object defines a ``__truediv__`` method but not ``__floordiv__`` then your
|
||||
code would begin to fail (e.g. a user-defined class that uses ``/`` to
|
||||
signify some operation but not ``//`` for the same thing or at all).
|
||||
|
||||
|
||||
Text versus binary data
|
||||
+++++++++++++++++++++++
|
||||
|
||||
In Python 2 you could use the ``str`` type for both text and binary data.
|
||||
Unfortunately this confluence of two different concepts could lead to brittle
|
||||
code which sometimes worked for either kind of data, sometimes not. It also
|
||||
could lead to confusing APIs if people didn't explicitly state that something
|
||||
that accepted ``str`` accepted either text or binary data instead of one
|
||||
specific type. This complicated the situation especially for anyone supporting
|
||||
multiple languages as APIs wouldn't bother explicitly supporting ``unicode``
|
||||
when they claimed text data support.
|
||||
|
||||
Python 3 made text and binary data distinct types that cannot simply be mixed
|
||||
together. For any code that deals only with text or only binary data, this
|
||||
separation doesn't pose an issue. But for code that has to deal with both, it
|
||||
does mean you might have to now care about when you are using text compared
|
||||
to binary data, which is why this cannot be entirely automated.
|
||||
|
||||
Decide which APIs take text and which take binary (it is **highly** recommended
|
||||
you don't design APIs that can take both due to the difficulty of keeping the
|
||||
code working; as stated earlier it is difficult to do well). In Python 2 this
|
||||
means making sure the APIs that take text can work with ``unicode`` and those
|
||||
that work with binary data work with the ``bytes`` type from Python 3
|
||||
(which is a subset of ``str`` in Python 2 and acts as an alias for ``bytes``
|
||||
type in Python 2). Usually the biggest issue is realizing which methods exist
|
||||
on which types in Python 2 and 3 simultaneously (for text that's ``unicode``
|
||||
in Python 2 and ``str`` in Python 3, for binary that's ``str``/``bytes`` in
|
||||
Python 2 and ``bytes`` in Python 3).
|
||||
|
||||
The following table lists the **unique** methods of each data type across
|
||||
Python 2 and 3 (e.g., the ``decode()`` method is usable on the equivalent binary
|
||||
data type in either Python 2 or 3, but it can't be used by the textual data
|
||||
type consistently between Python 2 and 3 because ``str`` in Python 3 doesn't
|
||||
have the method). Do note that as of Python 3.5 the ``__mod__`` method was
|
||||
added to the bytes type.
|
||||
|
||||
======================== =====================
|
||||
**Text data** **Binary data**
|
||||
------------------------ ---------------------
|
||||
\ decode
|
||||
------------------------ ---------------------
|
||||
encode
|
||||
------------------------ ---------------------
|
||||
format
|
||||
------------------------ ---------------------
|
||||
isdecimal
|
||||
------------------------ ---------------------
|
||||
isnumeric
|
||||
======================== =====================
|
||||
|
||||
Making the distinction easier to handle can be accomplished by encoding and
|
||||
decoding between binary data and text at the edge of your code. This means that
|
||||
when you receive text in binary data, you should immediately decode it. And if
|
||||
your code needs to send text as binary data then encode it as late as possible.
|
||||
This allows your code to work with only text internally and thus eliminates
|
||||
having to keep track of what type of data you are working with.
|
||||
|
||||
The next issue is making sure you know whether the string literals in your code
|
||||
represent text or binary data. You should add a ``b`` prefix to any
|
||||
literal that presents binary data. For text you should add a ``u`` prefix to
|
||||
the text literal. (There is a :mod:`__future__` import to force all unspecified
|
||||
literals to be Unicode, but usage has shown it isn't as effective as adding a
|
||||
``b`` or ``u`` prefix to all literals explicitly)
|
||||
|
||||
You also need to be careful about opening files. Possibly you have not always
|
||||
bothered to add the ``b`` mode when opening a binary file (e.g., ``rb`` for
|
||||
binary reading). Under Python 3, binary files and text files are clearly
|
||||
distinct and mutually incompatible; see the :mod:`io` module for details.
|
||||
Therefore, you **must** make a decision of whether a file will be used for
|
||||
binary access (allowing binary data to be read and/or written) or textual access
|
||||
(allowing text data to be read and/or written). You should also use :func:`io.open`
|
||||
for opening files instead of the built-in :func:`open` function as the :mod:`io`
|
||||
module is consistent from Python 2 to 3 while the built-in :func:`open` function
|
||||
is not (in Python 3 it's actually :func:`io.open`). Do not bother with the
|
||||
outdated practice of using :func:`codecs.open` as that's only necessary for
|
||||
keeping compatibility with Python 2.5.
|
||||
|
||||
The constructors of both ``str`` and ``bytes`` have different semantics for the
|
||||
same arguments between Python 2 and 3. Passing an integer to ``bytes`` in Python 2
|
||||
will give you the string representation of the integer: ``bytes(3) == '3'``.
|
||||
But in Python 3, an integer argument to ``bytes`` will give you a bytes object
|
||||
as long as the integer specified, filled with null bytes:
|
||||
``bytes(3) == b'\x00\x00\x00'``. A similar worry is necessary when passing a
|
||||
bytes object to ``str``. In Python 2 you just get the bytes object back:
|
||||
``str(b'3') == b'3'``. But in Python 3 you get the string representation of the
|
||||
bytes object: ``str(b'3') == "b'3'"``.
|
||||
|
||||
Finally, the indexing of binary data requires careful handling (slicing does
|
||||
**not** require any special handling). In Python 2,
|
||||
``b'123'[1] == b'2'`` while in Python 3 ``b'123'[1] == 50``. Because binary data
|
||||
is simply a collection of binary numbers, Python 3 returns the integer value for
|
||||
the byte you index on. But in Python 2 because ``bytes == str``, indexing
|
||||
returns a one-item slice of bytes. The six_ project has a function
|
||||
named ``six.indexbytes()`` which will return an integer like in Python 3:
|
||||
``six.indexbytes(b'123', 1)``.
|
||||
|
||||
To summarize:
|
||||
|
||||
#. Decide which of your APIs take text and which take binary data
|
||||
#. Make sure that your code that works with text also works with ``unicode`` and
|
||||
code for binary data works with ``bytes`` in Python 2 (see the table above
|
||||
for what methods you cannot use for each type)
|
||||
#. Mark all binary literals with a ``b`` prefix, textual literals with a ``u``
|
||||
prefix
|
||||
#. Decode binary data to text as soon as possible, encode text as binary data as
|
||||
late as possible
|
||||
#. Open files using :func:`io.open` and make sure to specify the ``b`` mode when
|
||||
appropriate
|
||||
#. Be careful when indexing into binary data
|
||||
|
||||
|
||||
Use feature detection instead of version detection
|
||||
++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
Inevitably you will have code that has to choose what to do based on what
|
||||
version of Python is running. The best way to do this is with feature detection
|
||||
of whether the version of Python you're running under supports what you need.
|
||||
If for some reason that doesn't work then you should make the version check be
|
||||
against Python 2 and not Python 3. To help explain this, let's look at an
|
||||
example.
|
||||
|
||||
Let's pretend that you need access to a feature of :mod:`importlib` that
|
||||
is available in Python's standard library since Python 3.3 and available for
|
||||
Python 2 through importlib2_ on PyPI. You might be tempted to write code to
|
||||
access e.g. the :mod:`importlib.abc` module by doing the following::
|
||||
|
||||
import sys
|
||||
|
||||
if sys.version_info[0] == 3:
|
||||
from importlib import abc
|
||||
else:
|
||||
from importlib2 import abc
|
||||
|
||||
The problem with this code is what happens when Python 4 comes out? It would
|
||||
be better to treat Python 2 as the exceptional case instead of Python 3 and
|
||||
assume that future Python versions will be more compatible with Python 3 than
|
||||
Python 2::
|
||||
|
||||
import sys
|
||||
|
||||
if sys.version_info[0] > 2:
|
||||
from importlib import abc
|
||||
else:
|
||||
from importlib2 import abc
|
||||
|
||||
The best solution, though, is to do no version detection at all and instead rely
|
||||
on feature detection. That avoids any potential issues of getting the version
|
||||
detection wrong and helps keep you future-compatible::
|
||||
|
||||
try:
|
||||
from importlib import abc
|
||||
except ImportError:
|
||||
from importlib2 import abc
|
||||
|
||||
|
||||
Prevent compatibility regressions
|
||||
---------------------------------
|
||||
|
||||
Once you have fully translated your code to be compatible with Python 3, you
|
||||
will want to make sure your code doesn't regress and stop working under
|
||||
Python 3. This is especially true if you have a dependency which is blocking you
|
||||
from actually running under Python 3 at the moment.
|
||||
|
||||
To help with staying compatible, any new modules you create should have
|
||||
at least the following block of code at the top of it::
|
||||
|
||||
from __future__ import absolute_import
|
||||
from __future__ import division
|
||||
from __future__ import print_function
|
||||
|
||||
You can also run Python 2 with the ``-3`` flag to be warned about various
|
||||
compatibility issues your code triggers during execution. If you turn warnings
|
||||
into errors with ``-Werror`` then you can make sure that you don't accidentally
|
||||
miss a warning.
|
||||
|
||||
You can also use the Pylint_ project and its ``--py3k`` flag to lint your code
|
||||
to receive warnings when your code begins to deviate from Python 3
|
||||
compatibility. This also prevents you from having to run Modernize_ or Futurize_
|
||||
over your code regularly to catch compatibility regressions. This does require
|
||||
you only support Python 2.7 and Python 3.4 or newer as that is Pylint's
|
||||
minimum Python version support.
|
||||
|
||||
|
||||
Check which dependencies block your transition
|
||||
----------------------------------------------
|
||||
|
||||
**After** you have made your code compatible with Python 3 you should begin to
|
||||
care about whether your dependencies have also been ported. The caniusepython3_
|
||||
project was created to help you determine which projects
|
||||
-- directly or indirectly -- are blocking you from supporting Python 3. There
|
||||
is both a command-line tool as well as a web interface at
|
||||
https://caniusepython3.com.
|
||||
|
||||
The project also provides code which you can integrate into your test suite so
|
||||
that you will have a failing test when you no longer have dependencies blocking
|
||||
you from using Python 3. This allows you to avoid having to manually check your
|
||||
dependencies and to be notified quickly when you can start running on Python 3.
|
||||
|
||||
|
||||
Update your ``setup.py`` file to denote Python 3 compatibility
|
||||
--------------------------------------------------------------
|
||||
|
||||
Once your code works under Python 3, you should update the classifiers in
|
||||
your ``setup.py`` to contain ``Programming Language :: Python :: 3`` and to not
|
||||
specify sole Python 2 support. This will tell anyone using your code that you
|
||||
support Python 2 **and** 3. Ideally you will also want to add classifiers for
|
||||
each major/minor version of Python you now support.
|
||||
|
||||
|
||||
Use continuous integration to stay compatible
|
||||
---------------------------------------------
|
||||
|
||||
Once you are able to fully run under Python 3 you will want to make sure your
|
||||
code always works under both Python 2 and 3. Probably the best tool for running
|
||||
your tests under multiple Python interpreters is tox_. You can then integrate
|
||||
tox with your continuous integration system so that you never accidentally break
|
||||
Python 2 or 3 support.
|
||||
|
||||
You may also want to use the ``-bb`` flag with the Python 3 interpreter to
|
||||
trigger an exception when you are comparing bytes to strings or bytes to an int
|
||||
(the latter is available starting in Python 3.5). By default type-differing
|
||||
comparisons simply return ``False``, but if you made a mistake in your
|
||||
separation of text/binary data handling or indexing on bytes you wouldn't easily
|
||||
find the mistake. This flag will raise an exception when these kinds of
|
||||
comparisons occur, making the mistake much easier to track down.
|
||||
|
||||
|
||||
Consider using optional static type checking
|
||||
--------------------------------------------
|
||||
|
||||
Another way to help port your code is to use a :term:`static type checker` like
|
||||
mypy_ or pytype_ on your code. These tools can be used to analyze your code as
|
||||
if it's being run under Python 2, then you can run the tool a second time as if
|
||||
your code is running under Python 3. By running a static type checker twice like
|
||||
this you can discover if you're e.g. misusing binary data type in one version
|
||||
of Python compared to another. If you add optional type hints to your code you
|
||||
can also explicitly state whether your APIs use textual or binary data, helping
|
||||
to make sure everything functions as expected in both versions of Python.
|
||||
|
||||
|
||||
.. _caniusepython3: https://pypi.org/project/caniusepython3
|
||||
.. _cheat sheet: https://python-future.org/compatible_idioms.html
|
||||
.. _coverage.py: https://pypi.org/project/coverage
|
||||
.. _Futurize: https://python-future.org/automatic_conversion.html
|
||||
.. _importlib2: https://pypi.org/project/importlib2
|
||||
.. _Modernize: https://python-modernize.readthedocs.io/
|
||||
.. _mypy: https://mypy-lang.org/
|
||||
.. _Porting to Python 3: http://python3porting.com/
|
||||
.. _Pylint: https://pypi.org/project/pylint
|
||||
|
||||
.. _Python 3 Q & A: https://ncoghlan-devs-python-notes.readthedocs.io/en/latest/python3/questions_and_answers.html
|
||||
|
||||
.. _pytype: https://github.com/google/pytype
|
||||
.. _python-future: https://python-future.org/
|
||||
.. _python-porting: https://mail.python.org/pipermail/python-porting/
|
||||
.. _six: https://pypi.org/project/six
|
||||
.. _tox: https://pypi.org/project/tox
|
||||
.. _trove classifier: https://pypi.org/classifiers
|
||||
|
||||
.. _Why Python 3 exists: https://snarky.ca/why-python-3-exists
|
||||
|
|
|
@ -158,15 +158,6 @@ For information about building Python's documentation, refer to `Doc/README.rst
|
|||
<https://github.com/python/cpython/blob/main/Doc/README.rst>`_.
|
||||
|
||||
|
||||
Converting From Python 2.x to 3.x
|
||||
---------------------------------
|
||||
|
||||
Significant backward incompatible changes were made for the release of Python
|
||||
3.0, which may cause programs written for Python 2 to fail when run with Python
|
||||
3. For more information about porting your code from Python 2 to Python 3, see
|
||||
the `Porting HOWTO <https://docs.python.org/3/howto/pyporting.html>`_.
|
||||
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
||||
|
|
Loading…
Reference in a new issue