languages/cpython
languages/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython synced 2024-09-15 21:20:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

bpo-40275: Add import_helper submodule in test.support (GH-20794)

Browse source
This commit is contained in:
Hai Shi 2020-06-11 07:51:18 +08:00 committed by GitHub
parent 3b3b83c965
commit 7f888c7ef9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 359 additions and 330 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

219
Doc/library/test.rst
View file

@ -398,25 +398,6 @@ The :mod:`test.support` module defines the following constants:
The :mod:`test.support` module defines the following functions:
.. function:: forget(module_name)
Remove the module named *module_name* from ``sys.modules`` and delete any
byte-compiled files of the module.
.. function:: unload(name)
Delete *name* from ``sys.modules``.
.. function:: make_legacy_pyc(source)
Move a :pep:`3147`/:pep:`488` pyc file to its legacy pyc location and return the file
system path to the legacy pyc file. The *source* value is the file system
path to the source file. It does not need to exist, however the PEP
3147/488 pyc file must exist.
.. function:: is_resource_enabled(resource)
Return ``True`` if *resource* is enabled and available. The list of
@ -889,67 +870,6 @@ The :mod:`test.support` module defines the following functions:
Open *url*. If open fails, raises :exc:`TestFailed`.
.. function:: import_module(name, deprecated=False, *, required_on())
This function imports and returns the named module. Unlike a normal
import, this function raises :exc:`unittest.SkipTest` if the module
cannot be imported.
Module and package deprecation messages are suppressed during this import
if *deprecated* is ``True``. If a module is required on a platform but
optional for others, set *required_on* to an iterable of platform prefixes
which will be compared against :data:`sys.platform`.
.. versionadded:: 3.1
.. function:: import_fresh_module(name, fresh=(), blocked=(), deprecated=False)
This function imports and returns a fresh copy of the named Python module
by removing the named module from ``sys.modules`` before doing the import.
Note that unlike :func:`reload`, the original module is not affected by
this operation.
*fresh* is an iterable of additional module names that are also removed
from the ``sys.modules`` cache before doing the import.
*blocked* is an iterable of module names that are replaced with ``None``
in the module cache during the import to ensure that attempts to import
them raise :exc:`ImportError`.
The named module and any modules named in the *fresh* and *blocked*
parameters are saved before starting the import and then reinserted into
``sys.modules`` when the fresh import is complete.
Module and package deprecation messages are suppressed during this import
if *deprecated* is ``True``.
This function will raise :exc:`ImportError` if the named module cannot be
imported.
Example use::
# Get copies of the warnings module for testing without affecting the
# version being used by the rest of the test suite. One copy uses the
# C implementation, the other is forced to use the pure Python fallback
# implementation
py_warnings = import_fresh_module('warnings', blocked=['_warnings'])
c_warnings = import_fresh_module('warnings', fresh=['_warnings'])
.. versionadded:: 3.1
.. function:: modules_setup()
Return a copy of :data:`sys.modules`.
.. function:: modules_cleanup(oldmodules)
Remove modules except for *oldmodules* and ``encodings`` in order to
preserve internal cache.
.. function:: reap_children()
Use this at the end of ``test_main`` whenever sub-processes are started.
@ -1113,29 +1033,6 @@ The :mod:`test.support` module defines the following classes:
On both platforms, the old value is restored by :meth:`__exit__`.
.. class:: CleanImport(*module_names)
A context manager to force import to return a new module reference. This
is useful for testing module-level behaviors, such as the emission of a
DeprecationWarning on import. Example usage::
with CleanImport('foo'):
importlib.import_module('foo') # New reference.
.. class:: DirsOnSysPath(*paths)
A context manager to temporarily add directories to sys.path.
This makes a copy of :data:`sys.path`, appends any directories given
as positional arguments, then reverts :data:`sys.path` to the copied
settings when the context ends.
Note that *all* :data:`sys.path` modifications in the body of the
context manager, including replacement of the object,
will be reverted at the end of the block.
.. class:: SaveSignals()
Class to save and restore signal handlers registered by the Python signal
@ -1646,3 +1543,119 @@ The :mod:`test.support.os_helper` module provides support for os tests.
Call :func:`os.unlink` on *filename*. On Windows platforms, this is
wrapped with a wait loop that checks for the existence fo the file.
:mod:`test.support.import_helper` --- Utilities for import tests
================================================================
.. module:: test.support.import_helper
:synopsis: Support for import tests.
The :mod:`test.support.import_helper` module provides support for import tests.
.. versionadded:: 3.10
.. function:: forget(module_name)
Remove the module named *module_name* from ``sys.modules`` and delete any
byte-compiled files of the module.
.. function:: import_fresh_module(name, fresh=(), blocked=(), deprecated=False)
This function imports and returns a fresh copy of the named Python module
by removing the named module from ``sys.modules`` before doing the import.
Note that unlike :func:`reload`, the original module is not affected by
this operation.
*fresh* is an iterable of additional module names that are also removed
from the ``sys.modules`` cache before doing the import.
*blocked* is an iterable of module names that are replaced with ``None``
in the module cache during the import to ensure that attempts to import
them raise :exc:`ImportError`.
The named module and any modules named in the *fresh* and *blocked*
parameters are saved before starting the import and then reinserted into
``sys.modules`` when the fresh import is complete.
Module and package deprecation messages are suppressed during this import
if *deprecated* is ``True``.
This function will raise :exc:`ImportError` if the named module cannot be
imported.
Example use::
# Get copies of the warnings module for testing without affecting the
# version being used by the rest of the test suite. One copy uses the
# C implementation, the other is forced to use the pure Python fallback
# implementation
py_warnings = import_fresh_module('warnings', blocked=['_warnings'])
c_warnings = import_fresh_module('warnings', fresh=['_warnings'])
.. versionadded:: 3.1
.. function:: import_module(name, deprecated=False, *, required_on())
This function imports and returns the named module. Unlike a normal
import, this function raises :exc:`unittest.SkipTest` if the module
cannot be imported.
Module and package deprecation messages are suppressed during this import
if *deprecated* is ``True``. If a module is required on a platform but
optional for others, set *required_on* to an iterable of platform prefixes
which will be compared against :data:`sys.platform`.
.. versionadded:: 3.1
.. function:: modules_setup()
Return a copy of :data:`sys.modules`.
.. function:: modules_cleanup(oldmodules)
Remove modules except for *oldmodules* and ``encodings`` in order to
preserve internal cache.
.. function:: unload(name)
Delete *name* from ``sys.modules``.
.. function:: make_legacy_pyc(source)
Move a :pep:`3147`/:pep:`488` pyc file to its legacy pyc location and return the file
system path to the legacy pyc file. The *source* value is the file system
path to the source file. It does not need to exist, however the PEP
3147/488 pyc file must exist.
.. class:: CleanImport(*module_names)
A context manager to force import to return a new module reference. This
is useful for testing module-level behaviors, such as the emission of a
DeprecationWarning on import. Example usage::
with CleanImport('foo'):
importlib.import_module('foo') # New reference.
.. class:: DirsOnSysPath(*paths)
A context manager to temporarily add directories to sys.path.
This makes a copy of :data:`sys.path`, appends any directories given
as positional arguments, then reverts :data:`sys.path` to the copied
settings when the context ends.
Note that *all* :data:`sys.path` modifications in the body of the
context manager, including replacement of the object,
will be reverted at the end of the block.

232
Lib/test/support/__init__.py
View file

@ -8,8 +8,6 @@
import fnmatch
import functools
import glob
import importlib
import importlib.util
import os
import re
import stat
@ -21,6 +19,11 @@
import unittest
import warnings
from .import_helper import (
CleanImport, DirsOnSysPath, _ignore_deprecated_imports,
_save_and_block_module, _save_and_remove_module,
forget, import_fresh_module, import_module, make_legacy_pyc,
modules_cleanup, modules_setup, unload)
from .os_helper import (
FS_NONASCII, SAVEDCWD, TESTFN, TESTFN_NONASCII,
TESTFN_UNENCODABLE, TESTFN_UNDECODABLE,
@ -39,10 +42,6 @@
"PIPE_MAX_SIZE", "verbose", "max_memuse", "use_resources", "failfast",
# exceptions
"Error", "TestFailed", "TestDidNotRun", "ResourceDenied",
# imports
"import_module", "import_fresh_module", "CleanImport",
# modules
"unload", "forget",
# io
"record_original_stdout", "get_original_stdout", "captured_stdout",
"captured_stdin", "captured_stderr",
@ -132,22 +131,6 @@ class ResourceDenied(unittest.SkipTest):
and unexpected skips.
"""
@contextlib.contextmanager
def _ignore_deprecated_imports(ignore=True):
"""Context manager to suppress package and module deprecation
warnings when importing them.
If ignore is False, this context manager has no effect.
"""
if ignore:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", ".+ (module|package)",
DeprecationWarning)
yield
else:
yield
def ignore_warnings(*, category):
"""Decorator to suppress deprecation warnings.
@ -164,52 +147,6 @@ def wrapper(self, *args, **kwargs):
return decorator
def import_module(name, deprecated=False, *, required_on=()):
"""Import and return the module to be tested, raising SkipTest if
it is not available.
If deprecated is True, any module or package deprecation messages
will be suppressed. If a module is required on a platform but optional for
others, set required_on to an iterable of platform prefixes which will be
compared against sys.platform.
"""
with _ignore_deprecated_imports(deprecated):
try:
return importlib.import_module(name)
except ImportError as msg:
if sys.platform.startswith(tuple(required_on)):
raise
raise unittest.SkipTest(str(msg))
def _save_and_remove_module(name, orig_modules):
"""Helper function to save and remove a module from sys.modules
Raise ImportError if the module can't be imported.
"""
# try to import the module and raise an error if it can't be imported
if name not in sys.modules:
__import__(name)
del sys.modules[name]
for modname in list(sys.modules):
if modname == name or modname.startswith(name + '.'):
orig_modules[modname] = sys.modules[modname]
del sys.modules[modname]
def _save_and_block_module(name, orig_modules):
"""Helper function to save and block a module in sys.modules
Return True if the module was in sys.modules, False otherwise.
"""
saved = True
try:
orig_modules[name] = sys.modules[name]
except KeyError:
saved = False
sys.modules[name] = None
return saved
def anticipate_failure(condition):
"""Decorator to mark a test that is known to be broken in some cases
@ -240,56 +177,6 @@ def load_tests(*args):
return standard_tests
def import_fresh_module(name, fresh=(), blocked=(), deprecated=False):
"""Import and return a module, deliberately bypassing sys.modules.
This function imports and returns a fresh copy of the named Python module
by removing the named module from sys.modules before doing the import.
Note that unlike reload, the original module is not affected by
this operation.
*fresh* is an iterable of additional module names that are also removed
from the sys.modules cache before doing the import.
*blocked* is an iterable of module names that are replaced with None
in the module cache during the import to ensure that attempts to import
them raise ImportError.
The named module and any modules named in the *fresh* and *blocked*
parameters are saved before starting the import and then reinserted into
sys.modules when the fresh import is complete.
Module and package deprecation messages are suppressed during this import
if *deprecated* is True.
This function will raise ImportError if the named module cannot be
imported.
"""
# NOTE: test_heapq, test_json and test_warnings include extra sanity checks
# to make sure that this utility function is working as expected
with _ignore_deprecated_imports(deprecated):
# Keep track of modules saved for later restoration as well
# as those which just need a blocking entry removed
orig_modules = {}
names_to_remove = []
_save_and_remove_module(name, orig_modules)
try:
for fresh_name in fresh:
_save_and_remove_module(fresh_name, orig_modules)
for blocked_name in blocked:
if not _save_and_block_module(blocked_name, orig_modules):
names_to_remove.append(blocked_name)
fresh_module = importlib.import_module(name)
except ImportError:
fresh_module = None
finally:
for orig_name, module in orig_modules.items():
sys.modules[orig_name] = module
for name_to_remove in names_to_remove:
del sys.modules[name_to_remove]
return fresh_module
def get_attribute(obj, name):
"""Get an attribute, raising SkipTest if AttributeError is raised."""
try:
@ -318,12 +205,6 @@ def record_original_stdout(stdout):
def get_original_stdout():
return _original_stdout or sys.stdout
def unload(name):
try:
del sys.modules[name]
except KeyError:
pass
def _force_run(path, func, *args):
try:
@ -336,34 +217,6 @@ def _force_run(path, func, *args):
return func(*args)
def make_legacy_pyc(source):
"""Move a PEP 3147/488 pyc file to its legacy pyc location.
:param source: The file system path to the source file. The source file
does not need to exist, however the PEP 3147/488 pyc file must exist.
:return: The file system path to the legacy pyc file.
"""
pyc_file = importlib.util.cache_from_source(source)
up_one = os.path.dirname(os.path.abspath(source))
legacy_pyc = os.path.join(up_one, source + 'c')
os.rename(pyc_file, legacy_pyc)
return legacy_pyc
def forget(modname):
"""'Forget' a module was ever imported.
This removes the module from sys.modules and deletes any PEP 3147/488 or
legacy .pyc files.
"""
unload(modname)
for dirname in sys.path:
source = os.path.join(dirname, modname + '.py')
# It doesn't matter if they exist or not, unlink all possible
# combinations of PEP 3147/488 and legacy pyc files.
unlink(source + 'c')
for opt in ('', 1, 2):
unlink(importlib.util.cache_from_source(source, optimization=opt))
# Check whether a gui is actually available
def _is_gui_available():
if hasattr(_is_gui_available, 'result'):
@ -870,63 +723,6 @@ def check_no_resource_warning(testcase):
yield
class CleanImport(object):
"""Context manager to force import to return a new module reference.
This is useful for testing module-level behaviours, such as
the emission of a DeprecationWarning on import.
Use like this:
with CleanImport("foo"):
importlib.import_module("foo") # new reference
"""
def __init__(self, *module_names):
self.original_modules = sys.modules.copy()
for module_name in module_names:
if module_name in sys.modules:
module = sys.modules[module_name]
# It is possible that module_name is just an alias for
# another module (e.g. stub for modules renamed in 3.x).
# In that case, we also need delete the real module to clear
# the import cache.
if module.__name__ != module_name:
del sys.modules[module.__name__]
del sys.modules[module_name]
def __enter__(self):
return self
def __exit__(self, *ignore_exc):
sys.modules.update(self.original_modules)
class DirsOnSysPath(object):
"""Context manager to temporarily add directories to sys.path.
This makes a copy of sys.path, appends any directories given
as positional arguments, then reverts sys.path to the copied
settings when the context ends.
Note that *all* sys.path modifications in the body of the
context manager, including replacement of the object,
will be reverted at the end of the block.
"""
def __init__(self, *paths):
self.original_value = sys.path[:]
self.original_object = sys.path
sys.path.extend(paths)
def __enter__(self):
return self
def __exit__(self, *ignore_exc):
sys.path = self.original_object
sys.path[:] = self.original_value
class TransientResource(object):
"""Raise ResourceDenied if an exception is raised while the context manager
@ -1553,24 +1349,6 @@ def print_warning(msg):
for line in msg.splitlines():
print(f"Warning -- {line}", file=sys.__stderr__, flush=True)
def modules_setup():
return sys.modules.copy(),
def modules_cleanup(oldmodules):
# Encoders/decoders are registered permanently within the internal
# codec cache. If we destroy the corresponding modules their
# globals will be set to None which will trip up the cached functions.
encodings = [(k, v) for k, v in sys.modules.items()
if k.startswith('encodings.')]
sys.modules.clear()
sys.modules.update(encodings)
# XXX: This kind of problem can affect more than just encodings. In particular
# extension modules (such as _ssl) don't cope with reloading properly.
# Really, test modules should be cleaning out the test specific modules they
# know they added (ala test_runpy) rather than relying on this function (as
# test_importhooks and test_pkg do currently).
# Implicitly imported *real* modules should be left alone (see issue 10556).
sys.modules.update(oldmodules)
# Flag used by saved_test_environment of test.libregrtest.save_env,
# to check if a test modified the environment. The flag should be set to False

238
Lib/test/support/import_helper.py Normal file
View file

@ -0,0 +1,238 @@
import contextlib
import importlib
import importlib.util
import os
import sys
import unittest
import warnings
from .os_helper import unlink
@contextlib.contextmanager
def _ignore_deprecated_imports(ignore=True):
"""Context manager to suppress package and module deprecation
warnings when importing them.
If ignore is False, this context manager has no effect.
"""
if ignore:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", ".+ (module|package)",
DeprecationWarning)
yield
else:
yield
def unload(name):
try:
del sys.modules[name]
except KeyError:
pass
def forget(modname):
"""'Forget' a module was ever imported.
This removes the module from sys.modules and deletes any PEP 3147/488 or
legacy .pyc files.
"""
unload(modname)
for dirname in sys.path:
source = os.path.join(dirname, modname + '.py')
# It doesn't matter if they exist or not, unlink all possible
# combinations of PEP 3147/488 and legacy pyc files.
unlink(source + 'c')
for opt in ('', 1, 2):
unlink(importlib.util.cache_from_source(source, optimization=opt))
def make_legacy_pyc(source):
"""Move a PEP 3147/488 pyc file to its legacy pyc location.
:param source: The file system path to the source file. The source file
does not need to exist, however the PEP 3147/488 pyc file must exist.
:return: The file system path to the legacy pyc file.
"""
pyc_file = importlib.util.cache_from_source(source)
up_one = os.path.dirname(os.path.abspath(source))
legacy_pyc = os.path.join(up_one, source + 'c')
os.rename(pyc_file, legacy_pyc)
return legacy_pyc
def import_module(name, deprecated=False, *, required_on=()):
"""Import and return the module to be tested, raising SkipTest if
it is not available.
If deprecated is True, any module or package deprecation messages
will be suppressed. If a module is required on a platform but optional for
others, set required_on to an iterable of platform prefixes which will be
compared against sys.platform.
"""
with _ignore_deprecated_imports(deprecated):
try:
return importlib.import_module(name)
except ImportError as msg:
if sys.platform.startswith(tuple(required_on)):
raise
raise unittest.SkipTest(str(msg))
def _save_and_remove_module(name, orig_modules):
"""Helper function to save and remove a module from sys.modules
Raise ImportError if the module can't be imported.
"""
# try to import the module and raise an error if it can't be imported
if name not in sys.modules:
__import__(name)
del sys.modules[name]
for modname in list(sys.modules):
if modname == name or modname.startswith(name + '.'):
orig_modules[modname] = sys.modules[modname]
del sys.modules[modname]
def _save_and_block_module(name, orig_modules):
"""Helper function to save and block a module in sys.modules
Return True if the module was in sys.modules, False otherwise.
"""
saved = True
try:
orig_modules[name] = sys.modules[name]
except KeyError:
saved = False
sys.modules[name] = None
return saved
def import_fresh_module(name, fresh=(), blocked=(), deprecated=False):
"""Import and return a module, deliberately bypassing sys.modules.
This function imports and returns a fresh copy of the named Python module
by removing the named module from sys.modules before doing the import.
Note that unlike reload, the original module is not affected by
this operation.
*fresh* is an iterable of additional module names that are also removed
from the sys.modules cache before doing the import.
*blocked* is an iterable of module names that are replaced with None
in the module cache during the import to ensure that attempts to import
them raise ImportError.
The named module and any modules named in the *fresh* and *blocked*
parameters are saved before starting the import and then reinserted into
sys.modules when the fresh import is complete.
Module and package deprecation messages are suppressed during this import
if *deprecated* is True.
This function will raise ImportError if the named module cannot be
imported.
"""
# NOTE: test_heapq, test_json and test_warnings include extra sanity checks
# to make sure that this utility function is working as expected
with _ignore_deprecated_imports(deprecated):
# Keep track of modules saved for later restoration as well
# as those which just need a blocking entry removed
orig_modules = {}
names_to_remove = []
_save_and_remove_module(name, orig_modules)
try:
for fresh_name in fresh:
_save_and_remove_module(fresh_name, orig_modules)
for blocked_name in blocked:
if not _save_and_block_module(blocked_name, orig_modules):
names_to_remove.append(blocked_name)
fresh_module = importlib.import_module(name)
except ImportError:
fresh_module = None
finally:
for orig_name, module in orig_modules.items():
sys.modules[orig_name] = module
for name_to_remove in names_to_remove:
del sys.modules[name_to_remove]
return fresh_module
class CleanImport(object):
"""Context manager to force import to return a new module reference.
This is useful for testing module-level behaviours, such as
the emission of a DeprecationWarning on import.
Use like this:
with CleanImport("foo"):
importlib.import_module("foo") # new reference
"""
def __init__(self, *module_names):
self.original_modules = sys.modules.copy()
for module_name in module_names:
if module_name in sys.modules:
module = sys.modules[module_name]
# It is possible that module_name is just an alias for
# another module (e.g. stub for modules renamed in 3.x).
# In that case, we also need delete the real module to clear
# the import cache.
if module.__name__ != module_name:
del sys.modules[module.__name__]
del sys.modules[module_name]
def __enter__(self):
return self
def __exit__(self, *ignore_exc):
sys.modules.update(self.original_modules)
class DirsOnSysPath(object):
"""Context manager to temporarily add directories to sys.path.
This makes a copy of sys.path, appends any directories given
as positional arguments, then reverts sys.path to the copied
settings when the context ends.
Note that *all* sys.path modifications in the body of the
context manager, including replacement of the object,
will be reverted at the end of the block.
"""
def __init__(self, *paths):
self.original_value = sys.path[:]
self.original_object = sys.path
sys.path.extend(paths)
def __enter__(self):
return self
def __exit__(self, *ignore_exc):
sys.path = self.original_object
sys.path[:] = self.original_value
def modules_setup():
return sys.modules.copy(),
def modules_cleanup(oldmodules):
# Encoders/decoders are registered permanently within the internal
# codec cache. If we destroy the corresponding modules their
# globals will be set to None which will trip up the cached functions.
encodings = [(k, v) for k, v in sys.modules.items()
if k.startswith('encodings.')]
sys.modules.clear()
sys.modules.update(encodings)
# XXX: This kind of problem can affect more than just encodings.
# In particular extension modules (such as _ssl) don't cope
# with reloading properly. Really, test modules should be cleaning
# out the test specific modules they know they added (ala test_runpy)
# rather than relying on this function (as test_importhooks and test_pkg
# do currently). Implicitly imported *real* modules should be left alone
# (see issue 10556).
sys.modules.update(oldmodules)