languages/cpython
languages/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython synced 2024-09-16 04:19:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
cpython/Lib
History
Joshua Herman 9e011e7c77
gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927) 
This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.

After porting we can see the direct impact on a multicore system.

Without this change:
  Running make test is 5 min 26 seconds
With this change:
  Running make test takes 3 min 39 seconds

That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.

The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.

Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.

---------

Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC]
2023-04-29 18:26:24 -07:00
..
__phello__
asyncio gh-83925: Make asyncio.subprocess communicate similar to non-asyncio (#18650) 2023-04-27 17:30:26 -07:00
collections gh-103092: Isolate _collections (#103093) 2023-04-12 18:21:28 +05:30
concurrent gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging (#94468) 2023-03-15 21:44:52 -07:00
ctypes Fix typo in _swappedbytes_ in ctypes comment (#102773) 2023-03-25 16:38:24 -07:00
curses gh-60436: fix curses textbox backspace/del (#103783) 2023-04-26 22:54:07 +02:00
dbm
email GH-103857: Deprecate utcnow and utcfromtimestamp (#103858) 2023-04-27 11:32:30 -06:00
encodings gh-98433: Fix quadratic time idna decoding. (#99092) 2022-11-07 16:54:41 -08:00
ensurepip gh-103872: Update bundled pip version to 23.1.2 (gh-103873) 2023-04-26 10:22:25 +00:00
html gh-100210: Correct the comment link for unescaping HTML (#100212) 2023-02-19 11:18:12 +01:00
http GH-103857: Deprecate utcnow and utcfromtimestamp (#103858) 2023-04-27 11:32:30 -06:00
idlelib Change 'dependant' to 'dependent' (#103745) 2023-04-24 09:40:30 -04:00
importlib gh-87729: specialize LOAD_SUPER_ATTR_METHOD (#103809) 2023-04-25 17:45:51 +00:00
json gh-96959: Update HTTP links which are redirected to HTTPS (GH-96961) 2022-09-24 14:38:53 +03:00
lib2to3 gh-54781: Move Lib/lib2to3/tests/ to Lib/test/test_lib2to3/ (#94049) 2022-06-21 15:21:22 +02:00
logging gh-103357: Add logging.Formatter defaults support to logging.config fileConfig and dictConfig (GH-103359) 2023-04-12 08:35:56 +01:00
msilib gh-91217: deprecate msilib (GH-91515) 2022-04-14 12:50:11 -07:00
multiprocessing gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551) 2023-02-03 15:20:46 -08:00
pydoc_data Python 3.12.0a7 2023-04-04 17:52:42 +02:00
re gh-91524: Speed up the regular expression substitution (#91525) 2022-10-23 15:57:30 -07:00
site-packages
sqlite3 gh-102628: Fix sqlite3 CLI prompt in IDLE on Windows (#103945) 2023-04-27 21:22:26 +00:00
test gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927) 2023-04-29 18:26:24 -07:00
tkinter gh-103685: Fix tkinter.Menu.index() for Tk 8.7 (#103686) 2023-04-23 21:31:44 -04:00
tomllib bpo-40059: Add tomllib (PEP-680) (GH-31498) 2022-03-08 09:26:13 +01:00
turtledemo gh-91212: Fixed flickering when the tracer is turned off (#95129) 2022-09-28 23:40:51 -07:00
unittest gh-62432: unittest runner: Exit code 5 if no tests were run (#102051) 2023-04-27 01:28:46 +00:00
urllib gh-81403: Fix for CacheFTPHandler in urllib (#13951) 2023-04-22 21:41:23 -07:00
venv gh-95299: Stop installing setuptools as a part of ensurepip and venv (#101039) 2023-04-17 23:43:34 -05:00
wsgiref bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
xml gh-83122: Deprecate testing element truth values in ElementTree (#31149) 2023-01-22 17:16:48 -08:00
xmlrpc bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
zipfile gh-86094: Add support for Unicode Path Extra Field in ZipFile (gh-102566) 2023-04-05 20:54:48 +09:00
zoneinfo Fix typos in documentation and comments (GH-102374) 2023-03-02 07:26:49 -08:00
__future__.py gh-93626: Set the release for __future__.annotations to None (GH-93628) 2022-07-05 10:46:39 +02:00
__hello__.py bpo-47084: Clear Unicode cached representations on finalization (GH-32032) 2022-03-22 13:53:51 +01:00
_aix_support.py gh-96305: Fix AIX build by avoiding subprocess during bootstrap (#96429) 2023-02-02 12:30:49 -08:00
_collections_abc.py gh-102721: Improve coverage of _collections_abc._CallableGenericAlias (#102722) 2023-03-16 14:47:30 +00:00
_compat_pickle.py
_compression.py
_markupbase.py
_osx_support.py
_py_abc.py
_pydecimal.py gh-86682: Adds sys._getframemodulename as an alternative to using _getframe (GH-99520) 2023-01-13 11:31:06 +00:00
_pyio.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
_pylong.py gh-102515: Remove unused imports in the Lib/ directory (#102516) 2023-03-08 11:45:38 +00:00
_sitebuiltins.py
_strptime.py GH-90750: Use datetime.fromisocalendar in _strptime (#103802) 2023-04-27 10:27:27 -06:00
_threading_local.py
_weakrefset.py bpo-26579: Add object.__getstate__(). (GH-2821) 2022-04-06 20:00:14 +03:00
abc.py gh-87864: Use correct function definition syntax in the docs (#103312) 2023-04-11 16:50:25 +03:00
aifc.py gh-47061: Deprecate chunk (GH-91419) 2022-04-11 15:02:41 -07:00
antigravity.py
argparse.py gh-101979: argparse: fix a bug where parentheses in metavar argument of add_argument() were dropped (#102318) 2023-03-05 06:54:33 -08:00
ast.py gh-103285: Rewrite _splitlines_no_ff to improve performance (#103307) 2023-04-23 23:03:49 -06:00
base64.py GH-102456: Fix docstring and getopt options for base64 (gh-102457) 2023-04-02 14:05:50 +09:00
bdb.py gh-101517: fix line number propagation in code generated for except* (#103550) 2023-04-24 21:58:51 +01:00
bisect.py GH-102833: Mention the key function in the docstrings (GH-103009) 2023-03-25 02:19:20 -05:00
bz2.py
calendar.py gh-103636: issue warning for deprecated calendar constants (#103833) 2023-04-29 01:16:46 -06:00
cgi.py bpo-47061: deprecate cgi and cgitb (GH-32410) 2022-04-08 17:15:35 -07:00
cgitb.py bpo-44712: Replace "type(literal)" with corresponding builtin types (GH-27294) 2022-05-08 17:10:11 +03:00
chunk.py gh-47061: Deprecate chunk (GH-91419) 2022-04-11 15:02:41 -07:00
cmd.py gh-67248: cmd: Sort miscellaneous help topics (#92254) 2022-05-03 21:36:52 -06:00
code.py gh-102778: Add sys.last_exc, deprecate sys.last_type, sys.last_value,sys.last_traceback (#102779) 2023-03-18 11:47:11 +00:00
codecs.py gh-51511: Note that codecs.open()'s encoding parameter affects automatic conversion to binary mode (#94370) 2022-10-21 16:01:05 -07:00
codeop.py gh-96052: codeop: fix handling compiler warnings in incomplete input (GH-96132) 2022-09-16 17:37:30 +03:00
colorsys.py
compileall.py GH-84559: Deprecate fork being the multiprocessing default. (#100618) 2023-02-02 15:50:35 -08:00
configparser.py gh-100520: Fix rst markup in configparser docstrings (#100524) 2022-12-26 17:08:49 +05:30
contextlib.py gh-103791: Make contextlib.suppress also act on exceptions within an ExceptionGroup (#103792) 2023-04-24 22:17:02 +00:00
contextvars.py
copy.py gh-100815: Normalize types module usage in copy module (#100816) 2023-01-07 21:29:53 +00:00
copyreg.py gh-99325: Remove unused NameError handling (#99326) 2022-11-11 09:56:57 +00:00
cProfile.py gh-103935: Use io.open_code() when executing code in trace and profile modules (GH-103947) 2023-04-27 20:29:35 +00:00
crypt.py gh-95231: Disable md5 & crypt modules if FIPS is enabled (GH-94742) 2022-08-15 07:48:07 -07:00
csv.py gh-67230: add quoting rules to csv module (GH-29469) 2023-04-12 15:32:30 -07:00
dataclasses.py gh-103449: Fix a bug in dataclass docstring generation (#103454) 2023-04-17 17:33:22 -06:00
datetime.py GH-103857: Deprecate utcnow and utcfromtimestamp (#103858) 2023-04-27 11:32:30 -06:00
decimal.py
difflib.py Correct method name typo (#91970) 2022-04-27 15:28:56 -06:00
dis.py GH-99944: Make dis display the value of oparg of KW_NAMES (#103856) 2023-04-26 19:00:36 +01:00
doctest.py gh-86682: Adds sys._getframemodulename as an alternative to using _getframe (GH-99520) 2023-01-13 11:31:06 +00:00
enum.py gh-103596: [Enum] do not shadow mixed-in methods/attributes (GH-103600) 2023-04-18 16:19:23 -07:00
filecmp.py gh-93991: Use boolean instead of 0/1 for condition check (GH-93992) 2022-06-19 07:12:59 -07:00
fileinput.py gh-101961 fileinput.hookcompressed should not set the encoding value for the binary mode (gh-102068) 2023-02-21 12:10:29 +09:00
fnmatch.py gh-89973: Fix re.error in the fnmatch module. (GH-93072) 2022-06-05 11:46:29 +03:00
fractions.py gh-101825: Clarify that as_integer_ratio() output is always normalized (#101843) 2023-02-27 19:11:28 +00:00
ftplib.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
functools.py gh-87634: remove locking from functools.cached_property (GH-101890) 2023-02-22 17:49:22 -08:00
genericpath.py gh-101196: Make isdir/isfile/exists faster on Windows (GH-101324) 2023-02-08 14:34:24 +00:00
getopt.py bpo-44712: Replace "type(literal)" with corresponding builtin types (GH-27294) 2022-05-08 17:10:11 +03:00
getpass.py
gettext.py
glob.py
graphlib.py
gzip.py gh-95534: Improve gzip reading speed by 10% (#97664) 2022-10-16 19:10:58 -07:00
hashlib.py gh-99108: Refactor _sha256 & _sha512 into _sha2. (#101924) 2023-02-15 22:08:20 -08:00
heapq.py Update: usage doc for heappushpop (GH-91451) 2022-04-17 23:12:33 -05:00
hmac.py
imaplib.py gh-94172: Remove keyfile, certfile and check_hostname parameters (#94173) 2022-11-03 18:32:25 +01:00
imghdr.py gh-85455: Add missing doc strings and improve docs (#21573) 2022-10-15 09:31:06 -04:00
inspect.py gh-103556: [inspect.Signature] disallow pos-or-kw params without default after pos-only with default (#103557) 2023-04-21 23:45:10 -06:00
io.py gh-94169: Remove deprecated io.OpenWrapper (#94170) 2022-06-24 08:46:53 +02:00
ipaddress.py ipaddress: Remove non-existent ip_str param from docstring (#103461) 2023-04-14 00:44:10 -07:00
keyword.py
linecache.py gh-92336: linecache.getline should not raise exceptions on decoding errors (GH-94410) 2022-06-30 10:18:18 +01:00
locale.py gh-74940: Allow fallback to UTF-8 encoding on systems with no locales installed (GH-14925) 2023-04-26 15:33:09 +09:00
lzma.py
mailbox.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
mailcap.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
mimetypes.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
modulefinder.py
netrc.py netrc: Remove unused "import shlex" (#93311) 2022-06-03 20:14:58 -07:00
nntplib.py gh-91217: deprecate nntplib (GH-91543) 2022-04-15 12:32:56 -07:00
ntpath.py GH-103220: Fix ntpath.join() of partial UNC drive with trailing slash (GH-103221) 2023-04-11 17:26:45 +01:00
nturl2path.py
numbers.py gh-68163: Correct conversion of Rational instances to float (GH-25619) 2022-09-04 13:15:59 +01:00
opcode.py gh-87729: specialize LOAD_SUPER_ATTR_METHOD (#103809) 2023-04-25 17:45:51 +00:00
operator.py
optparse.py
os.py gh-89727: Improve os.walk complexity (#100671) 2023-01-02 13:41:19 -08:00
pathlib.py GH-78079: Fix UNC device path root normalization in pathlib (GH-102003) 2023-04-14 21:55:41 +01:00
pdb.py gh-103578: Fix pdb reading code with non-utf8 encoding (#103581) 2023-04-25 23:04:51 -06:00
pickle.py gh-102799: use exception instance instead of sys.exc_info() (#102885) 2023-03-31 11:23:02 +01:00
pickletools.py
pipes.py gh-91217: deprecate-pipes (GH-91779) 2022-04-21 19:28:34 -07:00
pkgutil.py gh-98040: Remove just the imp module (#98573) 2023-04-28 16:17:58 -07:00
platform.py gh-102491: Remove IronPython version check in sys_version (#102492) 2023-03-19 15:52:47 -07:00
plistlib.py gh-101992: update pstlib module documentation (#102133) 2023-03-04 13:35:25 -08:00
poplib.py gh-94172: Remove keyfile, certfile and check_hostname parameters (#94173) 2022-11-03 18:32:25 +01:00
posixpath.py gh-101196: Make isdir/isfile/exists faster on Windows (GH-101324) 2023-02-08 14:34:24 +00:00
pprint.py gh-92546: Move pprint benchmark into pyperformance (GH-94613) 2022-07-25 11:30:13 -07:00
profile.py gh-103935: Use io.open_code() when executing code in trace and profile modules (GH-103947) 2023-04-27 20:29:35 +00:00
pstats.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
pty.py gh-85984: Remove legacy Lib/pty.py code. (#92365) 2023-02-08 17:00:17 -08:00
py_compile.py
pyclbr.py
pydoc.py gh-98040: Remove just the imp module (#98573) 2023-04-28 16:17:58 -07:00
queue.py gh-90879: Fix missing parameter for put_nowait() (GH-91514) 2022-04-14 17:23:57 +09:00
quopri.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
random.py Move binomialvariate() to a section for discrete distributions (GH-102955) 2023-03-23 12:10:12 -05:00
reprlib.py gh-92734: Add indentation feature to reprlib.Repr (GH-92735) 2022-09-08 13:51:44 -05:00
rlcompleter.py gh-92345: Import rlcompleter before sys.path is extended (#92346) 2022-05-05 21:24:16 +02:00
runpy.py gh-98040: Remove just the imp module (#98573) 2023-04-28 16:17:58 -07:00
sched.py
secrets.py gh-99305: Speed up secrets.token_hex() ~2x (#99306) 2022-11-11 14:45:24 -08:00
selectors.py
shelve.py
shlex.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
shutil.py gh-82814: fix shutil access error on WSL (#103790) 2023-04-25 00:45:38 +00:00
signal.py
site.py gh-102038: Skip a sometimes unnecessary stat in site.py (#102039) 2023-04-02 15:47:31 -07:00
smtplib.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
sndhdr.py gh-85525: Indicate supported sound header formats (#21575) 2022-10-15 09:30:05 -04:00
socket.py Fix broken link to MSDN (#102355) 2023-03-13 15:15:52 +02:00
socketserver.py gh-103673: Add missing ForkingUnixStreamServer and ForkingUnixDatagramServer socketservers (#103674) 2023-04-24 22:35:52 +00:00
sre_compile.py bpo-47152: Convert the re module into a package (GH-32177) 2022-04-02 11:35:13 +03:00
sre_constants.py bpo-47152: Convert the re module into a package (GH-32177) 2022-04-02 11:35:13 +03:00
sre_parse.py bpo-47152: Convert the re module into a package (GH-32177) 2022-04-02 11:35:13 +03:00
ssl.py gh-94199: Remove the ssl.wrap_socket() function (#94203) 2022-07-08 15:20:15 +02:00
stat.py
statistics.py GH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions (GH-102649) 2023-03-13 20:06:43 -05:00
string.py
stringprep.py
struct.py
subprocess.py gh-101283: Fix use of unbound variable (GH-101712) 2023-02-08 23:38:56 +00:00
sunau.py bpo-44712: Replace "type(literal)" with corresponding builtin types (GH-27294) 2022-05-08 17:10:11 +03:00
symtable.py Change list to view object (#93661) 2022-06-11 11:54:31 +01:00
sysconfig.py gh-102515: Remove unused imports in the Lib/ directory (#102516) 2023-03-08 11:45:38 +00:00
tabnanny.py gh-63301: Set exit code when tabnanny CLI exits on error (#7699) 2023-03-04 13:32:13 -08:00
tarfile.py gh-102950: Implement PEP 706 – Filter for tarfile.extractall (#102953) 2023-04-24 10:58:06 +02:00
telnetlib.py gh-91217: deprecate telnetlib (GH-91958) 2022-04-26 10:45:08 -07:00
tempfile.py gh-51574: Make tempfile.mkdtemp() always return absolute paths (#94612) 2023-04-25 16:05:59 +00:00
textwrap.py
this.py
threading.py gh-100228: Warn from os.fork() if other threads exist. (#100229) 2022-12-29 14:41:39 -08:00
timeit.py gh-90085: Remove vestigial -t and -c timeit options (#94941) 2022-10-07 15:38:20 -07:00
token.py gh-102856: Initial implementation of PEP 701 (#102855) 2023-04-19 11:18:16 -05:00
tokenize.py
trace.py gh-103935: Use io.open_code() when executing code in trace and profile modules (GH-103947) 2023-04-27 20:29:35 +00:00
traceback.py gh-102778: Add sys.last_exc, deprecate sys.last_type, sys.last_value,sys.last_traceback (#102779) 2023-03-18 11:47:11 +00:00
tracemalloc.py
tty.py
turtle.py gh-101100: Fix Sphinx warnings in turtle module (#102340) 2023-03-13 11:24:52 +02:00
types.py gh-101688: Implement types.get_original_bases (#101827) 2023-04-23 20:24:30 +01:00
typing.py GH-103699: Add __orig_bases__ to various typing classes (#103698) 2023-04-23 08:33:39 -06:00
uu.py gh-91217: deprecate uu (GH-92009) 2022-04-27 20:26:33 -07:00
uuid.py Update MAC_address wikipedia URL in comments (#29019) 2023-04-26 14:27:03 -06:00
warnings.py gh-39615: Add warnings.warn() skip_file_prefixes support (#100840) 2023-01-27 18:35:14 -08:00
wave.py gh-77171: Fixes SubFormat check to compare the entire value. Add docs (GH-97509) 2022-09-23 16:08:21 +01:00
weakref.py Remove unnecessary registration of weakref.WeakSet to _collections_abc.Set (GH-32211) 2022-03-31 09:11:35 -05:00
webbrowser.py bpo-44844: Enable detection of Microsoft Edge browser in webbrowser module (GH-29908) 2023-04-04 17:00:03 +01:00
xdrlib.py bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
zipapp.py bpo-46951: Order contents of zipapps (GH-31713) 2022-05-27 19:04:29 +03:00
zipimport.py GH-87235: Make sure "python /dev/fd/9 9</path/to/script.py" works on macOS (#99768) 2022-11-27 11:56:14 +01:00