cpython/Lib
Marta Gómez Macías 6715f91edc
gh-102856: Python tokenizer implementation for PEP 701 (#104323)
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.

As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-21 01:03:02 +01:00
..
__phello__
asyncio gh-104340: Suppress warning about unawaited exception for closed pipe stdin (#104586) 2023-05-17 16:45:11 -07:00
collections gh-91896: Revert some very noisy DeprecationWarnings for ByteString (#104424) 2023-05-13 09:55:35 +01:00
concurrent gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging (#94468) 2023-03-15 21:44:52 -07:00
ctypes Fix typo in _swappedbytes_ in ctypes comment (#102773) 2023-03-25 16:38:24 -07:00
curses gh-60436: fix curses textbox backspace/del (#103783) 2023-04-26 22:54:07 +02:00
dbm
email GH-103857: Deprecate utcnow and utcfromtimestamp (#103858) 2023-04-27 11:32:30 -06:00
encodings
ensurepip gh-103872: Update bundled pip version to 23.1.2 (gh-103873) 2023-04-26 10:22:25 +00:00
html
http gh-69152: add method get_proxy_response_headers to HTTPConnection class (#104248) 2023-05-16 06:20:30 +00:00
idlelib gh-104499: Fix typo. (#104598) 2023-05-18 02:35:46 +00:00
importlib gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
json
lib2to3
logging gh-103606: raise RuntimeError if config file is invalid or empty (#104701) 2023-05-20 22:26:49 +00:00
msilib
multiprocessing gh-61460: Stronger HMAC in multiprocessing (#20380) 2023-05-20 23:33:09 +00:00
pydoc_data gh-104301: Allow leading whitespace in disambiguated pdb statements (#104342) 2023-05-11 18:12:02 +01:00
re
site-packages
sqlite3 gh-102628: Fix sqlite3 CLI prompt in IDLE on Windows (#103945) 2023-04-27 21:22:26 +00:00
test gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
tkinter gh-103685: Fix tkinter.Menu.index() for Tk 8.7 (#103686) 2023-04-23 21:31:44 -04:00
tomllib
turtledemo Rewrite the turtledemo makeGraphFrame method (#104224) 2023-05-06 11:09:08 -04:00
unittest gh-68968: Correcting message display issue with assertEqual (#103937) 2023-05-04 16:37:17 -06:00
urllib gh-102153: Start stripping C0 control and space chars in urlsplit (#102508) 2023-05-17 01:49:20 -07:00
venv gh-95299: Stop installing setuptools as a part of ensurepip and venv (#101039) 2023-04-17 23:43:34 -05:00
wsgiref
xml
xmlrpc
zipfile gh-103861: Fix Zip64 extensions not being properly applied in some cases (#103863) 2023-05-16 00:43:44 -07:00
zoneinfo Fix typos in documentation and comments (GH-102374) 2023-03-02 07:26:49 -08:00
__future__.py
__hello__.py
_aix_support.py
_collections_abc.py gh-91896: Deprecate collections.abc.ByteString (#102096) 2023-05-04 09:39:33 -07:00
_compat_pickle.py
_compression.py
_markupbase.py
_osx_support.py
_py_abc.py
_pydatetime.py GH-84976: Re-introduce datetime.py and fix reprs 2023-05-03 03:09:45 -06:00
_pydecimal.py
_pyio.py
_pylong.py gh-102515: Remove unused imports in the Lib/ directory (#102516) 2023-03-08 11:45:38 +00:00
_sitebuiltins.py
_strptime.py GH-90750: Use datetime.fromisocalendar in _strptime (#103802) 2023-04-27 10:27:27 -06:00
_threading_local.py
_weakrefset.py
abc.py gh-87864: Use correct function definition syntax in the docs (#103312) 2023-04-11 16:50:25 +03:00
aifc.py
antigravity.py
argparse.py gh-92248: Deprecate type, choices, metavar parameters of argparse.BooleanOptionalAction (#103678) 2023-05-19 16:44:43 +00:00
ast.py gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
base64.py GH-102456: Fix docstring and getopt options for base64 (gh-102457) 2023-04-02 14:05:50 +09:00
bdb.py gh-101517: fix line number propagation in code generated for except* (#103550) 2023-04-24 21:58:51 +01:00
bisect.py GH-102833: Mention the key function in the docstrings (GH-103009) 2023-03-25 02:19:20 -05:00
bz2.py
calendar.py gh-103822: [Calendar] change return value to enum for day and month APIs (GH-103827) 2023-05-02 13:13:31 -07:00
cgi.py
cgitb.py
chunk.py
cmd.py
code.py gh-102778: Add sys.last_exc, deprecate sys.last_type, sys.last_value,sys.last_traceback (#102779) 2023-03-18 11:47:11 +00:00
codecs.py
codeop.py
colorsys.py
compileall.py
configparser.py
contextlib.py gh-103791: Make contextlib.suppress also act on exceptions within an ExceptionGroup (#103792) 2023-04-24 22:17:02 +00:00
contextvars.py
copy.py
copyreg.py
cProfile.py gh-103935: Use io.open_code() when executing code in trace and profile modules (GH-103947) 2023-04-27 20:29:35 +00:00
crypt.py
csv.py gh-67230: add quoting rules to csv module (GH-29469) 2023-04-12 15:32:30 -07:00
dataclasses.py gh-103000: Optimise dataclasses.asdict for the common case (#104364) 2023-05-10 22:43:51 +01:00
datetime.py GH-84976: Re-introduce datetime.py and fix reprs 2023-05-03 03:09:45 -06:00
decimal.py
difflib.py
dis.py GH-103963: Make dis display names of args for intrinsics opcodes (#104029) 2023-05-02 19:00:17 -07:00
doctest.py
enum.py gh-103596: [Enum] do not shadow mixed-in methods/attributes (GH-103600) 2023-04-18 16:19:23 -07:00
filecmp.py
fileinput.py
fnmatch.py
fractions.py
ftplib.py
functools.py gh-104600: Make function.__type_params__ writable (#104601) 2023-05-18 16:45:37 -07:00
genericpath.py
getopt.py
getpass.py
gettext.py
glob.py
graphlib.py
gzip.py gh-89550: Buffer GzipFile.write to reduce execution time by ~15% (#101251) 2023-05-08 17:55:59 +00:00
hashlib.py
heapq.py
hmac.py
imaplib.py
imghdr.py
inspect.py gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
io.py
ipaddress.py ipaddress: Remove non-existent ip_str param from docstring (#103461) 2023-04-14 00:44:10 -07:00
keyword.py gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
linecache.py
locale.py gh-74940: Allow fallback to UTF-8 encoding on systems with no locales installed (GH-14925) 2023-04-26 15:33:09 +09:00
lzma.py
mailbox.py
mailcap.py
mimetypes.py
modulefinder.py
netrc.py
nntplib.py
ntpath.py GH-103220: Fix ntpath.join() of partial UNC drive with trailing slash (GH-103221) 2023-04-11 17:26:45 +01:00
nturl2path.py
numbers.py
opcode.py gh-103865: add monitoring support to LOAD_SUPER_ATTR (#103866) 2023-05-16 10:29:00 -06:00
operator.py
optparse.py
os.py
pathlib.py GH-104484: Add case_sensitive argument to pathlib.PurePath.match() (GH-104565) 2023-05-18 18:59:31 +01:00
pdb.py gh-104301: Allow leading whitespace in disambiguated pdb statements (#104342) 2023-05-11 18:12:02 +01:00
pickle.py gh-102799: use exception instance instead of sys.exc_info() (#102885) 2023-03-31 11:23:02 +01:00
pickletools.py
pipes.py
pkgutil.py gh-97850: Deprecate find_loader and get_loader in pkgutil (GH-98520) 2023-05-03 16:11:54 -07:00
platform.py gh-103977: compile re expressions in platform.py only if required (#103981) 2023-04-30 08:36:19 -07:00
plistlib.py gh-101992: update pstlib module documentation (#102133) 2023-03-04 13:35:25 -08:00
poplib.py
posixpath.py
pprint.py
profile.py gh-103935: Use io.open_code() when executing code in trace and profile modules (GH-103947) 2023-04-27 20:29:35 +00:00
pstats.py
pty.py gh-96522: Fix deadlock in pty.spawn (#96639) 2023-05-19 13:22:43 +00:00
py_compile.py
pyclbr.py
pydoc.py pydoc.safeimport: Use importlib.import_module instead of __import__ (GH-103118) 2023-05-03 16:26:39 -07:00
queue.py
quopri.py
random.py Move binomialvariate() to a section for discrete distributions (GH-102955) 2023-03-23 12:10:12 -05:00
reprlib.py
rlcompleter.py
runpy.py gh-98040: Remove just the imp module (#98573) 2023-04-28 16:17:58 -07:00
sched.py
secrets.py
selectors.py
shelve.py
shlex.py
shutil.py gh-82814: fix shutil access error on WSL (#103790) 2023-04-25 00:45:38 +00:00
signal.py
site.py gh-102038: Skip a sometimes unnecessary stat in site.py (#102039) 2023-04-02 15:47:31 -07:00
smtplib.py
sndhdr.py
socket.py Fix broken link to MSDN (#102355) 2023-03-13 15:15:52 +02:00
socketserver.py gh-103673: Add missing ForkingUnixStreamServer and ForkingUnixDatagramServer socketservers (#103674) 2023-04-24 22:35:52 +00:00
sre_compile.py
sre_constants.py
sre_parse.py
ssl.py
stat.py
statistics.py GH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions (GH-102649) 2023-03-13 20:06:43 -05:00
string.py
stringprep.py
struct.py
subprocess.py gh-87474: Fix file descriptor leaks in subprocess.Popen (#96351) 2023-05-16 20:23:53 +00:00
sunau.py
symtable.py
sysconfig.py gh-102515: Remove unused imports in the Lib/ directory (#102516) 2023-03-08 11:45:38 +00:00
tabnanny.py gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
tarfile.py gh-102950: Implement PEP 706 – Filter for tarfile.extractall (#102953) 2023-04-24 10:58:06 +02:00
telnetlib.py
tempfile.py gh-51574: Make tempfile.mkdtemp() always return absolute paths (#94612) 2023-04-25 16:05:59 +00:00
textwrap.py
this.py
threading.py
timeit.py
token.py gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
tokenize.py gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
trace.py gh-103935: Use io.open_code() when executing code in trace and profile modules (GH-103947) 2023-04-27 20:29:35 +00:00
traceback.py gh-102799: replace sys.exc_info by sys.exception in inspect and traceback modules (#104032) 2023-05-01 18:19:47 +01:00
tracemalloc.py
tty.py gh-85984: New additions and improvements to the tty library. (#101832) 2023-05-19 18:13:30 +00:00
turtle.py gh-65772: Clean-up turtle module (#104218) 2023-05-06 11:04:41 -04:00
types.py gh-101688: Implement types.get_original_bases (#101827) 2023-04-23 20:24:30 +01:00
typing.py Improve readability of typing._ProtocolMeta.__instancecheck__ (#104649) 2023-05-19 14:30:02 +01:00
uu.py gh-99889: Fix directory traversal security flaw in uu.decode() (#104096) 2023-05-09 16:01:58 +00:00
uuid.py gh-104396: uuid.py to skip platform check for emscripten and wasi (gh-104397) 2023-05-12 07:45:55 +09:00
warnings.py
wave.py
weakref.py
webbrowser.py bpo-44844: Enable detection of Microsoft Edge browser in webbrowser module (GH-29908) 2023-04-04 17:00:03 +01:00
xdrlib.py
zipapp.py
zipimport.py