Commit graph

1124 commits

Author SHA1 Message Date
Chenxi Mao 7703def37e
GH-102711: Fix warnings found by clang (#102712)
There are some warnings if build python via clang:

Parser/pegen.c:812:31: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
_PyPegen_clear_memo_statistics()
                              ^
                               void

Parser/pegen.c:820:29: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
_PyPegen_get_memo_statistics()
                            ^
                             void

Fix it to make clang happy.

Signed-off-by: Chenxi Mao <chenxi.mao@suse.com>
2023-03-28 10:52:22 +02:00
Max Bachmann c6858d1e7f
gh-102255: Improve build support for Windows API partitions (GH-102256)
Add `MS_WINDOWS_DESKTOP`, `MS_WINDOWS_APPS`, `MS_WINDOWS_SYSTEM` and `MS_WINDOWS_GAMES` preprocessor definitions to allow switching off functionality missing from particular API partitions ("partitions" are used in Windows to identify overlapping subsets of APIs).
CPython only officially supports `MS_WINDOWS_DESKTOP` and `MS_WINDOWS_SYSTEM` (APPS is included by normal desktop builds, but APPS without DESKTOP is not covered). Other configurations are a convenience for people building their own runtimes.
`MS_WINDOWS_GAMES` is for the Xbox subset of the Windows API, which is also available on client OS, but is restricted compared to `MS_WINDOWS_DESKTOP`. These restrictions may change over time, as they relate to the build headers rather than the OS support, and so we assume that Xbox builds will use the latest available version of the GDK.
2023-03-09 21:09:12 +00:00
Pablo Galindo Salgado f533f216e6
gh-102416: Do not memoize incorrectly loop rules in the parser (#102467) 2023-03-06 14:41:53 +01:00
Eric Snow 880437d4ec
gh-100227: Move _str_replace_inf to PyInterpreterState (gh-102333)
https://github.com/python/cpython/issues/100227
2023-02-28 14:16:39 -07:00
abel1502 448c7d154e
Fix some typos in asdl_c.py (GH-101757) 2023-02-09 21:10:46 -06:00
Mark Shannon feec49c407
GH-101578: Normalize the current exception (GH-101607)
* Make sure that the current exception is always normalized.

* Remove redundant type and traceback fields for the current exception.

* Add new API functions: PyErr_GetRaisedException, PyErr_SetRaisedException

* Add new API functions: PyException_GetArgs, PyException_SetArgs
2023-02-08 09:31:12 +00:00
Stepfen Shawn a1e051a237
gh-100940: Change "char *str" to "const char *str" in KeywordToken: It is an immutable string. (#100936) 2023-01-18 21:02:48 +00:00
Pablo Galindo Salgado 1de4395f62
gh-101046: Fix a potential memory leak in the parser when raising MemoryError (#101051) 2023-01-16 18:45:37 +00:00
Eric Snow 0415cf895f
gh-81057: Move the Cached Parser Dummy Name to _PyRuntimeState (#100277) 2022-12-16 13:48:03 +00:00
Eric Snow 91a8e002c2
gh-81057: Move More Globals to _PyRuntimeState (gh-100092)
https://github.com/python/cpython/issues/81057
2022-12-07 15:56:31 -07:00
Eric Snow d47ffeb9e3
gh-90110: Clean Up the C-analyzer Globals Lists (gh-100091)
https://github.com/python/cpython/issues/90110
2022-12-07 15:02:47 -07:00
Pablo Galindo Salgado 97e7004cfe
gh-100050: Fix an assertion error when raising unclosed parenthesis errors in the tokenizer (GH-100065)
Automerge-Triggered-By: GH:pablogsal
2022-12-06 15:09:56 -08:00
Pablo Galindo Salgado 417206a05c
gh-99891: Fix infinite recursion in the tokenizer when showing warnings (GH-99893)
Automerge-Triggered-By: GH:pablogsal
2022-11-30 03:36:06 -08:00
Lysandros Nikolaou 6d8da238cc
gh-90994: Improve error messages upon call arguments syntax errors (GH-96893) 2022-11-21 00:15:05 +01:00
Pablo Galindo Salgado e13d1d9dda
gh-99581: Fix a buffer overflow in the tokenizer when copying lines that fill the available buffer (#99605) 2022-11-20 20:20:03 +00:00
Lysandros Nikolaou 9c4232ae89
gh-99211: Point to except/except* on syntax errors when mixing them (GH-99215)
Automerge-Triggered-By: GH:lysnikolaou
2022-11-20 09:11:02 -08:00
Eric Snow 3c57971a2d
gh-81057: Move Globals in Core Code to _PyRuntimeState (gh-99496)
This is the first of several changes to consolidate non-object globals in core code.

https://github.com/python/cpython/issues/81057
2022-11-15 09:45:11 -07:00
Victor Stinner f13f466474
gh-99300: Use Py_NewRef() in Python/Python-ast.c (#99499)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Python/Python-ast.c.

Update Parser/asdl_c.py to regenerate code.
2022-11-15 10:29:56 +01:00
Eric Snow a088290f9d
gh-81057: Move Global Variables Holding Objects to _PyRuntimeState. (gh-99487)
This moves nearly all remaining object-holding globals in core code (other than static types).

https://github.com/python/cpython/issues/81057
2022-11-14 13:50:56 -07:00
Victor Stinner 4ce2a202c7
gh-99300: Use Py_NewRef() in Parser/ directory (#99330)
Replace Py_INCREF() with Py_NewRef() in C files of the Parser/
directory and in the PEG generator.
2022-11-10 15:30:05 +01:00
Victor Stinner 231d83b724
gh-99300: Use Py_NewRef() in Python/ directory (#99317)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in C files of the Python/ directory.

Update Parser/asdl_c.py to regenerate Python/Python-ast.c.
2022-11-10 11:23:36 +01:00
Irit Katriel 61b6c40b64
gh-99153: set location on SyntaxError for try with both except and except* (GH-99160) 2022-11-06 15:36:19 +00:00
Victor Stinner a60ddd31be
gh-98401: Invalid escape sequences emits SyntaxWarning (#99011)
A backslash-character pair that is not a valid escape sequence now
generates a SyntaxWarning, instead of DeprecationWarning.  For
example, re.compile("\d+\.\d+") now emits a SyntaxWarning ("\d" is an
invalid escape sequence), use raw strings for regular expression:
re.compile(r"\d+\.\d+"). In a future Python version, SyntaxError will
eventually be raised, instead of SyntaxWarning.

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated
in Python 3.11, now produce a SyntaxWarning, instead of
DeprecationWarning. In a future Python version they will be
eventually a SyntaxError.

codecs.escape_decode() and codecs.unicode_escape_decode() are left
unchanged: they still emit DeprecationWarning.

* The parser only emits SyntaxWarning for Python 3.12 (feature
  version), and still emits DeprecationWarning on older Python
  versions.
* Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and
  wasm_build.py.
2022-11-03 17:53:25 +01:00
Pablo Galindo Salgado 395d4285bf
gh-98931: Improve error message when the user types 'import x from y' instead of 'from y import x' (#98932) 2022-11-01 13:01:20 +00:00
Victor Stinner 1863302d61
gh-97669: Create Tools/build/ directory (#97963)
Create Tools/build/ directory. Move the following scripts from
Tools/scripts/ to Tools/build/:

* check_extension_modules.py
* deepfreeze.py
* freeze_modules.py
* generate_global_objects.py
* generate_levenshtein_examples.py
* generate_opcode_h.py
* generate_re_casefix.py
* generate_sre_constants.py
* generate_stdlib_module_names.py
* generate_token.py
* parse_html5_entities.py
* smelly.py
* stable_abi.py
* umarshal.py
* update_file.py
* verify_ensurepip_wheels.py

Update references to these scripts.
2022-10-17 12:01:00 +02:00
Lysandros Nikolaou 3de08ce8c1
gh-97997: Add col_offset field to tokenizer and use that for AST nodes (#98000) 2022-10-07 14:38:35 -07:00
Lysandros Nikolaou cbf0afd8a1
gh-97973: Return all necessary information from the tokenizer (GH-97984)
Right now, the tokenizer only returns type and two pointers to the start and end of the token.
This PR modifies the tokenizer to return the type and set all of the necessary information,
so that the parser does not have to this.
2022-10-06 16:07:17 -07:00
Mark Shannon 76449350b3
GH-91079: Decouple C stack overflow checks from Python recursion checks. (GH-96510) 2022-10-05 01:34:03 +01:00
Pablo Galindo Salgado aab01e3524
gh-96670: Raise SyntaxError when parsing NULL bytes (#97594) 2022-09-27 23:23:42 +01:00
Lysandros Nikolaou 7e36abbb78
gh-91210: Improve error message when non-default param follows default (GH-95933)
- Improve error message when parameter without a default follows one with a default
- Show same error message when positional-only params precede the default/non-default sequence
2022-09-17 10:09:28 -07:00
Matthias Görgens 81e36f350b
gh-96678: Fix UB of null pointer arithmetic (GH-96782)
Automerge-Triggered-By: GH:pablogsal
2022-09-13 06:14:35 -07:00
Michael Droettboom 8bc356a7dd
gh-96268: Fix loading invalid UTF-8 (#96270)
This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

It also fixes an off-by-one error introduced in 3.10 for the line number when the tokenizer reports bad UTF8.
2022-09-07 14:23:54 -07:00
Michael Droettboom 05692c67c5
gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (#96623) 2022-09-07 00:12:16 +01:00
Nikita Sobolev 2c7d2e8d46
gh-96587: Raise SyntaxError for PEP654 on older feature_version (#96588) 2022-09-05 17:54:09 +01:00
Gregory P. Smith 511ca94520
gh-95778: CVE-2020-10735: Prevent DoS by very large int() (#96499)
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.

This PR comes fresh from a pile of work done in our private PSRT security response team repo.

Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>
Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).

<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->

I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#). Much of that text wound up in the Issue. Backports PRs already exist. See the issue for links.
2022-09-02 09:35:08 -07:00
Shantanu a965db37f2
gh-94996: Disallow lambda pos only params with feature_version < (3, 8) (GH-95934) 2022-08-12 20:41:02 +02:00
Shantanu b5e3ea2862
gh-94996: Disallow parsing pos only params with feature_version < (3, 8) (GH-94997) 2022-08-12 19:27:50 +02:00
Christian Heimes b4c857d0fd
gh-95876: Fix format string in pegen error location code (#95877) 2022-08-11 09:55:57 +01:00
Honglin Zhu b946f529ef
gh-95355: Check tokens[0] after allocating memory (GH-95356)
#95355

Automerge-Triggered-By: GH:pablogsal
2022-07-28 03:00:34 -07:00
Pablo Galindo Salgado 0047447294
gh-95185: Check recursion depth in the AST constructor (#95186)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-07-24 15:58:52 +01:00
Shantanu 0daba82221
gh-94949: Disallow parsing parenthesised ctx mgr with old feature_version (#94950)
* gh-94949: Disallow parsing parenthesised ctx manager with old feature_version

* 📜🤖 Added by blurb_it.

* Allow it with feature_version=(3, 9) as well

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-18 22:10:49 +01:00
Shantanu ae0be5a53b
gh-94947: Disallow parsing walrus with feature_version < (3, 8) (#94948)
* gh-94947: Disallow parsing walrus with feature_version < (3, 8)

* oops, commit the parser

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-18 10:20:12 +01:00
Pablo Galindo Salgado 2e9da8e352
gh-94869: Fix the location in some expressions for multi-line f-string ast nodes (#94895) 2022-07-16 19:51:53 +01:00
Paul m. p. Peny bbb2ab70b6
[3.11] bpo-14916: interactive fd is not tied to stdin [type-bug] (#91469)
* bpo-14916: interactive fd is not always stdin

related to https://github.com/python/cpython/pull/31006 merged bugfix

following https://bugs.python.org/issue14916

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-16 09:35:19 +01:00
Pablo Galindo Salgado 36fcde61ba
gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin (#94386)
* gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>

* nitty nit

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2022-07-05 17:39:21 +01:00
wookie184 8c237a7a71
gh-94192: Fix error for dictionary literals with invalid expression as value. (#94304)
* Fix error for dictionary literals with invalid expression as value.

* Remove trailing whitespace
2022-06-26 19:42:47 +01:00
wookie184 2fc83ac3af
gh-92858: Improve error message for some suites with syntax error before ':' (#92894) 2022-06-23 17:31:09 +01:00
Victor Stinner 47e35625ff
gh-84623: Remove unused imports (#94132) 2022-06-22 19:14:27 +02:00
Victor Stinner cfb986a1a2
gh-93937: PyOS_StdioReadline() uses PyConfig.legacy_windows_stdio (#94024)
On Windows, PyOS_StdioReadline() now gets
PyConfig.legacy_windows_stdio from _PyOS_ReadlineTState, rather than
using the deprecated global Py_LegacyWindowsStdioFlag variable.

Fix also a compiler warning in Py_SetStandardStreamEncoding().
2022-06-20 16:10:47 +02:00
Serhiy Storchaka 6fd4c8ec77
gh-93741: Add private C API _PyImport_GetModuleAttrString() (GH-93742)
It combines PyImport_ImportModule() and PyObject_GetAttrString()
and saves 4-6 lines of code on every use.

Add also _PyImport_GetModuleAttr() which takes Python strings as arguments.
2022-06-14 07:15:26 +03:00