Commit graph

109343 commits

Author SHA1 Message Date
Victor Stinner 57364ce34e
bpo-43244: Remove parser_interface.h header file (GH-25001)
Remove parser functions using the "struct _mod" type, because the
AST C API was removed:

* PyParser_ASTFromFile()
* PyParser_ASTFromFileObject()
* PyParser_ASTFromFilename()
* PyParser_ASTFromString()
* PyParser_ASTFromStringObject()

These functions were undocumented and excluded from the limited C
API.

Add pycore_parser.h internal header file. Rename functions:

* PyParser_ASTFromFileObject() => _PyParser_ASTFromFile()
* PyParser_ASTFromStringObject() => _PyParser_ASTFromString()

These functions are no longer exported (replace PyAPI_FUNC() with
extern).

Remove also _PyPegen_run_parser_from_file() function. Update
test_peg_generator to use _PyPegen_run_parser_from_file_pointer()
instead.
2021-03-24 01:29:09 +01:00
Pablo Galindo a054f6b2b1
bpo-43452: Document the PyType_Lookup optimizations in the What's New for 3.10 (GH-24949) 2021-03-24 00:04:52 +00:00
Pablo Galindo d9692027f4
bpo-31861: Complete the C-API docs for PyObject_GetAiter and PyAiter_Check (GH-25004) 2021-03-23 23:57:03 +00:00
Victor Stinner a81fca6ec8
bpo-43244: Add pycore_compile.h header file (GH-25000)
Remove the compiler functions using "struct _mod" type, because the
public AST C API was removed:

* PyAST_Compile()
* PyAST_CompileEx()
* PyAST_CompileObject()
* PyFuture_FromAST()
* PyFuture_FromASTObject()

These functions were undocumented and excluded from the limited C API.

Rename functions:

* PyAST_CompileObject() => _PyAST_Compile()
* PyFuture_FromASTObject() => _PyFuture_FromAST()

Moreover, _PyFuture_FromAST() is no longer exported (replace
PyAPI_FUNC() with extern). _PyAST_Compile() remains exported for
test_peg_generator.

Remove also compatibility functions:

* PyAST_Compile()
* PyAST_CompileEx()
* PyFuture_FromAST()
2021-03-24 00:51:50 +01:00
Joshua Bronson f0a6fde882
bpo-31861: Add aiter and anext to builtins (#23847)
Co-authored-by: jab <jab@users.noreply.github.com>
Co-authored-by: Daniel Pope <mauve@mauveweb.co.uk>
Co-authored-by: Justin Wang <justin39@gmail.com>
2021-03-23 15:47:21 -07:00
Victor Stinner 94faa0724f
bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933)
These functions were undocumented and excluded from the limited C
API.

Most names defined by these header files were not prefixed by "Py"
and so could create names conflicts. For example, Python-ast.h
defined a "Yield" macro which was conflict with the "Yield" name used
by the Windows <winbase.h> header.

Use the Python ast module instead.

* Move Include/asdl.h to Include/internal/pycore_asdl.h.
* Move Include/Python-ast.h to Include/internal/pycore_ast.h.
* Remove ast.h header file.
* pycore_symtable.h no longer includes Python-ast.h.
2021-03-23 20:47:40 +01:00
Victor Stinner cd27af70d5
bpo-41718: Update runpy startup time What's New (GH-24998) 2021-03-23 20:22:40 +01:00
Victor Stinner bd9154a495
bpo-41718: runpy now imports pkgutil in functions (GH-24996)
Reduce the number of modules imported by "python3 -m module".

The runpy module no longer imports at startup (in the module body),
but only in functions using it: _get_code_from_file() and run_path().
2021-03-23 19:22:57 +01:00
Victor Stinner d72e8d4875
bpo-41718: subprocess imports grp and pwd on demand (GH-24987)
The shutil and subprocess modules now only import the grp and pwd
modules when they are needed, which is not the case by default in
subprocess.
2021-03-23 17:42:51 +01:00
Brett Cannon 76b5d714e4
Clarify attribute docs on types.ModuleType (GH-24974) 2021-03-23 08:25:39 -07:00
Victor Stinner 9feae41c4f
bpo-41718: libregrtest avoids importing datetime (GH-24985)
* libregrtest reimplements datetime.timedelta.__str__()
* support.testresult only imports datetime if USE_XML is true.
2021-03-23 01:40:31 +01:00
Victor Stinner 30793e81bd
bpo-41718: Disable support.testresult XML output by default (GH-24982)
RegressionTestResult.USE_XML must now be set to True to get the JUnit
XML output.

Reduce the number of imports when --junit-xml=FILE option is not
used: 153 => 144 (-9).
2021-03-23 01:11:31 +01:00
Victor Stinner 0473fb2229
bpo-41718: libregrtest runtest avoids import_helper (GH-24983)
Inline import_helper.unload() in libregrtest.runtest to avoid one
import.
2021-03-23 01:08:49 +01:00
Victor Stinner 10417dd15f
bpo-41718: Reduce libregrtest runtest imports (GH-24980)
Move clear_caches() from libregrtest.refleak to libregrtest.utils to
avoid importing libregrtest.refleak when it's not needed.

clear_caches() now only calls re.purge() if 're' is in sys.modules.
2021-03-23 00:17:05 +01:00
Victor Stinner 532e063fc2
bpo-41718: regrtest saved_test_environment avoids imports (GH-24934)
Reduce the number of modules imported by libregrtest.

saved_test_environment no longer imports modules at startup, but try
to get them from sys.modules. If an module is missing, skip the test.
It also sets directly support.environment_altered.

runtest() now now two saved_test_environment instances: one before
importing the test module, one after importing it.

Remove imports from test.libregrtest.save_env:

* asyncio
* logging
* multiprocessing
* shutil
* sysconfig
* urllib.request
* warnings

When a test method imports a module (ex: warnings) and the test
has a side effect (ex: add a warnings filter), the side effect is not
detected, because the module was not imported when Python
enters the saved_test_environment context manager.
2021-03-22 23:52:13 +01:00
Pablo Galindo 96eeff5162
bpo-43555: Report the column offset for invalid line continuation character (GH-24939) 2021-03-22 17:28:11 +00:00
Pablo Galindo 123ff266cd
bpo-43591: Fix error location in interactive mode for errors at the end of the line (GH-24973)
Co-authored-by: Erlend Egeberg Aasland
2021-03-22 16:24:39 +00:00
Victor Stinner 39f643614d
Revert "bpo-40521: Make dtoa bigint free list per-interpreter (GH-24821)" (GH-24964)
This reverts commit 5bd1059184.
2021-03-22 11:58:59 +01:00
Dong-hee Na 86883d40e9
bpo-43575: Use PEP 590 vectorcall to speed up map() (GH-24955) 2021-03-22 19:01:14 +09:00
junyixie 88d9983b56
bpo-43551: Fix PyImport_Import() for subinterpreters (GH-24929)
Avoid static variables.
2021-03-22 10:47:10 +01:00
Hai Shi 56f031ec52
bpo-35134: Add include/cpython/compile.h (GH-24922)
Move C API excluded from the limited C API from Include/compile.h
to a new Include/cpython/compile.h header file.
2021-03-22 09:32:11 +01:00
Sergey B Kirpichev 690aca7811
bpo-43420: Simple optimizations for Fraction's arithmetics (GH-24779)
bpo-43420: Implement standard transformations in + - * / that can often reduce the size of intermediate integers needed. For rationals with large components, this can yield dramatic speed improvements, but for small rationals can run 10-20% slower, due to increased fixed overheads in the longer-winded code. If those slowdowns turn out to be a problem, see the PR discussion for low-level implementation tricks that could cut other fixed overheads.

Co-authored-by: Tim Peters <tim.peters@gmail.com>
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
2021-03-21 21:30:55 -05:00
Jason R. Coombs 9a50ef43e4
bpo-43569: Add test/test_importlib/namespacedata01 to TESTSUBDIRS (GH-24952) 2021-03-21 18:26:45 +01:00
Antoine Pitrou cdddc2b742
bpo-43422: Revert _decimal C API addition (GH-24960)
Stefan Krah requested the reversal of these (unreleased) changes, quoting him:
> The capsule API does not meet my testing standards, since I've focused
on the upstream mpdecimal in the last couple of months.
> Additionally, I'd like to refine the API, perhaps together with the
Arrow community.

Automerge-Triggered-By: GH:pitrou
2021-03-21 09:27:54 -07:00
Christian Heimes 77cde5042a
bpo-43577: Fix deadlock with SSLContext._msg_callback and sni_callback (GH-24957)
OpenSSL copies the internal message callback from SSL_CTX->msg_callback to
SSL->msg_callback. SSL_set_SSL_CTX() does not update SSL->msg_callback
to use the callback value of the new context.

PySSL_set_context() now resets the callback and _PySSL_msg_callback()
resets thread state in error path.

Signed-off-by: Christian Heimes <christian@python.org>
2021-03-21 16:13:09 +01:00
Ilya Stepin 20a5b7e986
bpo-43542: Add heif/heic formats in mimetypes (GH-24917)
* bpo-43542: Add heif/heic formats in mimetypes

Add HEIF and HEIC format to list of media types. It has IANA registration.

IANA: https://www.iana.org/assignments/media-types/image/heic
HEIF Github: https://github.com/nokiatech/heif

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2021-03-20 16:06:27 -07:00
Rui Cunha b05b48dd7a
bpo-43571: Add IPPROTO_MPTCP macro (GH-24946)
Add IPPROTO_MPTCP constant when defined in the system headers.
2021-03-20 15:04:56 -07:00
Antoine Pitrou 66c8adfa27
bpo-43517 (followup): Install additional test directory (GH-24950)
Should fix some CI failures on buildbots that test an installed version of Python.

Automerge-Triggered-By: GH:pitrou
2021-03-20 13:45:58 -07:00
Dino Viehland ee48c7d541
bpo-43452: Micro-optimizations to PyType_Lookup (GH-24804)
The common case going through _PyType_Lookup is to have a cache hit. There are some small tweaks that can make this a little cheaper:

* The name field identity is used for a cache hit and is kept alive by the cache. So there's no need to read the hash code o the name - instead, the address can be used as the hash.

*  There's no need to check if the name is cachable on the lookup either, it probably is, and if it is, it'll be in the cache.

*  If we clear the version tag when invalidating a type then we don't actually need to check for a valid version tag bit.
2021-03-20 19:12:05 +00:00
Antoine Pitrou 2fd16ef406
bpo-43517: Fix false positive in detection of circular imports (#24895) 2021-03-20 20:07:44 +01:00
Skip Montanaro 7cb033c423
bpo-43494: Make some minor changes to lnotab notes (GH-24861)
This cleanup makes no substantive changes.
2021-03-19 19:10:54 -04:00
Mark Shannon 148bc05844
Mention that code.co_lnotab is deprecated in what's new for 3.10. (#24902) 2021-03-19 17:30:24 +00:00
Victor Stinner 28ad12f8fe
bpo-43244: Remove symtable.h header file (GH-24910)
Rename Include/symtable.h to to Include/internal/pycore_symtable.h,
don't export symbols anymore (replace PyAPI_FUNC and PyAPI_DATA with
extern) and rename functions:

* PyST_GetScope() to _PyST_GetScope()
* PySymtable_BuildObject() to _PySymtable_Build()
* PySymtable_Free() to _PySymtable_Free()

Remove PySymtable_Build(), Py_SymtableString() and
Py_SymtableStringObject() functions.

The Py_SymtableString() function was part the stable ABI by mistake
but it could not be used, since the symtable.h header file was
excluded from the limited C API.

The Python symtable module remains available and is unchanged.
2021-03-19 12:41:49 +01:00
Christian Heimes 32eba61ea4
bpo-43466: Add --with-openssl-rpath configure option (GH-24820) 2021-03-19 10:29:25 +01:00
Ikko Ashimine ff8c77fe96
Fix a typo in c-analyzer (GH-24468) 2021-03-19 16:38:12 +09:00
Christian Heimes f6c6b5821b
bpo-41561: Add workaround for Ubuntu's custom security level (GH-24915)
Ubuntu 20.04 comes with a patched OpenSSL 1.1.1. Default security level
2 blocks TLS 1.0 and 1.1 connections. Regular OpenSSL 1.1.1 builds allow
TLS 1.0 and 1.1 on security level 2.

See: 
See: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878
See: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1917625
Signed-off-by: Christian Heimes <christian@python.org>
2021-03-18 15:06:50 -07:00
Kodi Arfer 08ff4369af
bpo-43521: Allow ast.unparse with empty sets and NaN (GH-24897)
Automerge-Triggered-By: GH:pablogsal
2021-03-18 10:36:06 -07:00
Victor Stinner eec8e61992
bpo-43244: Remove the PyAST_Validate() function (GH-24911)
Remove the PyAST_Validate() function. It is no longer possible to
build a AST object (mod_ty type) with the public C API. The function
was already excluded from the limited C API (PEP 384).

Rename PyAST_Validate() function to _PyAST_Validate(), move it to the
internal C API, and don't export it anymore (replace PyAPI_FUNC with
extern).

The function was added in bpo-12575 by
the commit 832bfe2ebd.
2021-03-18 14:57:49 +01:00
Victor Stinner fc980e0be1
bpo-43541: Fix PyEval_EvalCodeEx() regression (GH-24918)
* Remove an assertion which required CO_NEWLOCALS and CO_OPTIMIZED
  code flags. It is ok to call this function on a code with these
  flags set.
* Fix reference counting on builtins: remove Py_DECREF().
  Fix regression introduced in the
  commit 46496f9d12.

Add also a comment to document that _PyEval_BuiltinsFromGlobals()
returns a borrowed reference.
2021-03-18 14:51:24 +01:00
Victor Stinner 6af528b4ab
bpo-43244: Fix test_peg_generators on Windows (GH-24913)
Don't redefine Py_DebugFlag, it's already defined in pydebug.h which
is included by Python.h
2021-03-18 09:54:13 +01:00
Chris Burr e0b4aa0f5c
bpo-39342: Expose X509_V_FLAG_ALLOW_PROXY_CERTS in ssl module (GH-18011)
Exposes the `X509_V_FLAG_ALLOW_PROXY_CERTS` constant as `ssl.VERIFY_ALLOW_PROXY_CERTS` to allow for proxy certificate validation as described in: https://www.openssl.org/docs/man1.1.1/man7/proxy-certificates.html
2021-03-18 01:24:01 -07:00
Victor Stinner e0bf70d08c
bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912)
test_peg_generator now defines _Py_TEST_PEGEN macro when building C
code to not call PyAST_Validate() in Parser/pegen.c. Moreover, it
defines Py_BUILD_CORE_MODULE macro to get access to the internal
C API.

Remove "global_ast_state" from Python-ast.c when it's built by
test_peg_generator: always get the AST state from the current interpreter.
2021-03-18 02:46:06 +01:00
Pablo Galindo 08fb8ac99a
bpo-42128: Add 'missing :' syntax error message to match statements (GH-24733) 2021-03-18 01:03:11 +00:00
Victor Stinner 526fdeb227
bpo-43244: Add pycore_ast.h header file (GH-24908)
Move _PyAST_GetDocString() and _PyAST_ExprAsUnicode() functions the
internal C API: from Include/ast.h to a new
Include/internal/pycore_ast.h header file. Don't export these
functions anymore: replace PyAPI_FUNC() with extern.

Remove also unused includes.
2021-03-17 23:50:50 +01:00
Victor Stinner b4536e1c6a
bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (GH-24907) 2021-03-17 23:11:03 +01:00
Elisha Hollander e272528bbd
Remove unnecessary imports in the grammar parser (GH-24904) 2021-03-17 22:07:17 +00:00
Victor Stinner 9976834f80
bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843)
Python no longer fails at startup with a fatal error if a command
line argument contains an invalid Unicode character.

The Py_DecodeLocale() function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff]
range.

Use MAX_UNICODE constant in unicodeobject.c.
2021-03-17 21:46:53 +01:00
Mark Shannon 6086ae7fd4
Add PEP 626 to what's new in 3.10. (#24892) 2021-03-16 13:43:58 +00:00
tsukasa-au a8ef4572a6
bpo-43497: Emit SyntaxWarnings for assertions with tuple constants. (GH-24867)
* bpo-43497: Emit SyntaxWarnings for assertions with tuple constants.

Add a test that shows that a tuple constant (a tuple, where all of its
members are also compile-time constants) produces a SyntaxWarning. Then
fix this failure.

* Make SyntaxWarnings also work when "optimized".

* Split tests for SyntaxWarning to SyntaxError conversion

SyntaxWarnings emitted by the compiler when configured to be errors are
actually raised as SyntaxError exceptions.

Move these tests into their own method and add a test to ensure they are
raised. Previously we only tested that they were not raised for a
"valid" assertion statement.
2021-03-16 11:14:41 +00:00
Inada Naoki 1330338583
bpo-43506: Doc: Update removal schedule for Py_UNICODE encoder APIs (GH-24885)
See PEP 624.
2021-03-16 13:24:27 +09:00