gh-148891: prevent crash in HAMT caused by GC during traverse (GH-148928)
`_PyHamt_Assoc` allocates HAMT nodes, which can trigger GC. A weakref callback or finaliser fired during GC could re-enter `contextvar_set`, replacing `ctx->ctx_vars` via `Py_SETREF` and freeing the old HAMT while `_PyHamt_Assoc` was still traversing it, causing a use-after-free.
Hold a strong reference to `ctx->ctx_vars` during the `_PyHamt_Assoc` call, so that a re-entrant free cannot affect the HAMT being traversed.
(cherry picked from commit 1798d4fb3c)
Co-authored-by: Thomas Kowalski <[email protected]>
gh-160962: Fix PyInitConfig_SetStr() with NULL value (GH-161258)
Previously, passing NULL to an optional string option crashed.
Document optional string options.
Add more tests on the PyInitConfig API.
(cherry picked from commit 6d69297b2b)
Co-authored-by: Victor Stinner <[email protected]>
gh-150868: Fix sum() deallocating a float subclass as an exact float (GH-151063)
* gh-151060: Fix sum() deallocating a float subclass as an exact float
The complex-total fast path in builtin_sum_impl() guarded with
PyFloat_Check() (which accepts subclasses) but freed the item with
_Py_DECREF_SPECIALIZED(item, _PyFloat_ExactDealloc), which assumes an
exact float. A float subclass instance reaching refcount 0 in that loop
aborts on an assertion in debug builds, and in release builds skips the
subclass tp_dealloc (type refcount leak, __del__ never runs, subclass
object pushed onto the float freelist).
Use PyFloat_CheckExact() instead, matching the float-total branch, so
subclass instances fall through to the generic PyNumber_Add() path.
* gh-150868: Also use exact-type checks for ints in the fast sum() loops
Per review: use PyLong_CheckExact with the specialized exact-int
deallocator in the float and complex fast-sum loops, so int subclasses
take the generic path.
---------
(cherry picked from commit 47c249adf3)
Co-authored-by: Pieter Eendebak <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Setting __name__, __qualname__, __doc__ or __module__ on a function
stopped the world in the free-threaded build since gh-154826 and
gh-154851, making functools.wraps() orders of magnitude slower with
multiple threads.
Use the per-object critical section and _PyObject_XSetRefDelayed()
instead, as gen_set_name() already does. Use acquire loads in the
readers and hold strong references across error reporting, tracing, and
eval-frame recording that may execute Python or detach the thread. The
__code__, __defaults__, __kwdefaults__ and __closure__ setters still
stop the world to clear func_version.
Document synchronization requirements for direct field access and the
lifetime of borrowed function attributes.
(cherry picked from commit 010bebbf9f)
gh-159652: Fix import lock leak when the C stack is nearly exhausted (GH-159660)
When an import failed with a stack overflow RecursionError, releasing the import locks on the error path could itself raise RecursionError, because the cleanup runs in a new C frame. This left the global import lock or a module lock held forever, and later imports in other threads blocked indefinitely.
_imp.acquire_lock() now raises RecursionError when there is not enough stack left to release the lock. importlib calls it before taking any import lock, so this covers the per-module locks too. The module lock weakref callback skips cleanup in that case instead of
reporting an unraisable exception.
(cherry picked from commit 926b0bf311)
Co-authored-by: Cary Hawkins <[email protected]>
Concurrent calls to codecs.unregister() can shrink the search path between its length check and indexed lookup in a free-threaded build. The lookup then returns NULL and an unconditional decref crashes the interpreter; hold the registry mutex across the scan and removal. Also reported in gh-157078. Backport to 3.15 and 3.14.
(cherry picked from commit 784b305df8)
CPython copies the trampoline templates into executable memory. Keep the
ELF source in .rodata so BOLT preserves its bytes and boundaries.
(cherry picked from commit 1ff83ad7ad)
Co-authored-by: Zanie Blue <[email protected]>
Adapt the reference-flag propagation from GH-156819 to fix gh-160299 without backporting the comprehensive changes to comprehension scope representation and compilation.
(cherry picked from commit cc3a0c9d3a)
Resetting a ContextVar can finalize the replaced value, whose finalizer may set the variable again. Updating the lookup cache after releasing the old context root overwrites that newer cached value, so get() can return a stale value or use a freed object. Update the cache before the old root can be released. Backport to 3.15 and 3.14.
(cherry picked from commit f4f8b933e3)
Co-authored-by: Jelle Zijlstra <[email protected]>
A context watcher callback can clear a later watcher while a context-switch notification is being dispatched. The dispatcher still uses its earlier snapshot of active watchers, so it calls a NULL callback and crashes; check that each callback is still registered before invoking it. Backport to 3.15 and 3.14.
(cherry picked from commit 0a5762621f)
Co-authored-by: Jelle Zijlstra <[email protected]>
Propagate the error from PyUnicode_FromOrdinal() before using the
result.
(cherry picked from commit 5b0ab21364)
Co-authored-by: Jelle Zijlstra <[email protected]>
Use free() instead of PyMem_RawFree() in pythread_wrapper() to avoid
a data race if another thread calls PyMem_SetAllocator() in parallel.
(cherry picked from commit d1878ccb44)
Co-authored-by: Nathan Goldbaum <[email protected]>
gh-159320: Enable stack size detection on Android (GH-159321)
Extend the use of pthread_getattr_np in stack size detection from any glibc linux to any glibc *or* android linux. Update tests for android's new recursion limit.
(cherry picked from commit 8ff41c2ae7)
Co-authored-by: T. Wouters <[email protected]>
Co-authored-by: Malcolm Smith <[email protected]>
Set elements are written in the order of their dumps. They were sorted
as (dump, element) pairs, so elements with the same dump were compared.
That failed for a set of several complex NaNs, which the compiler can
produce by constant folding, making the module impossible to marshal.
On this branch the failure was reported as MemoryError.
Sort (dump, index, element) tuples, in which the indexes are all
different. Elements with the same dump stay in the order of the set,
as float NaNs did before.
(cherry picked from commit 1967a87ad3)
Marshalling a set element can run Python code (__buffer__, an audit
hook) which changes the set. The number of elements was read before
the loop, so a set which grew was written past the end of the list and
one which shrank left NULL items in the list which was then sorted.
Raise RuntimeError("set changed size during iteration") instead.
(cherry picked from commit cd71799c47)
[3.15] gh-158893: Make os.strerror() thread-safe (#159120)
* gh-157695: Parse also _Py_HAVE_xxx variables in sysconfig (#158926)
Add parse_config_h() tests to test_sysconfig.
(cherry picked from commit 978a8befa5)
* gh-158893: Make os.strerror() thread-safe (#158927)
Make os.strerror() thread-safe: use the reentrant strerror_r()
function if available.
* The configure script now checks if strerror_r() is supported.
* Add a stress test to test_free_threading.test_os (new module).
* Add a comment on decode_current_locale() assertion which fails if
the input string is mutated.
* Add an assertion to _Py_DecodeLocale() to detect if the input
string was mutated during the function call.
* gh-158893: Add internal _Py_strerror() function (#158982)
Add a new internal _Py_strerror() function to Python/fileutils.c. It
uses strerror_r() if available, or use strerror() otherwise.
Replace all strerror(code) calls with _Py_strerror(code).
(cherry picked from commit 15dd7356f3)
(cherry picked from commit c774d1526f)
gh-152075: Avoid lock contention in _Py_Specialize_LoadGlobal under free threading (gh-153720)
Under high thread concurrency in free-threaded builds, `_Py_Specialize_LoadGlobal` suffers from lock contention when acquiring the critical section mutexes for the `globals` and `builtins` dictionaries during bytecode specialization.
This PR skips LOAD_GLOBAL bytecode specialization if acquiring the two object mutexes would block.
(cherry picked from commit ac8ba0ca5a)
Co-authored-by: Peter Hawkins <[email protected]>
* Run a job that the test suite with MSan to the CI (#158625)
* Run the test suite with MSan in CI
* Additional fixes
* Add `_Py_MSAN_UNPOISON_STRING`
* Apply Victor's suggestions
Co-authored-by: Victor Stinner <[email protected]>
* Apply Victor's suggestions
Co-authored-by: Victor Stinner <[email protected]>
---------
Co-authored-by: Victor Stinner <[email protected]>
(cherry picked from commit b93fb19a6e)
* Mark bytes returned by getrandom as initialized for MSan
* Handle invalid non-ASCII struct formats in the fuzz harness
---------
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
gh-158446: Reject float format precision near INT_MAX (GH-158474)
Formatting a float or complex with a precision within about 1000 of
INT_MAX could crash or produce incorrect output.
PyOS_double_to_string() now raises ValueError("precision too big") for
such precisions, as the format string parsers already do for
precisions above INT_MAX.
The limit applies regardless of presentation type or value, so a few
calls that previously succeeded (inf, nan, or 'g' with such a
precision) now raise as well.
(cherry picked from commit b7b4f3ecf2)
Co-authored-by: Gregory P. Smith <[email protected]>
gh-157549: Use sysconf() for the page size in remote_debug.h (GH-157800)
getpagesize() is not an ISO C or POSIX function. Darwin hides the prototype under -std=c11, so the _remote_debugging build fails with -Werror=implicit-function-declaration. That is the failure in the reported x86_64-apple-darwin → arm64-apple-darwin cross build.
Call sysconf(_SC_PAGESIZE) instead, which is already how resource.getpagesize() works. Fall back to 4096 if sysconf() fails.
--without-remote-debug is unchanged. That flag only disables the target-side attach protocol (Py_REMOTE_DEBUG); it does not exclude the _remote_debugging client module.
(cherry picked from commit 9193375382)
Co-authored-by: ノウラ | Flare <[email protected]>
gh-128509: Use bytes singletons in marshal (GH-157398)
Replace soft deprecated PyBytes_FromStringAndSize(NULL, n) with
PyBytes_FromStringAndSize(str, n) so marshal can get bytes singleton.
(cherry picked from commit 658612ae77)
Co-authored-by: Victor Stinner <[email protected]>
gh-155504: Set the interpreter before fallible thread state initialization (GH-155590)
(cherry picked from commit 94a75f74a1)
Co-authored-by: Abdul Mohammad <[email protected]>
In the legacy Windows stdio mode PyConfig.stdio_encoding is now initialized
with the console output code page instead of the ANSI code page.
Only one encoding can be specified, so sys.stdin.encoding is still wrong if
the input code page differs from the output one.
(cherry picked from commit 0bb312a4ea)
Co-authored-by: Serhiy Storchaka <[email protected]>
