gh-148891: prevent crash in HAMT caused by GC during traverse (GH-148928)
`_PyHamt_Assoc` allocates HAMT nodes, which can trigger GC. A weakref callback or finaliser fired during GC could re-enter `contextvar_set`, replacing `ctx->ctx_vars` via `Py_SETREF` and freeing the old HAMT while `_PyHamt_Assoc` was still traversing it, causing a use-after-free.
Hold a strong reference to `ctx->ctx_vars` during the `_PyHamt_Assoc` call, so that a re-entrant free cannot affect the HAMT being traversed.
(cherry picked from commit 1798d4fb3ca0e1cd8f9e6ab0ac99a0a9fb1f7527)
Co-authored-by: Thomas Kowalski <thom.kowa@gmail.com>
gh-161229: Accept `None` for `globals` and `locals` in `__lazy_import__()` (GH-161231)
(cherry picked from commit 1941319efb6fa4184b7fa14a8638a870530ffe20)
Co-authored-by: Lazizbek Ergashev <lazerg2@gmail.com>
gh-160962: Fix PyInitConfig_SetStr() with NULL value (GH-161258)
Previously, passing NULL to an optional string option crashed.
Document optional string options.
Add more tests on the PyInitConfig API.
(cherry picked from commit 6d69297b2b1007991640310e8699979a2ef66287)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-150868: Fix sum() deallocating a float subclass as an exact float (GH-151063)
* gh-151060: Fix sum() deallocating a float subclass as an exact float
The complex-total fast path in builtin_sum_impl() guarded with
PyFloat_Check() (which accepts subclasses) but freed the item with
_Py_DECREF_SPECIALIZED(item, _PyFloat_ExactDealloc), which assumes an
exact float. A float subclass instance reaching refcount 0 in that loop
aborts on an assertion in debug builds, and in release builds skips the
subclass tp_dealloc (type refcount leak, __del__ never runs, subclass
object pushed onto the float freelist).
Use PyFloat_CheckExact() instead, matching the float-total branch, so
subclass instances fall through to the generic PyNumber_Add() path.
* gh-150868: Also use exact-type checks for ints in the fast sum() loops
Per review: use PyLong_CheckExact with the specialized exact-int
deallocator in the float and complex fast-sum loops, so int subclasses
take the generic path.
---------
(cherry picked from commit 47c249adf3bc2ae3264a7373f7f1f5e7c4d84955)
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Setting __name__, __qualname__, __doc__ or __module__ on a function
stopped the world in the free-threaded build since gh-154826 and
gh-154851, making functools.wraps() orders of magnitude slower with
multiple threads.
Use the per-object critical section and _PyObject_XSetRefDelayed()
instead, as gen_set_name() already does. Use acquire loads in the
readers and hold strong references across error reporting, tracing, and
eval-frame recording that may execute Python or detach the thread. The
__code__, __defaults__, __kwdefaults__ and __closure__ setters still
stop the world to clear func_version.
Document synchronization requirements for direct field access and the
lifetime of borrowed function attributes.
(cherry picked from commit 010bebbf9fd12bba1e3f40bdbddacd51a9c2bf20)
gh-159652: Fix import lock leak when the C stack is nearly exhausted (GH-159660)
When an import failed with a stack overflow RecursionError, releasing the import locks on the error path could itself raise RecursionError, because the cleanup runs in a new C frame. This left the global import lock or a module lock held forever, and later imports in other threads blocked indefinitely.
_imp.acquire_lock() now raises RecursionError when there is not enough stack left to release the lock. importlib calls it before taking any import lock, so this covers the per-module locks too. The module lock weakref callback skips cleanup in that case instead of
reporting an unraisable exception.
(cherry picked from commit 926b0bf3110bf86550ef248cf911e24fc123ec64)
Co-authored-by: Cary Hawkins <hawkinscary23@gmail.com>
CPython copies the trampoline templates into executable memory. Keep the
ELF source in .rodata so BOLT preserves its bytes and boundaries.
(cherry picked from commit 1ff83ad7addf96fffdca863ac0f02f86538bb757)
Co-authored-by: Zanie Blue <contact@zanie.dev>
Adapt the reference-flag propagation from GH-156819 to fix gh-160299 without backporting the comprehensive changes to comprehension scope representation and compilation.
Concurrent calls to codecs.unregister() can shrink the search path between its length check and indexed lookup in a free-threaded build. The lookup then returns NULL and an unconditional decref crashes the interpreter; hold the registry mutex across the scan and removal. Also reported in gh-157078. Backport to 3.15 and 3.14.
(cherry picked from commit 784b305df8821d57fb6fb6c52794ef67bda0a75d)
Co-authored-by: Jelle Zijlstra <jelle@openai.com>
Resetting a ContextVar can finalize the replaced value, whose finalizer may set the variable again. Updating the lookup cache after releasing the old context root overwrites that newer cached value, so get() can return a stale value or use a freed object. Update the cache before the old root can be released. Backport to 3.15 and 3.14.
(cherry picked from commit f4f8b933e3cf73c1ec8ca1d33ff62faee9f0ffd8)
Co-authored-by: Jelle Zijlstra <jelle@openai.com>
A context watcher callback can clear a later watcher while a context-switch notification is being dispatched. The dispatcher still uses its earlier snapshot of active watchers, so it calls a NULL callback and crashes; check that each callback is still registered before invoking it. Backport to 3.15 and 3.14.
(cherry picked from commit 0a5762621ff6b4110fae01caf2f85ea93c54a238)
Co-authored-by: Jelle Zijlstra <jelle@openai.com>
On allocation failure, close the remaining live inputs and pop all three
inputs before unwinding, so the consumed argument is not closed twice.
Backport the regression test and regenerate the interpreter files using
the 3.15 cases generator.
(cherry picked from commit 4ecf6ca1b3d1eb8bb9afd0c0b9dfdbebdc9e05fc)
Co-authored-by: Timur Mamedov <timurmamedov742@gmail.com>
Do not release the borrowed skip_file_prefixes argument, and stop if
allocation of the fallback filename fails.
(cherry picked from commit fab1405da8c93429ab1a1d3d3e1779d15b7223a2)
Co-authored-by: Jelle Zijlstra <jelle@openai.com>
gh-159603: Use pthread_attr_get_np() on FreeBSD (#159604)
Use pthread_attr_get_np() on FreeBSD to get the stack size (and guard
size) of the current thread.
Check if pthread_attr_get_np() function and <pthread_np.h> header are
available in configure.
Update test_json.test_recursion. On FreeBSD 16 with a default stack size of
512 MB, the json C implementation no longer raises RecursionError in
test_highly_nested_objects_encoding(). Reduce also max_depth from 5,000 to
2,000 so test_highly_nested_objects_encoding() of the Python implementation
only takes 1.6 seconds, instead of 12 seconds.
(cherry picked from commit 71ba39dc057941fb8ea032b813862a215bb3d5aa)
gh-158121: Preserve instruction events when line monitoring is disabled (GH-158456)
(cherry picked from commit f0f76de0c8521e51c2e5c1da6de23373ec39231c)
Co-authored-by: limadog9 <jakewallin42@gmail.com>
gh-130279: Fix missing exception handlers for jumps in loops (GH-159460)
(cherry picked from commit 036d309cf542986a39eeba5ca805b2ec7619731d)
Co-authored-by: Han <180024497+LH-and-FPGA@users.noreply.github.com>
Fix typo in free-threaded GC prefetch check for 32-bit x86 MSVC.
(cherry picked from commit 1b015e66f058f29de4d97031b465587ec15cace9)
Co-authored-by: Pengyu Lee <lipengyu@kylinos.cn>
gh-159320: Enable stack size detection on Android (GH-159321)
Extend the use of pthread_getattr_np in stack size detection from any glibc linux to any glibc *or* android linux. Update tests for android's new recursion limit.
(cherry picked from commit 8ff41c2ae723a62abace675b058dd4da8f90a397)
Co-authored-by: T. Wouters <thomas@python.org>
Co-authored-by: Malcolm Smith <smith@chaquo.com>
A single-phase init function that creates its module using only the
last component of a dotted name (as pybind11 does for submodules) got
a module whose __name__ was the short name. Set the package context
while calling the init function, as the dynamic loader does, so the
module gets the full name from the spec.
(cherry picked from commit 04862280a2ccbf9be9883eb4bce886f6b1bc4ca0)
Co-authored-by: Itamar Oren <itamarost@gmail.com>
Preserve active attach waiters across successive stop-the-world pauses
so a tight collection loop cannot keep them from making progress.
Keep the waiter-aware resume operation local to pystate.c because this
branch does not have the newer BRC suspend and resume helpers.
(cherry picked from commit f1565109603123c39bd2a0c6002f5ea15a62b6e7)
gh-157415: Fix data race in pthread wrapper (#157430)
Use free() instead of PyMem_RawFree() in pythread_wrapper() to avoid
a data race if another thread calls PyMem_SetAllocator() in parallel.
Reenable @nomemtest tests on TSAN.
(cherry picked from commit d1878ccb44999bfa30f5a0e84b4383e6068e5b72)
Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>
* Expose two possible builtin JIT gaps
* Add trace and runtime guard for builtin dict
* Check _GUARD_BUILTINS_IS_CANONICAL once per frame
* Add check and fix for globals being folded
(cherry picked from commit 0906d2ab93fc9f4c7e0db031c965e7d8a0db7d37)
Set elements are written in the order of their dumps. They were sorted
as (dump, element) pairs, so elements with the same dump were compared.
That failed for a set of several complex NaNs, which the compiler can
produce by constant folding, making the module impossible to marshal.
On this branch the failure was reported as MemoryError.
Sort (dump, index, element) tuples, in which the indexes are all
different. Elements with the same dump stay in the order of the set,
as float NaNs did before.
(cherry picked from commit 1967a87ad395552dbd1e16fd3f13d3b4b2f694de)
Marshalling a set element can run Python code (__buffer__, an audit
hook) which changes the set. The number of elements was read before
the loop, so a set which grew was written past the end of the list and
one which shrank left NULL items in the list which was then sorted.
Raise RuntimeError("set changed size during iteration") instead.
(cherry picked from commit cd71799c47802f87e1469cc97cc99f5ce6cea27c)
* [3.15] gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) (#158813)
gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365)
* let me declare single limit
* use our new limit in process_frame_chain()
* add it in parse_async_frame_chain()
* parse_coro_chain()
* NEWS
* async in the message?
* test
* no race
* process_task_awaited_by
* process_task_awaited_by limit test
* NEWS
* MAX_TASK_WAITER_CHAIN_DEPTH
* TASK_WAITER_CHAIN_DEPTH in test
* TASK_WAITER_CHAIN_DEPTH 256
* prevent the drift with the comment
* better naming, better style
* MAX_TASK_WAITER_CHAIN_DEPTH comment
* task-waiter iterative bfs walk
* iterative coro-walk
* nicer news
* 1 << 14
* comment
* unused read_Py_ssize_t
* fix tombstones
* simplify
* correct msg
* better test
* news for tombstones
* left-over from when testing buggy version
* redundant new line
(cherry picked from commit e0861c6ae70c7e6f16c28399bd2b28c521e11e84)
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
(cherry picked from commit 9e401cf7f5a53d8b46702e6c4bcb8849b9bf2829)
* [3.15] gh-155811: Add a seqcount to `gc_stats` to prevent torn reads (GH-155828) (#158829)
* update_seq
* no need for XCHGL, MOVL is enough?
* gh-155811: Retry an inconsistent GC snapshot once
---------
(cherry picked from commit 5fecd448bb120378978a37dde65dfce233d88c0d)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 028124039c276f73fbc26036f1629c91be4c2891)
* [3.15] gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) (#158830)
gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892)
* the kolektor
* test
* better test
* news
* =Q, move const to the base, not self.running
* gh-151292: Track binary writer finalization state
---------
(cherry picked from commit f839c061d383bc87c3456d19bbdcbc1c30ed399b)
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
(cherry picked from commit 8a7c23f316a6bdb3cc60c6920d465d2a81d3cf35)
* [3.15] gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (#158691)
gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584)
(cherry picked from commit 9d22a5334bd5273962adceeb697a1337e9a0ca21)
Co-authored-by: Victor Stinner <vstinner@python.org>
(cherry picked from commit 4f7af46e25f09bf54b6f499c5c40b661786f0d69)
* [3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) (#158831)
* gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195)
* degrade gracefully
* news
* better NEWS wording
* do not raise on MAX_REMOTE_STR_READ
* bye MAX_REMOTE_STR_READ
* fix -m asyncio ps|pstree
* test truncation and linetable sentinel
* simpler
* simpler
* redundant now
* respect #157790 in the news
---------
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
(cherry picked from commit 7d25916b41de4fc28e04b4944864ad1094a9e8e0)
* Preserve the stable ABI when creating fallback frame names
---------
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
(cherry picked from commit c27f4940612b094fe225875a815f8ee6a4f7f9ec)
* [3.15] Add MSan to CI (GH-158625) (#158832)
Run a job that the test suite with MSan to the CI (#158625)
* Run the test suite with MSan in CI
* Additional fixes
* Add `_Py_MSAN_UNPOISON_STRING`
* Apply Victor's suggestions
* Apply Victor's suggestions
---------
(cherry picked from commit b93fb19a6e3118857d9a1dc4ffcc179b632a88c1)
Co-authored-by: Stan Ulbrych <stan@python.org>
Co-authored-by: Victor Stinner <victor.stinner@gmail.com>
(cherry picked from commit 1bc78dea18dfaff6900eeac5edb1ca651ff700d7)
* [3.15] gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (#156814)
gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811)
(cherry picked from commit c3706f4c687d76b85702c1f946abd3f3d3eafcb8)
Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com>
(cherry picked from commit fce28da9b75d429a7b65eec450df8e804fada001)
* [3.15] gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) (#158845)
gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802)
* gh-158552: Wait for Windows threads to suspend before blocking sampling
* Use a named Windows thread enumeration status constant
(cherry picked from commit 1643525f0f89438d87daf4cdbb1cbb1de9df08e8)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
(cherry picked from commit 31288da21d4536eccd8df2e8222687919164e867)
* [3.15] gh-152721: Fix quadratic RLE replay time in the profiling binary reader (GH-152722) (#158850)
Backport of GH-152722.
Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
(cherry picked from commit ebaca2ac2d3ac29cd2760b6a5e4617c3fc5f6333)
* [3.15] gh-156545: Fix flamegraph export RecursionError on deeply recursive programs (GH-156546) (#158851)
Backport of GH-156546.
Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com>
(cherry picked from commit 0ac721704092d8f96720c6fd1d68c5941a4a530a)
* [3.15] gh-158540: Add the profiled script's directory to sys.path (GH-158548) (#158844)
gh-158540: Add the profiled script's directory to sys.path (GH-158548)
* gh-158540: Add the profiled script's directory to sys.path
When a script is profiled with ``python -m profiling.sampling run
script.py`` from another directory, the script cannot import modules
placed next to it, because ``_sync_coordinator._execute_script()``
executes it with the working directory (added by
``_setup_environment()`` for the module case) as ``sys.path[0]``
instead of the script's own directory.
Make the script's directory importable in ``_execute_script()``,
matching the behavior of ``python script.py``.
Add a regression test that runs the coordinator on a script importing
a sibling module.
* Update _sync_coordinator.py Comment simplified
* gh-158540: Resolve symlinks when adding the script directory to sys.path
``python script.py`` resolves symlinks when computing ``sys.path[0]``, so
a script reached through a symlink (``link.py -> sub/where.py``) imports
modules from the real script's directory. Apply ``os.path.realpath()``
before taking the directory name, and make sure the result is placed at
the front of ``sys.path`` even if it was already listed.
Add a regression test for a symlinked script.
---------
(cherry picked from commit 3f02aab9faa338853425da23541e44e4bd24d478)
Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com>
Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com>
(cherry picked from commit f52d831397ec77cf969c71c1ef339c1e28494eca)
* [3.15] gh-153838: Skip non-regular source files in the heatmap exporter (GH-153839) (#158853)
Backport of GH-153839.
Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
(cherry picked from commit 48998df2510c0daa837f98bbbbe6cc49ef60dc43)
* [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) (#158852)
* [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581)
Backport of GH-158581.
Co-authored-by: LucasZhou <donghao.zhou@outlook.com>
* [3.15] gh-158539: Use portable static assertion messages
* [3.15] gh-158539: Keep layout assertions with debug-offset validation
* [3.15] gh-158539: Use the platform guard for in-process inspection tests
---------
Co-authored-by: LucasZhou <donghao.zhou@outlook.com>
(cherry picked from commit d625ecbb8d36b7dd26c61b13d1c97de117d36642)
* [3.15] gh-158522: Fix truncated stack for a task whose coroutine recurses (GH-158526) (#158870)
Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
(cherry picked from commit a904b397ed324e404a8abd682b056c146cb48263)
* gh-156545, gh-158539: Fix deep flamegraph export on small C stacks and macOS runtime lookup (#158874)
(cherry picked from commit 114de198c0a05d8169cdbbd8f1e0976dc1319284)
Include the C-stack test helper from main, introduced by
ce5ae29ef9b8ccb32683c922d135339311c36c9d, which the regression test
requires but 3.15 does not yet provide.
---------
Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Stan Ulbrych <stan@python.org>
Co-authored-by: Victor Stinner <victor.stinner@gmail.com>
Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com>
Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com>
Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com>
Co-authored-by: LucasZhou <donghao.zhou@outlook.com>
Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
* gh-157695: Parse also _Py_HAVE_xxx variables in sysconfig (#158926)
Add parse_config_h() tests to test_sysconfig.
(cherry picked from commit 978a8befa5ec9b28b161a56b16d326330aef4733)
* gh-158893: Make os.strerror() thread-safe (#158927)
Make os.strerror() thread-safe: use the reentrant strerror_r()
function if available.
* The configure script now checks if strerror_r() is supported.
* Add a stress test to test_free_threading.test_os (new module).
* Add a comment on decode_current_locale() assertion which fails if
the input string is mutated.
* Add an assertion to _Py_DecodeLocale() to detect if the input
string was mutated during the function call.
* gh-158893: Add internal _Py_strerror() function (#158982)
Add a new internal _Py_strerror() function to Python/fileutils.c. It
uses strerror_r() if available, or use strerror() otherwise.
Replace all strerror(code) calls with _Py_strerror(code).
(cherry picked from commit 15dd7356f300733b9c3fcc35f0d75e61a6ef3e05)
gh-152075: Avoid lock contention in _Py_Specialize_LoadGlobal under free threading (gh-153720)
Under high thread concurrency in free-threaded builds, `_Py_Specialize_LoadGlobal` suffers from lock contention when acquiring the critical section mutexes for the `globals` and `builtins` dictionaries during bytecode specialization.
This PR skips LOAD_GLOBAL bytecode specialization if acquiring the two object mutexes would block.
(cherry picked from commit ac8ba0ca5a04cd7b27b44822d31df640e817fd78)
Co-authored-by: Peter Hawkins <phawkins@google.com>
