mirror of
https://github.com/python/cpython.git
synced 2026-09-07 16:05:29 +08:00
09117bc317
Represent token boundaries as logical source offsets with explicit start and end locations, and adapt pegen and _tokenize to consume them. Use SourceText to own decoded input across all reader modes. Streaming readers discard consumed windows while preserving logical offsets and reusing the allocation. Prepared and interactive input retain their source. Force source relocation on growth in debug builds and add coverage for formatted-string growth, interactive input, storage reuse, and offset limits. Scanner positions remain pointers, rebased when storage moves.