gh-144872: fix heap buffer overflow

[AdamKorcz]

Fixes https://issues.oss-fuzz.com/issues/451112368 and #144872

cc @ammaraskar

• Issue: Heap buffer overflow in _PyTokenizer_ensure_utf8 #144872

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[ammaraskar]

Thanks for the contribution! Would you mind adding a regression test for this in https://github.com/python/cpython/blob/main/Lib/test/test_tokenize.py

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[gpshead]

I added a test.

(discussed with PSRT) We're not considering this a security issue, compile() is not safe against untrusted input (nor are the other paths that are actually executing code rather than just compiling it). It can read a couple bytes past the end of the input bytes at most. at worst a crash (ASAN in particular which I believe is how oss-fuzz found it), you'd likely get a SyntaxError as intended otherwise.

[gpshead]

We do like to have issues filed to track things - read up on https://devguide.python.org/getting-started/pull-request-lifecycle/#pullrequest
As I believe you're intending to fix up a bunch of fuzzer found items, consider making a tracking issue for similar themes of those "parser crashes" or similar for this one for example - we don't need one issue per item when they aren't actually security related.

while you're working on already public oss-fuzz findings, as they are public it is fine to go ahead and make public PRs.

But if you come across any that seem actually security relevant, please reach out via email to the to PSRT, or since you're producing fix PRs maybe just use https://github.com/python/cpython/security to drop those in GHSA's for us to look over (many will probably be things we just open directly to the public as a non-concern "more of a bug" like this one)

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[AdamKorcz]

@aisk thank you, is it me or you who needs to sign the CLA?

[AdamKorcz]

@aisk thank you, is it me or you who needs to sign the CLA?

it was me.