Repository navigation
Conversation
The utf-8-sig codec strips the BOM before decoding, causing UnicodeDecodeError offsets and its object to refer to the BOM-stripped input rather than the original input. Adjust the exception in decode(), IncrementalDecoder._buffer_decode() and StreamReader.decode() to restore the original input and offsets. Custom error handlers still see offsets relative to the BOM-stripped input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #159071.
Problem
utf-8-sigstrips the BOM before decoding. If exception, the resultingUnicodeDecodeErrorrefers to the stripped input:startandendare both small by 3, andobjectno longer contains the BOM.For example,
b'\xef\xbb\xbf\xff'.decode('utf-8-sig')reports position 0, while the invalid byte is at position 3. Theutf-16andutf-32codecs already report offsets relative to the original input.Fix
Add
_fix_decode_error()to restore the original input and correct the offsets. It was used indecode(),IncrementalDecoder._buffer_decode()andStreamReader.decode().The exception is adjusted in place to preserve its traceback. Incremental decoding uses the buffered input as the base, so offsets remain correct even when the BOM is split across chunks. The helper also handles
memoryviewinputs, converting them tobyteswhen needed.Tests
Add tests for the codec API, incremental decoding and stream readers, including invalid bytes, truncated sequences and a BOM split across chunks. All three tests fail without the fix, and
test_codecspasses with it.Notes
Custom error handlers still see offsets relative to the BOM-stripped input, since they run before the exception reaches the caller. Fixing that would require wrapping the handler and is outside this PR's scope.
No documentation changes are needed.
AI tools were used in preparing this change.
/cc @alicederyn Thanks to Alice.