![]() That is why in Example 4-2 you see b'caf\xc3\xa9': 3įor other byte values, a hexadecimal escape sequence is used (e.g., \x00 is the null byte). If both string delimiters ' and " appear in the byte sequence, the whole sequence is delimited by ', and any ' inside are escaped as \'. Therefore, four different displays are used, depending on each byte value:įor bytes with decimal codes 32 to 126âfrom space to ~ (tilde)âthe ASCII character itself is used.įor bytes corresponding to tab, newline, carriage return, and \, the escape sequences \t, \n, \r, and \\ are used. Their literal notation reflects the fact that ASCII text is often embedded in them. The fact that my_bytes retrieves an int but my_bytes returns a bytes sequence of length 1 is only surprising because we are used to Pythonâs str type, where s = s.Īlthough binary sequences are really sequences of integers, Or as the bytes \x41\x00 in UTF-16LE encoding.Īs another example, UTF-8 requires three bytesâ \xe2\x82\xacâto encode the Euro sign (U 20AC),īut in UTF-16LE the same code point is encoded as two bytes: \xac\x20.Ĭonverting from code points to bytes is encoding Ĭonverting from bytes to code points is decoding. The code point for the letter A (U 0041) is encoded as the single byte \x41 in the UTF-8 encoding, The actual bytes that represent a character depend on the encoding in use.Īn encoding is an algorithm that converts code points to byte sequences and vice versa. Shown in the Unicode standard as 4 to 6 hex digits with a âU â prefix, from U 0000 to U 10FFFF.įor example, the code point for the letter A is U 0041, the Euro sign is U 20AC,Īnd the musical symbol G clef is assigned to code point U 1D11E.Ībout 13% of the valid code points have characters assigned to them in Unicode 13.0.0, The identity of a characterâits code pointâis a number from 0 to 1,114,111 (base 10), The Unicode standard explicitly separates the identity of characters from specific byte representations: Just like the items of a unicode object in Python 2âand not the raw bytes we got from a Python 2 str. In 2021, the best definition of âcharacterâ we have is a Unicode character.Īccordingly, the items we get out of a Python 3 str are Unicode characters, The problem lies in the definition of âcharacter.â The concept of âstringâ is simple enough: a string is a sequence of characters. ![]() Proper sorting of Unicode text with locale and the pyuca libraryĬharacter metadata in the Unicode database Utility functions for normalization, case folding, and brute-force diacritic removal Safe Unicode text comparisons with normalization The default encoding trap and standard I/O issues Unique features of binary sequences: bytes, bytearray, and memoryviewĮncodings for full Unicode and legacy character setsĪvoiding and dealing with encoding errors In this chapter, we will visit the following topics:Ĭharacters, code points, and byte representations Thatâs unlikely, but anyway there is no escaping the str versus byte divide.Īs a bonus, youâll find that the specialized binary sequence types provideįeatures that the âall-purposeâ Python 2 str type did not have. You may think that understanding Unicode is not important. This chapter deals with Unicode strings, binary sequences, and the encodings used to convert between them.ĭepending on the kind of work you do with Python, Implicit conversion of byte sequences to Unicode text is a thing of the past. ![]() Python 3 introduced a sharp distinction between strings of human text and sequences of raw bytes. Esther Nam and Travis Fischer, âCharacter Encoding and Unicode in Pythonâ 1 ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |