Character Sets

Computers at their core, only deal with numbers, integers and floating point. So how is it that on the console we are able to type and display characters? This is actually done by sending small integer values back and forth to the console based on a well-defined mapping between the characters and these small numbers. This mapping is often referred to as the character set used by the computer. Different languages historically have required different mappings, i.e. different character sets.

ASCII

One of the most used character sets is the [[ASCII]] (American Standard Code for Information Interchange) character set. It contains 128 codes mapped to 95 characters, 32 control codes and the null character. The characters it contains are those needed to represent English text in the United States.

Control Codes in red
Printable Characters in blue

While mere humans prefer to think about numbers using base 10, i.e. decimal, it is often more convenient when dealing with computers to use base 16 or [[hexadecimal]]. In hexadecimal we use the normal digits 0-9 and then add six more digits using the letters A-F. So counting in hexadecimal we have 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, 10, 11, etc. Numbers written in hexadecimal will generally be prefixed by 0x as in 0x1A2B or with the base 16 subscript notation 1A2B₁₆

Bytes

ISO

Unicode

In an effort to unify all the various character sets in use, the [[Unicode]] standard was developed. Unicode defines codes for all the languages of the world in one character set. The first 128 codes in Unicode match those in ASCII. Currently Unicode contains over one million codes that range from 0 to 10FFFF₁₆ which can be seen on the [[Unicode Charts]]. Most Unicode codes (called codepoints) are thus too big to fit into one byte. There are different ways to encode Unicode codepoints into bytes, the most popular is [[UTF-8]].

In [[UTF-8]] the ASCII characters 0-127 are stored in one byte just like ASCII. This means that ASCII encoded text is also valid UTF-8. Higher codepoints are stored using two to six bytes. The terminal that we will use to run programs will process input and output as UTF-8. Also most files are stored as UTF-8 to keep file size down.

[[UCS-2]] uses two bytes (16 bits) for each character but can only encode the first 65,536 code points, the so-called Basic Multilingual Plane (BMP). Because of this, USC-2 is outdated, though you might still come across it.

[[UTF-16]] uses two bytes (16 bits) and extends UCS-2 using the same encoding for the Basic Multilingual Plane (BMP) and therefore USC-2 is valid UTF-16 coding. For the higher planes, UTF-16 uses a four byte encoding. UTF-16 text can thus be a combination of two and four byte codes depending on which Unicode codepoints are to be represented.

[[UTF-32]] (also referred to as UCS-4) uses four bytes for each character. Because the bytes per codepoint are fixed this simplifies encodeing but it also consumes the most space. This is why UTF-32 is not often used except for perhaps internal representation of Unicode text in memory.

Unicode text must be converted often between UTF-8 when sending to the console or to a file and the internal representations used by a program in memory. Below is a table that illustrates how different languages store Unicode text in memory.

There are varying levels of support for Unicode in different languages. Some like C++ really don't specify that you are using Unicode at all just implementation dependent character sets and Unicode could be one depending on your computer's locale setting. Other languages specifically use Unicode for internal representation of characters and strings. More recent languages have significantly deprecated the idea of characters and really only work with strings because of the complexities of Unicode.

What are these complexities? One codepoint represents one character right? Unfortunately this is not the case. One character as we think about it is a single graphical mark. In some languages these graphical marks can have modifiers above, below or around them changing the meaning of the mark. For example we can have the character (E). But if we combine that with a diacritical mark called a grave accent (`), then we get the mark (È). In Unicode there is one codepoint 0xC8 for (È), one codepoint 0x45 for (E) and one codepoint 0x60 for the grave accent (`). So in a Unicode text you can use the singular pre-composed code or you can use the two decompomposed codes. Either approach is acceptable and equivalent. European languages, Korean and Arabic are just some examples of languages that use this technique. This is why support for Unicode varies in different languages. In most languages if you ask for how many characters are in the string "È", you might get one or two as the answer depending on whether it is a pre-composed or decomponsed sequence of characters. Swift is an example of language where the answer to that question is always one because it operates not on Unicode characters but on what it called extended grapheme clusters which more closely correspond to what humans consider a character.

HelloWorlds.py

#!/usr/bin/env python3;
###############################################################################
# Simple first program with Unicode strings.
# 
# Copyright © 2020 Richard Lesh.  All rights reserved.
###############################################################################

# Begin Main
print("!مرحبا أيها العالم")
print("你好, 世界!")
print("Hello, world!")
print("Bonjour le monde!")
print("Hallo welt!")
print("Γειά σου Κόσμε!")
print("!שלום העולם")
print("नमस्ते दुनिया!")
print("こんにちは世界!")
print("안녕, 월드!")
print("Привет, мир!")
print("¡Hola mundo!")

Questions

• {{What is the ASCII decimal code for 'A'?}}
• {{What is the ASCII hexadecimal code for 'n'?}}
• {{What is the ASCII hexadecimal code for '5'?}}
• {{What character has ASCII hexadecimal code 0x2A?}}
• {{What is the Unicode decimal codepoint for 'A'?}}
• {{What is the Unicode decimal codepoint for 'ñ'?}}
• {{What is the Unicode hexadecimal code for 'Γ' (Greek capital gamma)?}}
• {{What is the Unicode hexadecimal code for '☺' (white smiling face)?}}
• {{What character has Unicode decimal codepoint 203?}}
• {{What character has Unicode hexadecimal codepoint 0x2260?}}

References

• [[Unicode Charts]]
• [[Unicode Lookup]]