Skip to content
Topos Uranos
English

Text is not ASCII

Summary

A std::string stores bytes, not letters, and in today’s text a letter can take from one to four bytes. This lesson separates three things that everyday speech confuses: the abstract character, which Unicode identifies with a number called a code point; the bytes with which UTF-8 encodes it; and the shape that the screen draws. It explains how UTF-8 distributes the bits of a code point among its bytes, why the Spanish word “año” (year) takes four, and how to count the characters of a string without decoding it. It then shows which operations break text (reversing byte by byte, converting to uppercase byte by byte, cutting in the middle of a character), which work without any care (storing, copying, concatenating, searching, writing) and what the C++23 standard library does not do: uppercase outside ASCII, alphabetical order according to the language, normalization.

Learning objectives

  1. Explain how UTF-8 encodes a code point in one to four bytes, and decode by hand the bytes of the “ñ”.
  2. Predict the size in bytes of a string with non-ASCII letters, such as "año" (4) or "programación" (13).
  3. Count the characters of a UTF-8 string by counting the bytes that are not continuation bytes, with the condition (c & 0xC0) != 0x80.
  4. Identify the operations that break UTF-8 text (reversing byte by byte, std::toupper per byte, cutting in the middle of a character) and decide what to do in each case.

Characters, code points and bytes

Three levels

The lesson Characters, Booleans, conversions and constants presented char as a one-byte integer that is written as a letter according to the ASCII table, and warned that 'ñ' does not fit in it; the lesson Compiling, running and configuring showed that "año" takes four bytes and prepared the console to write it. This lesson explains why. ASCII assigns a number to 128 characters, from 0 to 127, which fit in seven bits: the letters of English, the digits, the punctuation marks and some control codes. Now, real text contains the ñ, accented letters, diaereses, Greek letters, Chinese ideographs, mathematical signs and emojis, which add up to hundreds of thousands. It follows that what is needed is, first, a much larger table, and second, a way of writing its numbers in bytes.

It is worth distinguishing three levels. The first is the abstract character: the letter “lowercase ñ”, regardless of how it is stored or drawn. Unicode is the standard that assigns each abstract character a number, its code point, which is written U+ followed by at least four hexadecimal digits: the “ñ” is U+00F1 (241), the “A” is U+0041 (65, the same number as in ASCII), the “€” is U+20AC (8364). Code points go from 0 to U+10FFFF, a little over a million. The second level is the encoding: the rule that converts each code point into a sequence of bytes. UTF-8 is the encoding used by the web, current systems and this course. The third level is the rendering (the glyph): the shape that the typeface draws on the screen, which can change from one font to another without the character changing.

If this is so, a piece of textual data is not a text until its encoding is known: the bytes c3 b1 are the “ñ” in UTF-8, but read with the table of an old console they are two symbols unrelated to it, which is the failure that lesson 1.4 taught how to diagnose. The same question, seen from the typesetting of documents, appears in the lesson “Languages and fonts” of the course LaTeX for Academic Writing.

Seeing the bytes

To see the bytes of a string it is enough to traverse it: a std::string and a std::string_view are sequences of char, and the range-based for loop delivers its bytes one by one. To write them in hexadecimal, the format specification {:02x} serves (two hexadecimal digits, with a leading zero). As with {:d} in lesson 2.4, and by the same correction to the standard adopted in 2023 and applied by GCC 15, {:02x} writes a char as an unsigned byte. The rule matters, because char is signed in GCC on x86-64: the byte c3, converted to int, is the number −61-61, which {:02x} would write as -3d.

C++ program
#include <print>#include <string_view>​// Bytes of a text.// Precondition: none.// Postcondition: writes a line with text, its size in bytes and each byte in hexadecimal.void printBytes(std::string_view text){    std::print("{:<4}{}:", text, text.size());    for (const char c : text) {        std::print(" {:02x}", c);    }    std::println("");}​int main(){    printBytes("A");    printBytes("ñ");    printBytes("€");    printBytes("😀");    printBytes("año");}
Output
A   1: 41ñ   2: c3 b1€   3: e2 82 ac😀  4: f0 9f 98 80año 4: 61 c3 b1 6f

Each line shows the text, its size and its bytes. One character of each length: the “A” takes one byte, the “ñ” two, the “€” three and the emoji four. The word “año” takes four bytes, 61, c3 b1 and 6f, and its size() is 4, not 3. Note that the “A” is the byte 41, the same as in ASCII: UTF-8 writes the 128 characters of ASCII exactly as ASCII does.

ExampleFrom the ñ to its two bytes

The figure follows the encoding of the “ñ” from top to bottom: the code point, its bits, the distribution of those bits and the two resulting bytes, with the prefix bits highlighted.

Proof

  1. The “ñ” is the code point U+00F1, that is, 15⋅16+1=24115 \cdot 16 + 1 = 241.

  2. In binary, 241=128+64+32+16+1241 = 128 + 64 + 32 + 16 + 1 is written 11110001. It is greater than 127 and less than 2048, so UTF-8 writes it in two bytes, which carry eleven bits: it is padded with zeros on the left up to eleven, 00011110001.

  3. The eleven bits are split into two groups: the first five, 00011, and the last six, 110001.

  4. The first byte carries the prefix 110, which announces a two-byte character, followed by the five bits of the first group: 11000011. The second carries the prefix 10, which marks a continuation byte, followed by the remaining six bits: 10110001.

  5. In hexadecimal, 1100 0011 is c3 and 1011 0001 is b1: the two bytes that the previous program wrote for the “ñ”. The change of base between binary and hexadecimal, four bits at a time, is the one in the lesson “Positional numeral systems” of Algebra and Geometry I.