From letter to byte — from ASCII through UTF-8 to Base64

A computer knows no letters. It knows only voltage — present or absent — which we agree to read as zero and one. Everything you see on this screen had to be turned first into numbers, and the numbers into strings of bits. In that puzzle the letter "A" is the number 65. But the Polish "ł" is already two bytes, the euro sign "€" — three, and the rocket emoji "🚀" — as many as four. This is the story of how and why writing becomes bytes, and of the traps that along the way can garble text or blow a server wide open.

Why text is bytes

The smallest slice of information, the bit, holds one of two states. We group eight bits into a byte, which fits one of 2⁸ = 256 values — from 0 to 255. For a machine to store, send and display human text, someone had to fix rules mapping characters onto specific numbers. Such a set of rules is called an encoding.

It pays to separate two layers straight away, because confusing them is the source of most misunderstandings. The choice of encoding is a convention — deciding that "A" is 65 rather than 41 comes from history and compromise, not from any law of nature. But once a standard is agreed, converting a character into bits becomes strictly deterministic: bit shifts and logical operations leave no room for interpretation. The agreement is arbitrary; honouring it is mathematics.

Before the computer: Baudot's five bits

The problem is older than the transistor. Telegraphs already had to represent letters with signals back in the 19th century. Invented in the 1870s, the Baudot code (standardised as ITA2) was a five-bit scheme, which gives a mere 2⁵ = 32 combinations — too few to hold letters, digits and punctuation at once. Émile Baudot worked around the limit with shift registers: a special "letters" code and "figures" code switched the meaning of the sequences that followed, rather like the Shift key.

The solution had a flaw that will return when we reach UTF-8. If interference on the line dropped the shift character, all the rest of the message turned to gibberish, right up to the next control code. A single error cascaded across the entire remaining text.

ASCII: seven bits that were enough (only) for America

In 1963 came ASCII — a seven-bit encoding, so 2⁷ = 128 positions, laid out in readable ranges:

• 0–31 and 127 — control characters (Enter, Tab, Backspace), a legacy of driving teleprinters;
• 32 — the space;
• 48–57 — the digits 0–9;
• 65–90 — the capitals A–Z;
• 97–122 — the lowercase a–z.

That order is no accident and still makes life easier for programmers: since A is 65 and a is 97, the case difference is always the same 32. But ASCII had a built-in defect — it was American. Its 128 slots left no room for "ł", "ä" or "ñ".

As computers spread across the world, every country improvised its own fix, extending the code to a full 8 bits (256 characters) and cramming local letters into the upper half. The trouble was that they did it independently: dozens of incompatible "code pages" appeared. In Central Europe ISO-8859-2 competed with Microsoft's Windows-1250, and in Polish DOS there also circulated the homegrown Mazovia — from the 1980s, built for a domestic computer of the same name. It even carried a "zł" ligature that international Unicode never adopted, which is why converting archival documents can still call for a manual fix.

When a file saved in one code page was opened by a program assuming another, the letters turned into random symbols. This phenomenon has a technical name — mojibake — and, in Poland, a folksy one: "krzaczki" ("little weeds"). Anyone who in the 1990s received an email full of ¹æê³ knows exactly what it means.

Unicode: one catalogue for every script

The way out of the chaos was Unicode — a project that set out to give every character of every script in the world one unique number. That number is the code point, written in hexadecimal with a U+ prefix — for instance U+0142 is "ł", and U+20AC is "€".

A crucial subtlety: a code point is not a byte. It is an abstract identifier, divorced from how it is stored in memory. Unicode says "ł has number 322" but stays silent on how to write 322 into a file. That job falls to separate formats — UTF-8, UTF-16 and UTF-32 — which only then translate code points into actual bytes. You might say Unicode keeps the catalogue, and the UTF formats decide on the packaging.

UTF-8: variable length and self-synchronisation

Among them the winner was UTF-8, designed in September 1992 by Ken Thompson and Rob Pike at Bell Labs. Its genius is that it is a variable-length encoding: a character takes from 1 to 4 bytes, depending on how high its code point is. The bytes are laid out according to fixed templates:

Two wonderful consequences follow from this table. First, full compatibility with ASCII: every character in the U+0000–U+007F range is written as a single byte starting with zero, exactly as in ASCII. An old ASCII file is byte-for-byte identical to a UTF-8 file — nothing needs converting.

Second — and here UTF-8 solves Baudot's ancient problem — the encoding can synchronise itself. Every "continuation" byte starts with the bits 10, and no leading byte looks like that (those start with 0, 110, 1110 or 11110). So a decoder that jumps into the middle of a stream, or loses a chunk of data, instantly knows where one character ends and the next begins. Losing one byte corrupts one character — not, as in Baudot or UTF-16, all the rest of the text.

Three letters, three lengths

Let us watch the mechanism live. You can check all three examples in the text-encoding converter, which shows the bytes in hex, decimal and binary at once.

The Polish "ł" (U+0142, 322 in decimal). The value falls in the two-byte range. We write 322 on 11 bits: 00101 000010. The five higher bits drop into the first byte's template, the six lower ones into the second:

110+00101 → 11000101 = 0xC5
10+000010 → 10000010 = 0x82

So "ł" is the bytes C5 82.

The euro "€" (U+20AC, 8364 in decimal) falls in the three-byte range: 0010 000010 101100 distributes into E2 82 AC.

The rocket "🚀" (U+1F680, 128,640 in decimal) lies beyond Unicode's basic plane and needs four bytes: F0 9F 9A 80.

This arithmetic is absolute — there is no rounding here and no room for "roughly". The emoji takes four bytes not on a whim, but because its number simply will not fit a shorter template.

Base64: bytes that survive a text channel

Now that we can turn text into bytes, the reverse problem appears: how do you send raw bytes down a channel that accepts only plain text? That is how email works (the MIME standard), and part of HTTP headers too — a byte with a control value might be mistaken for the end of the message. The answer is Base64.

The idea rests on powers of two. Since 2⁶ = 64, one Base64 character carries exactly 6 bits. The algorithm takes bytes three at a time (3 × 8 = 24 bits) and slices them into four groups of 6 bits, turning each group into one of 64 safe characters: A–Z, a–z, 0–9, plus + and /. Because 3 bytes always become 4 characters, Base64 enlarges the data by a fixed ⅓ (≈ 33%). When the final group is short of three bytes, the empty slots are filled by the padding character =: the letter "a" encodes as YQ==, and the two-byte "ł" as xYI=.

Here comes the single most important sentence of this piece: Base64 is not encryption. The algorithm is public and fully reversible without any key — any intermediary reads the contents in a fraction of a second. Base64 changes only the representation of the data, not its confidentiality. A password "secured" with Base64 in an HTTP header is as exposed as one written out in the clear.

There is also a URL-safe variant: since + and / carry their own meaning in addresses, they are replaced by - and _, and the = padding is usually dropped.

URL encoding: percents in the address

A URL, too, may consist only of a narrow set of ASCII characters. Letters, digits and a handful of "safe" characters (-, ., _, ~) may be written directly; everything else must be written as a percent sequence — a % sign and the two hexadecimal digits of the byte's value (in UTF-8). That is why a space becomes %20, and the percent sign itself becomes %25. It is exactly what the converter does when you pick the URL format: it turns Hello world into Hello%20world, and Cześć into Cze%C5%9B%C4%87.

There is a classic trap here. In an address %20 means a space, but in HTML form data (application/x-www-form-urlencoded) a space is encoded as +. This small inconsistency between two standards can break digital signatures and authentication when one library uses + while another expects %20.

When encoding becomes a vulnerability

A final caution, because encoding is often confused with things it is not. It is not compression — Base64 and URL encoding always enlarge the data. It is not encryption — as we now know. And badly implemented, it can be a security hole.

The classic example is overlong UTF-8 encoding. The slash / has the number U+002F and, by specification, must be written as a single byte 0x2F. But if an attacker maliciously forced it into the two-byte template, they would get C0 AF — and a naive, non-conformant decoder would still read a slash out of it. Security filters checking that a request contained no ../ sequence would wave through the encoded form ..%C0%AF, and the server would expand it back to ../ — letting it read files outside the site's directory. This vulnerability (exploited on a mass scale on IIS servers, among others) is why today's specifications require rejecting overlong sequences as an error, rather than politely "fixing" them.

Representation table

The same four characters, and each column is a different agreement about how to write down that very same content.

One content, many outfits

The history of text encoding is a story about separating meaning from notation. The letter "ł" is one thing, but its number (U+0142), its bytes (C5 82), its Base64 (xYI=) and its form in an address (%C5%82) are four different outfits worn by the same idea. Choosing the outfit is a convention; the conversion itself is mathematical and lossless — as long as sender and receiver agree which standard they are speaking in. When they fail to agree, we get mojibake. When the decoder is too forgiving, we get a hole in the server. And when everything lines up, the text simply arrives at the other end of the world exactly as it left.

Further reading

• Rob Pike, The UTF-8 history — a co-author's recollection of the format's birth at Bell Labs.
• IETF, RFC 3629 — UTF-8 — the official UTF-8 specification, including error handling.
• IETF, RFC 4648 — Base16, Base32, Base64 — the full definition of Base64 and its URL-safe variant.
• IETF, RFC 3986 — Uniform Resource Identifier — address syntax and the rules of percent-encoding.
• Unicode Consortium, The Unicode Standard — the source catalogue of code points for every script.