Encoding

Suppose you write:

HELLO

As a human, you see five characters:

H E L L O

But a computer doesn't store the shape of the letter H.

A computer's basic storage and processing system works with bits:

0 1 0 1 1 0 0 1 ...

So we have a problem:

How do we represent H, E, L, L, O using data that a computer can store and process?

That's where encoding comes in.

Encoding is the process of representing information using a particular set of rules. The meaning doesn't change only the representation does.

The core idea is:

Take information in one representation → represent the same information in another representation according to an agreed-upon encoding scheme.

What Is Encoding?

Encoding is the process of converting information from one representation to another representation according to a defined set of rules. so that it can be safely stored, transmitted, and interpreted by different systems. In system design, encoding ensures that data remains consistent and readable across platforms, programming languages, operating systems, and network protocols.

The key word is rules.

For example, suppose we make a rule:

A → 1
B → 2
C → 3
D → 4

Then:

ADD

could be encoded as:

144

We haven't changed the message.

Add
and
144

represent the same information under our agreed rules.

Encoding does not provide security. Its goal is compatibility and correctness, not confidentiality. Any encoded data can be decoded back to its original form without the need for secret keys.

Encoding exists to answer a simple but critical question:

How do we represent the same bytes in a form that survives systems, protocols, and humans?

Formal Definition

Encoding is a reversible, deterministic transformation of data into another representation without secrecy, for the purpose of storage or transmission.

• No keys
• No secrets
• No security guarantee

If you know the encoding, you can always reverse it.

Encoding is all about data representation.

Why does encoding need rules?

Imagine I send you:

65

What does it mean?

It could mean:

• the number sixty-five
• the letter A under ASCII
• something completely different under another system

The data itself doesn't tell you what it means.

You need to know the rules used to interpret it.

For example:

65
↓
ASCII rules
↓
A

Without knowing that ASCII was being used, 65 is ambigous.

Therefore:

Encoding works because the sender and receiver agree on the encoding rules.

Why Encoding Is Essential in Distributed Systems

Distributed systems involve multiple components that may:

• Use different hardware architectures
• Run different operating systems
• Be written in different programming languages
• Communicate over text-based protocol

Encoding acts as a common language that allows these components to exchange data reliably.

Common scenarios where encoding is required:

• Sending binary data over HTTP
• Storing multilingual text in databases
• Embedding data inside URLs or JSON payloads
• Serializing objects for network communication

Without proper encoding, data corruption, parsing errors, and system failures can occur.

1 Character Encoding

Character encoding defines how characters are represented as bytes.

ASCII

• Uses 7 bits to represent characters
• Supports only basic English characters
• Limited and unsuitable for global applications

Unicode

• A universal character set covering most world languages
• Assign a unique code point to each character

UTF Encodings

• UTF-8: Variable-length, backward compatible with ASCII, most widely used
• UTF-16: Uses 2 or 4 bytes per character
• UTF-32: Fixed-length, larger storage size

UTF-8 is the de facto standard in modern system design due to its efficiency and compatibility.

2 Binary-to-Text Encoding

Binary-to-text encoding allows binary data to be transmitted over channels that support only text. It is used in the case when you have the binary data, but you need to represent it using ordinary text characters.

Binary data
    ↓
Base64 encoding
    ↓
Text

What does “Base” mean?

A base describes how many symbols are used to represent numbers/data.

Base 2 – Binary

It uses:

0 1

Example:

5 = 101

Computers fundamentally work with binary.

Base 10 – Decimal

It uses:

0 1 2 3 4 5 6 7 8 9

This is the normal number system humans use.

Base 16 – Hexadecimal

It uses:

0-9
A-F

Example:

255 = FF

Base64 Encoding

It uses the following 64 characters:

A-Z (26)
a-z (26)
0-9 (10)
+   (1)
/   (1)

Total characters = 26 + 26 + 10 + 1 + 1
                 = 64

• Converts binary data into ASCII characters
• Commonly used in APIs, email, and authentication tokens
• Increases data size by approximately 33%
• Safe for:
  • HTTP
  • JSON
  • XML
  • Email

How Base64 Works (Internals)

1. Take 3 bytes (24 bits)
2. Split into 4 chunks (6 bits each)
3. Map each chunk to a character set

Input bytes: 01001000 01101001
Bits padded: 010010 000110 100101 001000
Characters:  S      G      V      I

Output:

"SGVsbG8="

Padding (=)

• Used when input length is not multiple of 3
• Indicates how many bytes were missing
• Not optional

Size Expansion

Base64 increases size by ~33%

3 bytes → 4 characters

Use cases:

• Embedding images in JSON
• Transmitting cryptographic keys
• Encoding JWT payloads

Base32

It uses following 32 symbols, commonly:

A - Z  (26)
2 - 7  (6)
       ----
        32

Base58

It is designed to be human-friendly. It removes visually confusing characters such as:

0  O  I  l

It is famously used in Bitcoin-related representations.

3 URL Encoding

Problem

URLs have reserved characters:

? & = / %

Solution

Encode unsafe characters:

space → %20
/     → %2F

URL encoding ensures that special characters are safely transmitted in URLs.

• Reserved characters (?, &, =) have special meanings
• Unsafe characters are replaced with % followed by hex values

Example:

• Space -> %20
• @ -> %40

URL encoding is critical in web-based systems to prevent request misinterpretation.

4 Audio Encoding

Audio needs to be represented as digital data.

There are different ways of encoding audio.

Examples:

• PCM
• MP3
• AAC

5 Video Encoding

Video contains huge amounts of data, so it needs efficient representation and usually compression.

Examples:

• H.264
• H.265 / HEVC
• AV1
• VP9

Common Security Mistake

“The token looks random, so it must be encrypted.”

No.

Base64 output is fully reversible.