Introduction to Data Encoding, Hashing, and Cryptography

In system design, data is rarely used in its raw form. As data moves between applications, services, databases, and networks, it must be transformed, validated, and protected. Data encoding, hashing, and cryptography are three fundamental techniques that enable system to handle data efficiently, reliably, and securely.

Although these concepts are often mentioned together, they serve very different purposes. Encoding is used to convert data into a different format so it can be properly stored or transmitted. It does not provide security and is fully reversible. Hashing transforms data into a fixed-size value that cannot be reversed, making it ideal for integrity checks and secure comparisons. Cryptography, specifically encryption, is designed to protect data from unauthorized access while allowing authorized parties to recover the original data using keys.

Understanding the distinction between these techniques is critical for system designers. Misusing encoding where encryption is required, or using weak hashing mechanisms for sensitive data, can lead to serious security vulnerabilities.

From a system design perspective, these techniques appear in many places:

• APIs encoding data for network transmission
• Database hashing passwords and indexing data
• Distributed systems using hashes for sharding and load balancing
• Secure communication relying on encryption and key exchange

The Bit: The Atomic Unit of Information

What Is a Bit?

A bit is the smallest unit of information:

• 0 or 1
• Off or On
• False or True

Physically:

• Voltage high / low
• Charge present / absent
• Magnetic orientation

Why Binary?

Binary is:

• Reliable
• Noise-tolerant
• Easy to implement in hardware

All complex data – text, images, videos, cryptographic keys – reduce to bits.

Bytes, Words, and Memory

Byte

• 8 bits
• Smallest addressable unit in memory

Example:

01000001 → 65 → 'A'

Word

• CPU-dependent (32-bit, 64-bit)
• Determines how much data a CPU processes at once

Memory Is Just a Bit Array

RAM is essentially:

[ 0 ][ 1 ][ 0 ][ 1 ] ...

No “strings”, no “files”, no “numbers” – only bit patterns.

Number Systems

Computers use different bases to represent the same bits.

Binary (Base-2)

10101010

Decimal (Base-10)

170

Hexadecimal (Base-16)

AA

Same value. Different representations.

Why Hex Is Everywhere in Security

• Compact
• Human-readable
• Maps cleanly to bytes

Example:

Binary: 10101111
Hex:    AF

You will see hex in:

• Hash outputs
• Encryption keys
• Memory dumps
• Certificates

Understanding Data Encoding

Data encoding is the process of converting data from one representation to another to ensure compatibility, efficiency, or correctness during storage and transmission.

Key characteristics of encoding:

• Fully reversible
• Does not provide security
• Preserves the original meaning of the data

Common reasons for encoding:

• Transmitting binary data over text-based protocols
• Supporting multiple languages and character sets
• Ensuring data consistency across platforms

Examples:

• UTF-8 encoding for text representation
• Base64 encoding for binary data in APIs
• URL encoding for safe transmission in HTTP requests

Encoding solves communication problems, not security problems.

Understanding Hashing

Hashing is the process of transforming input data into a fixed-size string of characters, called a hash or digest.

Key characteristics of hashing:

• One-way (non-reversible)
• Deterministic (same input produces same output)
• Small changes in input produce large changes in output

Primary use cases:

• Data integrity verification
• Password storage
• Efficient data lookup and indexing

Examples:

• Hashing a file to detect corruption
• Storing hashed passwords instead of plaintext
• Using hashes to distribute data across servers

Hashing is essential for verification and efficiency, not for data recovery.

Understanding Cryptography and Encryption

Cryptography is the science of securing data and communication in the presence of adversaries. In system design, cryptography is primarily used to protect data confidentiality, integrity, and authenticity.

Encryption is the cryptographic technique that convert plaintext into ciphertext using a key.

Key characteristics of encryption:

• Reversible with the correct key
• Designed to prevent unauthorized access
• Can be symmetric or asymmetric

Common uses:

• Encrypting data stored in databases
• Securing communication over networks
• Protecting API secrets and credentials

Encryption ensures that even if data is intercepted or accessed, it remains unreadable without proper authorization.

Encoding vs Hashing vs Encryption