YouTube System Design
YouTube is a globally distributed video-sharing and streaming platform that allows users to upload large video files, publish them, and stream them smoothly, with low latency, to millions of concurrent viewers. The system must support:
- Video upload and storage
- Video processing and transcoding
- Efficient and scalable streaming
- Search and discovery
- Personalization and recommendations
- High availability and fault tolerance
YouTube must handle massive scale โ millions of uploads per day and billions of views โ with low latency and high reliability.
Functional Requirements
A YouTube-like system must support the following core functions:
- Video Uploading โ Users can upload videos (potentially hundreds of MBs to GBs).
- Video Storage & Encoding โ Videos are stored durably and transcoded into multiple formats.
- Video Streaming โ Users can stream videos with adaptive bitrate support (e.g., HLS or MPEG-DASH).
- User Interaction โ Users can watch, like, comment, subscribe, and interact with video content.
- Search โ Users can search for videos based on metadata (title, tags, description).
- Recommendations โ Personalized video suggestions are delivered based on viewing history and behavior.
In this discussion, we cover these main topics:
- Video Uploading
- Video Storage & Encoding
- Video Streaming
Non-Functional Requirements
The system must satisfy the following:
- High availability & reliability โ No single point of failure; the system is resilient to outages.
- Low latency โ Fast response times for streaming and search (latency < 500 ms).
- Scalability โ Support millions of uploads and hundreds of millions of views daily.
- Durability โ Store petabytes of data reliably.
- Cost efficiency โ Use caching and CDNs to reduce backend load.
Scale
- 100 million daily active users
- 1 million daily uploads
- 400 million daily watches
- The maximum video size on YouTube is 256 GB
Calculation:
-
1 million daily uploads = 10^6/10^5 uploads/sec = 10 TPS
-
400 million watches = 400 * 10^6 /10^5 watches/sec = 4k QPS
-
1 million daily uploads, assuming an average video size of 500 MB:
=> 10^6 * 500 MB = 500 TB/day
=> 500 *365 TB/year = approx 500 * 400 = 200k TB = 200 PB
Entities
The major entities in the system are:
- User
- Video
- VideoMetadata
API
Upload a Video
POST /videos/upload
video, videoMetadata
Watch a Video
GET /videos/videoId/watch
High-Level Design
At a conceptual level, a video platform such as YouTube can be understood through two primary user journeys:
- Upload Flow โ Getting videos into the system
- Watch Flow โ Delivering videos efficiently to viewers
These flows are supported by gateway layers, storage systems, metadata management, and global delivery infrastructure.
Core Components
Client Applications
Users interact with the system through:
- Web browsers
- Mobile applications
- Smart TVs and other devices
Two key interfaces drive the platform:
- Upload Page โ Handles large video uploads
- Watch Page โ Streams video content to viewers
API Gateway
All client requests pass through the API Gateway, which acts as the entry point to the system.
Its responsibilities include:
- Request routing (Upload, Watch, Metadata, etc.)
- Authentication and rate limiting
- TLS termination
- Protection of internal services
This layer prevents clients from accessing backend services directly.
Upload Service
When a user uploads a video:
- The client sends a request via the API Gateway.
- The Upload Service stores the video in object storage, such as Google Cloud Storage or S3.
- The video metadata (such as the title, description, and author) is stored in a metadata database.

The video metadata table is shown in the image; it stores the id, title, creator_id, description, channel, tags, video_url, etc.
A Content Delivery Network (CDN) is essential for YouTube to deliver billions of videos globally without lag, buffering, or server crashes. By caching content on edge servers close to users, it minimizes latency, reduces the load on central servers, and enables high-quality, seamless streaming across different devices and internet speeds. So we also store the videos in a CDN or edge caches for faster access.

Deep Dive: Uploading Large Videos
-
Video files are large (100MB to several GB).
-
Uploading through your app server:
-
Increases memory and CPU usage
-
Consumes bandwidth
-
-
If 100 users upload at once, your server must handle concurrent large streams.
-
It adds unnecessary network hops and delay, because files must travel:
-
From the client to the app server first,
-
Then from the app server to blob storage (like S3).
-
-
The API gateway has a hard payload size limit of 10 MB.
So we cannot upload large files through the Upload Server.
Pre-Signed URL
A better approach is for the client to upload directly to blob storage. However, the client then needs proper permission to upload directly to blob storage. This is handled with a pre-signed URL.
-
The client sends a request to the upload API.
-
The upload service gets a pre-signed URL from the object storage and passes it to the client. The client can then upload the video directly to object storage (the blob store), avoiding heavy application server load.

Multi-part upload
Object storage also has upload size limits. For example, S3 allows up to 5 GB in a single upload and recommends multipart upload for objects larger than 100 MB.
=> We should therefore break the video into chunks (parts) and upload it part by part, which is called Multi-Part Upload.
Multipart upload splits a large file into smaller parts for efficient and reliable transfer.
Multipart Upload โ Overview
Multipart upload is a strategy for handling large file transfers by breaking a file into smaller, independent chunks. Instead of sending a single massive request, the client uploads multiple parts, making the process faster, more reliable, and easier to recover from failures.
High-Level Flow
-
Upload Initiation
The client asks the backend to start an upload. The backend coordinates with object storage and receives anupload_id, which uniquely tracks the upload session. -
Signed URL Generation
The backend decides the number of parts needed (from the file size metadata) and generates pre-signed URLs for each file part. These URLs allow the client to upload directly to object storage without routing large payloads through the application servers. -
Parallel Part Uploads
The client splits the file into chunks and uploads them independently. Parts can be uploaded in parallel and retried individually if a network failure occurs. -
Upload Completion
After all parts are successfully uploaded, the client signals completion. Object storage validates the parts and assembles them into the final file.
Why Multipart Upload Is Important
- Failure Resilience โ Only failed parts are retried, not the entire file
- Performance โ Parallel uploads significantly reduce total transfer time
- Scalability โ Backend services avoid becoming bandwidth bottlenecks
- Efficiency โ Ideal for large media, video, and data-heavy systems


Deep Dive: Streaming Videos
Why Are Direct Video Downloads Not Ideal?

Serving large videos as direct downloads creates serious performance, reliability, and user-experience problems:
- Slow Start Time โ Users must wait for a large portion of the file to download before playback begins, which is unacceptable for multi-GB videos.
- Storage & Memory Constraints โ The client effectively needs sufficient buffer/storage capacity, making playback fragile on constrained devices.
- Wasted Bandwidth โ If the user stops watching midway, significant data may already have been transferred unnecessarily.
- No Adaptive Bitrate (ABR) โ Direct downloads deliver a single resolution/bitrate. Users on slower networks suffer buffering, while users on faster networks cannot benefit from higher quality.
- Poor Playback Resilience โ Network fluctuations directly interrupt viewing instead of gracefully adjusting quality.
- Weak Content Protection โ Entire files are easier to copy, redistribute, or pirate.
- Limited Observability โ Engagement metrics, QoE signals, and ad analytics become harder to capture accurately.
A better approach is to stream videos by downloading them in small sequential chunks rather than fetching the entire file at once. Videos are therefore segmented into small chunks (typically 2โ10 seconds) for streaming.
Media Processing / Transcoding
Modern video platforms do not store or deliver media in a single format. Instead, uploaded video and audio streams go through a media processing (transcoding) pipeline to ensure compatibility, performance, and efficient delivery across devices and network conditions.
Once a video is uploaded, the system converts the raw media into multiple variants:
- Formats โ Different container and codec combinations (e.g., MP4, WebM)
- Resolutions โ 240p, 480p, 720p, 1080p, etc.
- Bitrates โ Multiple quality levels for each resolution
- Audio Streams โ Encoded separately at different bitrates
This step is necessary because:
- Devices support different codecs and formats.
- Network conditions vary widely.
- Efficient streaming requires multiple quality options.

The output is a set of media segments and a manifest file describing the available streams.
After a video is uploaded to object storage (e.g., S3), the system starts an asynchronous media processing pipeline to transcode the chunks.
Transcoding & Chunking Flow
1. Raw Video Storage
- The client uploads the original video using multipart upload.
- The full-resolution raw file is stored in object storage (S3).
- Upload metadata (title, creator, status, etc.) is written to the Metadata DB.
- A processing event is published to a message broker (a Kafka topic).
This decouples user uploads from heavy media processing workloads.
2. Chunker Stage
A Chunker Service consumes events from the queue and performs:
- Retrieval of the raw video from storage
- Splitting of the video into small time-based segments (typically 2โ10 seconds)
- Extraction of audio/video streams if required
Why chunking is necessary:
- It enables adaptive bitrate streaming.
- It prevents large file transfers.
- It supports parallel processing.
- It improves CDN cache efficiency.
The generated chunks are written back to object storage.

3. Transcoding Stage
A Transcoder Service processes the chunker's output:
- Converts segments into multiple resolutions (240p โ 1080p+)
- Encodes using streaming-friendly codecs
- Produces multiple bitrate variants
- Generates manifest/playlist files (HLS / MPEG-DASH)
Key goals:
- Device compatibility
- Bandwidth adaptation
- Efficient streaming
Each resolution/bitrate combination becomes an independent stream variant.
4. Storage & Distribution
The processed assets are stored back in object storage:
- Video segments per bitrate/resolution
- Audio variants
- Manifest files
These assets are then served via a CDN for low-latency global delivery.

Why Does This Pipeline Work Well?
- Asynchronous โ Upload latency is unaffected
- Horizontally scalable โ Chunker/transcoder workers scale independently
- Fault tolerant โ Failed jobs can be retried
- Streaming optimized โ Supports ABR protocols
- Storage efficient โ Raw and processed assets are managed separately
This design is foundational for large-scale video platforms.
Adaptive Bitrate (ABR) Streaming
Adaptive bitrate streaming allows the client to dynamically adjust video quality during playback based on:
- Available network bandwidth
- Device performance and screen resolution
- Real-time playback conditions
Instead of downloading an entire video at one quality level, the player requests small segments at the most suitable bitrate.
Key Characteristics
- Quality can increase or decrease seamlessly.
- Playback continues without restarting.
- Buffering is minimized on fluctuating networks.
Common ABR Protocols
Two dominant protocols power adaptive streaming:
- HLS (HTTP Live Streaming)
- MPEG-DASH (Dynamic Adaptive Streaming over HTTP)
Both work by:
- Splitting media into small time-based chunks
- Providing multiple bitrate/resolution variants
- Allowing the client to switch streams dynamically
Why ABR Is Critical
- It ensures smooth playback across network conditions.
- It optimizes bandwidth consumption.
- It improves startup latency and user experience.
- It supports a wide range of devices and capabilities.
ABR streaming is a foundational requirement for any large-scale video delivery system.
Upload Chunks vs. Download Chunks
While both uploads and downloads may split data into smaller pieces, they serve very different purposes and follow different design constraints.
Upload Chunks (Multipart Upload)
Goal: Ensure reliable and efficient transfer of large files from client โ storage.
- The client splits the file into chunks.
- Chunks are uploaded independently.
- Failed chunks can be retried individually.
- Uploads may run in parallel.
- Storage assembles the parts after completion.
Typical Chunk Size
- Usually larger chunks (e.g., 5โ100 MB)
- Chosen to balance:
- Network overhead (too small โ inefficient)
- Retry cost (too large โ expensive failures)
Optimizes for
- Failure recovery
- Transfer efficiency
- Backend load reduction
Common use cases โ Video uploads, large file ingestion.
Download Chunks (Adaptive Streaming)
Goal: Deliver media efficiently for smooth playback.
- Videos are pre-segmented by the system.
- Chunks represent short time-based segments.
- The client requests chunks on demand.
- It supports adaptive bitrate switching (ABR).
- Playback continues without restarting.
Typical Chunk Size
- Usually small segments (2โ10 seconds of video)
- Size varies based on bitrate:
- Lower bitrate โ smaller files
- Higher bitrate โ larger files
Why small segments?
- Faster startup
- Quick quality adaptation
- Reduced buffering impact
Optimizes for
- Low-latency playback
- Bandwidth efficiency
- Network variability
Common use cases โ HLS / MPEG-DASH streaming.
Key Difference
- Upload chunking โ Reliability & transfer optimization
- Download chunking โ Playback & user experience optimization
Even though both use "chunks," their objectives, behavior, and sizing strategies are fundamentally different.
Manifest File
The manifest file (also called a playlist or index file) is simply a small static file containing streaming metadata. It is the control document that tells the video player how to stream a video.
Instead of downloading a single large file, the player first retrieves this lightweight metadata file, which describes:
- Available resolutions
- Bitrates
- Formats / codecs
- An ordered list of chunk URLs
Example structure:
{
"video_id": "yt12345",
"title": "System Design Basics",
"streams": [
{
"resolution": "1080p",
"bitrate": "3000kbps",
"format": "mp4",
"codec": "h264",
"chunks": [
"https://cdn.youtube.com/yt12345/1080p/1.mp4",
"https://cdn.youtube.com/yt12345/1080p/2.mp4"
]
},
{
"resolution": "720p",
"bitrate": "2000kbps",
"format": "mp4",
"codec": "h264",
"chunks": [
"https://cdn.youtube.com/yt12345/720p/1.mp4",
"https://cdn.youtube.com/yt12345/720p/2.mp4"
]
},
{
"resolution": "480p",
"bitrate": "1000kbps",
"format": "webm",
"codec": "vp9",
"chunks": [
"https://cdn.youtube.com/yt12345/480p/1.webm",
"https://cdn.youtube.com/yt12345/480p/2.webm"
]
}
]
}
Why the Manifest Exists
The manifest is the key enabler of adaptive bitrate (ABR) streaming:
- The player dynamically selects video quality based on available bandwidth.
- Quality can switch without restarting playback.
- Bandwidth usage becomes efficient and network-aware.
- Playback startup latency is significantly reduced.
Without a manifest, adaptive streaming is not possible.
Where Is the Manifest Stored?
The manifest is generated by the transcoding and packaging pipeline and stored alongside the video chunks in Object Storage (Origin Layer). It can also be cached in a CDN or on edge servers. The manifest file URL can be stored in our metadata database.
Object storage is well suited for this because it offers:
- High durability
- Low cost for static assets
- Simple versioning and regeneration
- Excellent compatibility with CDN caching
Manifests rarely change once created.
How the Client Uses the Manifest File
A typical streaming sequence:
- The player (client) requests the manifest file from the CDN; if it is not found there, it is fetched from object storage.
- The manifest describes the available stream variants.
- The player selects the optimal rendition.
- The player downloads chunks progressively.
