Designing a File Storage Service
A file storage service allows users to upload, download, organize, and synchronize files across devices. At small scale, files can be written directly to a server's local disk. At large scale, that approach breaks down because application servers are ephemeral, files can be gigabytes in size, storage grows into petabytes, and downloads can consume enormous amounts of network bandwidth.
The central design principle is to separate the control plane from the data plane. Application services manage authentication, permissions, metadata, and upload coordination, while clients transfer large file contents directly to distributed object storage. This keeps application servers away from the expensive byte-transfer path and allows metadata and binary storage to scale independently.
Table of Contents
- Requirements and Scale Estimation
- Separating Metadata from File Content
- Designing the Upload Path
- Designing the Download Path
- Large Files and Resumable Uploads
- Consistency, Versioning, and Deduplication
- Production Design Example
- Common Mistakes
- Interview Checklist
- Conclusion
- Key Takeaway
- More Articles to Read
Requirements and Scale Estimation
A file storage system can include collaboration, editing, search, synchronization, sharing, previews, malware scanning, and many other features. A system design interview should first establish a manageable scope.
Core functional requirements might include:
- upload files;
- download files;
- create folders;
- list files and folders;
- delete files;
- support file versions;
- share files with other users;
- resume interrupted large uploads.
Important non-functional requirements include high durability, high availability, horizontal scalability, secure access control, and efficient transfer of large files.
Assume:
Registered users: 100 million
Daily active users: 20 million
Uploads/user/day: 2
Downloads/user/day: 10
Average file size: 5 MB
Peak multiplier: 4×
Daily uploads:
20M × 2
= 40 million files/day
Average upload requests:
40,000,000 / 86,400
≈ 463 uploads/second
With a four-times peak:
≈ 1,850 uploads/second
The more important number is storage growth:
40 million × 5 MB
= 200 TB/day
Annual raw growth is approximately:
200 TB × 365
≈ 73 PB/year
Versions, replication, temporary upload parts, backups, and storage overhead can increase the physical requirement further.
Download traffic is even larger:
20M users × 10 downloads
= 200 million downloads/day
At an average of 5 MB:
200M × 5 MB
≈ 1 PB/day
This immediately reveals two architectural requirements: application servers should not store the files themselves, and they should generally not proxy every upload and download.
The methodology behind these calculations is covered in Estimating Scale and Capacity Planning.
Separating Metadata from File Content
A file storage service manages two very different types of data.
Metadata consists of small structured records:
file_id
owner_id
parent_folder_id
filename
size
content_type
checksum
object_key
current_version
created_at
updated_at
File content consists of potentially enormous binary objects:
photo.jpg 4 MB
video.mp4 2 GB
archive.zip 18 GB
backup.bin 500 GB
These workloads should usually use different storage systems:
File Service
/ \
v v
Metadata Database Object Storage
small records large objects
The metadata database supports operations such as:
SELECT file_id, filename, size, updated_at
FROM files
WHERE owner_id = ?
AND parent_folder_id = ?
AND deleted_at IS NULL
ORDER BY filename
LIMIT 100;
Object storage handles the actual bytes using an opaque object key:
Object key:
objects/8f/91/8f91e740-2bd4-4c17...
The physical object key should not need to match the user-visible filename. Renaming:
report.pdf
|
v
final-report.pdf
can then be a metadata update rather than copying a multi-gigabyte object.
This separation also means the metadata database can be optimized around directories, ownership, permissions, and file versions while object storage is optimized around durability and large sequential transfers.
Designing the Upload Path
A naive upload design sends the entire file through the application service:
Client
|
| 5 GB file
v
Application Server
|
| 5 GB file
v
Object Storage
This creates unnecessary bandwidth, memory, connection, and scaling pressure on the application tier.
A better architecture uses the application service as the control plane and object storage as the data plane:
Client
|
| 1. request upload
v
File Service
|
| 2. authorize + create upload
v
Metadata DB
File Service
|
| 3. signed upload authorization
v
Client
|
| 4. file bytes
v
Object Storage
The application service handles authorization and generates a short-lived signed upload URL or equivalent storage credential.
A possible API flow is:
POST /files/uploads
{
"filename": "video.mp4",
"size": 2147483648,
"content_type": "video/mp4"
}
The service returns:
{
"upload_id": "upl_81722",
"file_id": "file_19281",
"upload_url": "...",
"expires_at": "..."
}
The client then uploads directly to object storage.
After successful transfer, the upload can be finalized:
POST /files/uploads/upl_81722/complete
The finalization step verifies that the expected object exists and transitions metadata from an uploading state to an available state.
CREATED
|
v
UPLOADING
|
v
PROCESSING
|
v
AVAILABLE
Failed or abandoned uploads can later be cleaned up asynchronously.
Designing the Download Path
The same principle applies to downloads.
A naive design proxies every file through the File Service:
Object Storage
|
| file bytes
v
File Service
|
| file bytes
v
Client
If the service delivers 1 PB per day, this turns application servers into an expensive bandwidth proxy.
Instead, the control plane validates access and returns temporary download authorization:
Client
|
| GET /files/123/download
v
File Service
|
+------> Metadata DB
|
+------> Permission Check
|
v
Signed Download URL
|
v
Client ==================> Object Storage / CDN
file bytes
The application tier handles a small metadata request while the large binary transfer bypasses it.
A simplified authorization flow might look like:
def create_download(file_id: str, user_id: str) -> str:
file = repository.get(file_id)
if file is None:
raise FileNotFoundError()
if not permissions.can_read(user_id, file):
raise PermissionError()
return object_store.create_signed_download_url(
object_key=file.object_key,
expires_in=300,
)
Signed URLs should normally have limited lifetimes because anyone possessing a valid URL may be able to use it until expiration.
Using a CDN
Frequently downloaded public or shared files can be served through a CDN:
Client
|
v
CDN
/ \
HIT MISS
| |
v v
Client Object Storage
|
v
CDN
|
v
Client
This reduces origin bandwidth and improves latency for geographically distributed users.
Private files require an authorization model compatible with CDN caching, such as signed URLs or signed cookies. Cache keys must also avoid accidentally allowing one user's authorization context to expose another user's private object.
Large Files and Resumable Uploads
Uploading a 20 GB file as one request is fragile. A connection failure near the end can require restarting the entire transfer.
Large files should instead be uploaded in parts:
20 GB File
+---------+---------+---------+---------+
| Part 1 | Part 2 | Part 3 | ... |
+---------+---------+---------+---------+
| | |
v v v
Object Storage
A multipart upload can follow this flow:
1. Create multipart upload
2. Divide file into chunks
3. Upload chunks independently
4. Retry failed chunks only
5. Verify uploaded parts
6. Complete multipart upload
For example, a 10 GB file using 16 MB chunks requires approximately:
10 GB / 16 MB
≈ 640 parts
If part 381 fails, only that part needs to be retried.
Clients can also upload several chunks concurrently:
Client
/ | \
v v v
Part 21 Part 22 Part 23
\ | /
\ | /
v v v
Object Storage
Concurrency improves throughput but should be bounded. Hundreds of parallel chunks from every client can overload networks, storage APIs, or browser connection limits.
Resuming an Interrupted Upload
The server or object store maintains which parts have completed:
Upload upl_81722
Part 1 complete
Part 2 complete
Part 3 complete
Part 4 missing
Part 5 complete
Part 6 missing
After reconnecting, the client uploads only missing parts.
This is particularly important for mobile clients where connections frequently switch between Wi-Fi and cellular networks.
Consistency, Versioning, and Deduplication
Metadata and object storage are separate systems, which creates consistency problems.
Consider:
1. Object uploaded successfully
2. Metadata update fails
The result is an orphaned object.
The reverse can also happen:
1. Metadata says AVAILABLE
2. Object upload never completed
Users can now see a file that cannot be downloaded.
A state machine makes these transitions explicit:
PENDING
|
v
UPLOADING
|
v
UPLOADED
|
v
PROCESSING
|
+------> FAILED
|
v
AVAILABLE
Metadata should transition to AVAILABLE only after the object is confirmed complete.
Background reconciliation can detect:
- orphaned objects;
- stuck uploads;
- metadata referencing missing objects;
- expired multipart uploads.
File Versioning
Overwriting objects in place complicates synchronization and recovery. An alternative is immutable file versions:
File: architecture.pdf
Version 1 --> object_A
Version 2 --> object_B
Version 3 --> object_C
current_version --> Version 3
A new upload creates a new object and version record instead of modifying the old object.
A simplified model is:
File
file_id
owner_id
filename
current_version_id
FileVersion
version_id
file_id
object_key
size
checksum
created_at
This makes rollback straightforward:
current_version_id:
v3 --> v2
The old object remains unchanged.
Content Deduplication
Large storage systems may contain many identical files. A cryptographic content hash can identify duplicate content:
File A --> SHA-256 --> abc123...
File B --> SHA-256 --> abc123...
Same content
The physical content can potentially be stored once while multiple logical files reference it:
Logical File A ----\
\
--> Blob abc123
/
Logical File B ----/
This is commonly called content-addressed storage.
However, deduplication introduces complexity around reference counting, deletion, encryption, tenant isolation, privacy, and garbage collection. It should not be added automatically unless storage savings justify the operational cost.
Production Design Example
A production-oriented file storage architecture can separate metadata operations, large object transfers, asynchronous processing, and content delivery:
Clients
|
v
+---------------+
| API Gateway |
+---------------+
|
+---------------+
| File Service |
+---------------+
/ | \
/ | \
v v v
Metadata DB Cache Permission
Service
|
v
Upload Coordinator
|
v
Signed Upload URL
|
v
Client ===============================> Object Storage
file bytes |
|
+-----------+-----------+
| |
v v
Processing Queue CDN / Download
| |
v v
Workers Clients
/ | \
v v v
Virus Preview Metadata
Scan Generation Extraction
The upload flow begins with metadata rather than file transfer.
Step 1: Create the upload.
Client
|
| filename, size, checksum
v
File Service
The File Service validates quota and permissions and creates:
file_id: file_9128
upload_id: upl_7812
status: UPLOADING
object_key: objects/91/28/...
Step 2: Return multipart upload authorization.
The client receives short-lived credentials or signed URLs for object storage.
Step 3: Transfer chunks directly.
Client
|
+--> Part 1 ----+
+--> Part 2 ----+----> Object Storage
+--> Part 3 ----+
Application servers do not carry the binary traffic.
Step 4: Finalize the upload.
The service verifies object completion, size, and optionally checksum before transitioning the file.
def complete_upload(upload_id: str) -> None:
upload = repository.get_upload(upload_id)
obj = object_store.head(upload.object_key)
if obj.size != upload.expected_size:
raise InvalidUpload("size mismatch")
repository.mark_uploaded(upload_id)
processing_queue.publish({
"file_id": upload.file_id,
"object_key": upload.object_key,
})
Step 5: Process asynchronously.
Large or expensive operations should not block upload completion:
Processing Event
/ | \
v v v
Malware Preview Metadata
Scan Generator Extractor
Depending on security requirements, the file may remain unavailable until mandatory scanning succeeds.
Step 6: Download through the data plane.
Client
|
| authorize
v
File Service
|
v
Signed URL
|
v
CDN / Object Storage
|
| file bytes
v
Client
This architecture allows control-plane traffic and data-plane traffic to scale independently.
The system should monitor both metadata operations and binary transfer behavior:
| Metric | Why It Matters |
|---|---|
| Upload initiation latency | Measures control-plane responsiveness |
| Upload completion rate | Detects abandoned or failing transfers |
| Multipart retry rate | Reveals network or storage instability |
| Bytes uploaded/downloaded | Measures actual data-plane workload |
| CDN hit ratio | Shows how much origin bandwidth is avoided |
| Processing queue age | Detects delays in scanning or preview generation |
| Metadata DB latency | Affects browsing, sharing, and authorization |
| Orphaned object count | Reveals consistency and cleanup problems |
| Storage growth rate | Supports capacity and cost forecasting |
Storage growth should be measured by more than raw logical file size. Physical bytes after replication, versions, temporary objects, and deduplication determine actual infrastructure cost.
Common Mistakes
| Mistake | Why It Causes Problems | Better Approach |
|---|---|---|
| Storing files on application servers | Instances become stateful and files disappear or become inaccessible during scaling and failures. | Store binary content in durable distributed object storage. |
| Proxying every upload through the API | Application servers consume enormous bandwidth and long-lived connections. | Authorize through the control plane and upload directly to object storage. |
| Proxying every download | The application tier becomes an unnecessary bandwidth bottleneck. | Use signed storage or CDN URLs after authorization. |
| Uploading huge files as one request | A late connection failure can require retransmitting the entire file. | Use multipart and resumable uploads. |
| Storing metadata with file bytes | Small structured queries and massive binary transfers have very different storage requirements. | Scale metadata and object content independently. |
| Marking files available before upload verification | Metadata can reference incomplete or nonexistent objects. | Use explicit upload states and finalize only after object verification. |
| Ignoring orphaned objects | Failed workflows gradually leak expensive storage. | Run reconciliation and garbage-collection processes. |
| Overwriting file objects in place | Versioning, caching, and concurrent synchronization become harder. | Prefer immutable version objects when version history is required. |
| Using permanent public object URLs | Private content can bypass application authorization. | Use short-lived signed authorization for private files. |
| Adding deduplication automatically | Reference management, deletion, encryption, and privacy become more complex. | Add content deduplication only when expected savings justify it. |
Interview Checklist
- Define the scope: clarify uploads, downloads, folders, sharing, versions, and synchronization requirements.
- Estimate bytes, not only requests: file systems are frequently limited by storage and bandwidth rather than RPS.
- Separate metadata from content: use structured storage for metadata and object storage for binary files.
- Separate control and data planes: application services authorize transfers without carrying all file bytes.
- Use direct uploads: send large files from clients to object storage using temporary authorization.
- Use direct downloads: serve authorized content from object storage or a CDN.
- Support multipart uploads: retry individual chunks rather than entire large files.
- Support resumability: track completed parts across connection failures.
- Use explicit upload states: do not expose incomplete objects as valid files.
- Verify completion: validate object size, checksum, or required metadata before publication.
- Process expensive work asynchronously: scanning, previews, and extraction should not occupy upload requests.
- Use immutable versions: simplify history, rollback, caching, and synchronization when versioning is required.
- Protect private content: use authorization and short-lived access credentials.
- Reconcile storage: detect abandoned uploads, missing objects, and orphaned content.
- Monitor physical storage growth: include versions, replicas, temporary data, and processing artifacts.
Conclusion
A scalable file storage service should avoid treating large files like ordinary API payloads. Metadata operations and binary transfers have fundamentally different scaling characteristics, and separating them allows each layer to use infrastructure designed for its workload.
The application tier becomes a control plane responsible for identity, permissions, metadata, upload coordination, and temporary access authorization. Object storage and CDNs form the data plane responsible for moving and retaining large amounts of binary content. Multipart transfers, explicit upload states, immutable versions, and background reconciliation make that architecture reliable under failures.
Key Takeaway
Keep application servers out of the large-file data path. Separate metadata from binary content, authorize transfers through the control plane, move bytes directly between clients and object storage or CDNs, use resumable multipart uploads for large files, and treat consistency between metadata and objects as an explicit distributed-systems problem.
Comments (0)