Chengze Li

Code Once, Think Twice

1 minute read

Recently years cloud storage has been more and more popular, google drive is one of the most popular among them, let’s use it as example to understand some design thoughts behind cloud storage.

Step1: Scope the problem

As a storage service, its most important features are

  • upload & download files
  • file sync
  • notification

Also we should provide some features specific for cloud service

  • review file revisions
  • share files

Step2: High Level Design

  • block server: blocker storage, referred as block-level storage, is a technology to store data file on cloud-based environment, A file can be split into multiple blocks
  • cloud storage: we can leverage some cloud storage solution since this is not the important part of our system
  • metadata storage: stores metadata of users, files blocks, version etc.
  • notification service: notify the users who subscribe the file when the file has changes
    • if clients is offline, we will hold those message until the client come to online

Step3: Design Deep Dive

Block Server

The nature of file modification is, the modification will only happens on one part of file, so if we send the whole file for each modification, this is unnecessary and redundant.

We can only modify the blocks which has the change.

Metadata Storage Design

Upload Flow

  1. client 1 sends a request to add the metadata of new file
  2. store the new file metadata into DB, and mart file status as “uploading”
  3. client 1 uploads the content of file to block server
  4. block server chunk the file into blocks, upload them to cloud storage
  5. once the file is uploaded, blocker server will send request to web server
  6. file status change to “uploaded” in metadata DB
  7. notification service notify the relevant client

Download Flow

  1. once the client know a file is changed, it first request metadata via web server
  2. web server fetch metadata DB of the changes
  3. client get the metadata
  4. it sends the request to block server to download blocks
  5. block server download the blocks from cloud storage and return it to client

Step4: Wrap Up

We can discuss some trade-offs of the design, for example

users can upload file to cloud storage directly, without block server, so that the uploading experience is quicker, while in this way, editing is very expensive.

Also, we can bring presence service to notification service.

You May Also Enjoy