Store Media Files On Google Cloud Storage

1. Difference between Object Storage, Database and File System

• Object Storage: Each piece of data is stored as an object with no tree-shape index. Datas are searched by using unique address/ path, which ensures a higher efficiency.

• File System: Data are stored as a piece of information in folder, to help organize it among all other data.

2. Advantages of using Object Storage Service

• a. Highly Durable: Most of the Object Storage Service (e.g. GCS, Amazon S3 + CloudFront) are designed for a very high (~99.99999999%) durability. Datas are redundantly backed up, which ensures a greate data integrity.
• b. Highly Avaliable: All storage class offer high avaliablity around the world.
• c. Highly Scalable: Distributd storage.
• d. Inexpensive: Nearly free. Good choise to store unstructured media files like images, videos ets.

3. General idea of how to send images to GCS.

• From the backend side, after we got the request from corresponding URL, we need to add a middleware multer to store the object first and then post to GCS. Therefore, the router handler should be somthing like:

  router.post('<YOUR_TARGET_URL>', multer.single('<INPUT_FIELD_NAME>'), gcsMiddlewares.sendUploadToGCS, image_controller.addImageToGCS)

• First part is the URL where frontend sends requests.
• Second part is the multer middleware. According to our frontend design, we only support uploading one image at a time, so we use multer.single here.
• Third part is another middleware gcsMiddlewares. How to define this middleware function will be discussed later on.
• Finally, the fourth part is the function describes how we handle the success/ error cases. If the image is sent successfully, we will send back a response with {"id": 101}, otherwide, we will sendback a status with code 500 together with a message Unable to upload to GCS.

4. Setup GCS API.

• If you’d like to operate though command line, please first instal gsutil. Gsuril enable you to access Cloud Storage from command line. Installing gsutil as part of Google SDK curl https://sdk.cloud.google.com | bash. Restart shell exec -l $SHELL and initiate gcloud environment gcloud init.
• a. Create a new Bucket. You could ether create a new bucket through command line gsutil mb gs://[YOUR-BUCKET-NAME] or directly create though GCS webpage.
• b. Setup GCS client library follow the instructions.
• c. Create your new service account and download a Json/P12 file to local.
• d. Set google app credentials locally: export GOOGLE_APPLICATION_CREDENTIALS="YOUR_JSON_FILE_PATH. Double check your setting with $ echo $GOOGLE_APPLICATION_CREDENTIALS to enable posting.

5. Connect Frontend, Server and Cloud

Issues occured in process

1. Uploading successfully to GCS, but get fileSize = 0 on cloud.

• Reason: when uploading media files to cloud, the process is: frontend send request -> backend get request and check the corresponding input field -> get the media file uploaded in that input field -> use multer to temporarily stored on memory -> then the file will be sent to corresponding bucket we created befor.
• When setting multer, we must set the storage as storage: Multer.MemoryStorage. If we store the file to other destination by setting it manually: dest: 'tmp/uploads' it will result in a size: 0 on the cloud-side.

2. How to do name-based filtering and download multiple images at the same time from GCS.

• System Design - How to Design a YouTube/ Netfix System ???

  …