Needle in a haystack: efficient storage of billions of photos
Ref:
- https://engineering.fb.com/2009/04/30/core-infra/needle-in-a-haystack-efficient-storage-of-billions-of-photos/
- https://massivetechinterview.blogspot.com/2015/06/finding-needle-in-haystack-facebooks.html
The Photos application is one of Facebook’s most popular features. They handle billions of image scale. For each uploaded photo, Facebook generates and stores four images of different sizes, which translates to a total of 60 billion images and 1.5PB of storage.
These numbers pose a significant challenge for the Facebook photo storage infrastructure.
NFS photo infrastructure
The old photo infrastructure consisted of several tiers:
- Upload tier receives users’ photo uploads, scales the original images and saves them on the NFS storage tier.
- Photo serving tier receives HTTP requests for photo images and serves them from the NFS storage tier.
- NFS storage tier built on top of commercial storage appliances.
Since each image is stored in its own file, there is an enormous amount of metadata generated on the storage tier due to the namespace directories and file inodes. The amount of metadata far exceeds the caching abilities of the NFS storage tier, resulting in multiple I/O operations per photo upload or read request. The whole photo serving infrastructure is bottlenecked on the high metadata overhead of the NFS storage tier, which is one of the reasons why Facebook relies heavily on CDNs to serve photos. Two additional optimizations were deployed in order to mitigate this problem to some degree:
Cachr: a caching server tier caching smaller Facebook “profile” images.- NFS file handle cache – deployed on the photo serving tier eliminates some of the NFS storage tier metadata overhead
The major lesson we learned is that CDNs by themselves do not offer a practical solution to serving photos on a social networking site.
CDNs do effectively serve the hottest photos— profile pictures and photos that have been recently uploaded—but a social networking site like Facebook also generates a large number of requests for less popular (often older) content, which we refer to as the long tail.
Issues
Atleast 3 disk IOPS and because of nested inode, even more
Requirements
Haystack
Components
Reads from Store
Upload
Index File
Layout of Haystack Index file