User:Davidchen55/sandbox

SeaweedFS
Original author(s)	Chris Lu
Written in	Go, Java
Operating system	Linux, Windows, MacOS, FreeBSD, NetBSD, OpenBSD
Type	Distributed object store
License	Apache License 2.0
Website	github.com/chrislusf/seaweedfs

SeaweedFS is a simple and highly scalable distributed file system, to store and serve billions of files fast! SeaweedFS implements an object storage with O(1) disk seek, transparent cloud integration, and an optional Filer with POSIX interface, supporting S3 API, Rack-Aware Erasure Coding for warm storage, FUSE mount, Hadoop compatible, WebDAV.

Object storage

SeaweedFS started as a distributed object storage to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages file volumes, and it lets these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (O(1), usually just one disk read operation). Client applications directly read or write with the object-based storage system via HTTP REST APIs.

File system

SeaweedFS' file system runs on top of the object storage system. The SeaweedFS Filer server maps the directories and file names of the file system to objects stored within SeaweedFS volume servers. Filer is a separate linearly-scalable stateless server with customizable metadata stores on an embedded LevelDB, or other external data stores, such as MySql, Postgres, Redis, Etcd, Cassandra, LevelDB, MemSql, TiDB, TiKV, CockroachDB, etc. Metadata can be exported and imported between metadata stores. This provides flexible options to balance performance, atomicity, scalability, and maintenance. Clients can access the Filer via HTTP REST APIs, or mount the POSIX-compatible file system using FUSE, or via APIs for Hadoop Compatible File System to store data on Hadoop, HBase, Presto, etc.

Transparent Cloud Integration

SeaweedFS can optionally move warm data on to cloud. Hot data can be accessed with fast local access, and warm data can be stored with the cloud storage's elastic capacity^[3]. With fixed number of servers, this transparent cloud integration literally gives SeaweedFS unlimited capacity.

History

SeaweedFS started by implementing Facebook's Haystack design paper^[4]. Also, SeaweedFS implements erasure coding with ideas from f4: Facebook’s Warm BLOB Storage System^[5].

References

^ "GitHub Repository".
^ "SeaweedFS Releases".
^ "SeaweedFS Transparent Cloud Integration".
^ Doug Beaver, Sanjeev Kumar, Harry C. Li, Jason Sobel, Peter Vajgel. "Finding a needle in Haystack: Facebook's photo storage" (PDF). {{cite journal}}: Cite journal requires |journal= (help)CS1 maint: multiple names: authors list (link)
^ Subramanian Muralidhar, Wyatt Lloyd, Sabyasachi Roy, Cory Hill, Ernest Lin, Weiwen Liu, Satadru Pan, Shiva Shankar, and Viswanath Sivakumar, Linpeng Tang, Sanjeev Kumar. "f4: Facebook's Warm BLOB Storage System" (PDF). {{cite journal}}: Cite journal requires |journal= (help)CS1 maint: multiple names: authors list (link)

External links

Official website SeaweedFS

Category:Distributed file systems supported by the Linux kernel Category:Free software Category:Network file systems Category:Userspace file systems

[1] "GitHub Repository".

[2] "SeaweedFS Releases".

[3] "SeaweedFS Transparent Cloud Integration".

[4] Doug Beaver, Sanjeev Kumar, Harry C. Li, Jason Sobel, Peter Vajgel. "Finding a needle in Haystack: Facebook's photo storage" (PDF). {{cite journal}}: Cite journal requires |journal= (help)CS1 maint: multiple names: authors list (link)

[5] Subramanian Muralidhar, Wyatt Lloyd, Sabyasachi Roy, Cory Hill, Ernest Lin, Weiwen Liu, Satadru Pan, Shiva Shankar, and Viswanath Sivakumar, Linpeng Tang, Sanjeev Kumar. "f4: Facebook's Warm BLOB Storage System" (PDF). {{cite journal}}: Cite journal requires |journal= (help)CS1 maint: multiple names: authors list (link)

[1]

[2]

[3]

[4]

[5]