Distributed keystore for 80 TB total data size

TL DR:

I would like to have recommendations for distributed storage of key values ​​for avg. input size up to 50 KB, which will be installed in the Linux environment (dedicated servers).
File system solution.
I found several solutions: Ceph, Cassandra, Riak and several more.

More details

I am looking for a storage solution for one of our components, it should be a repository with key values, a flat namespace.

Scenario

Read / write patterns are very simple:

After the key value is written, several messages will be read over the next hours.

After that, nothing affects the set value of the key. We would like to save data for future purposes, "Storage Mode".

  • : Linux
  • / Python
  • : 80 ( ).
  • Avg Entry Size ( k-v): 10 50 , ,
  • : , .
  • :
  • : 1 ,
  • : ( )

"" :

  • : Linux
  • / Python API RESTful HTTP
  • 80 ( ).
  • . : , 30 " " (. ).
  • ( node)

  • RESTful
  • ( ).

  • Ceph
  • HDFS
  • HBase HDFS
  • Luster
  • GlusterFS
  • Mongo GridFS - Mongo?
  • Cassandra - ,
  • Riak - , , Cassandra,
  • Swift + OpenStack ( Amazon S3)
  • , , , , , .

( 50 ) , , , .

+5
1

Ceph ( librados). GlusterFS - .

0

All Articles