Geek asks and answers

Broadcast data between servers

This is a common question, but googling it gives a lot of crap.

As data volumes become higher and higher, along with processing technology and cloud capabilities, we observe a growing need for fast data transfer technologies that can unleash the full power of all available data by distributing / moving / sharing them across different servers / clients.

In our case, we record binary data in real time (50 gigs per day), and we need to upload / download it to / from subscribers every day (yes, all this is necessary locally by each subscriber server, for calculation and various data analysis data)

So, briefly, what options are available today to transfer many Gigs of Data REALLY FAST between remote Windows servers (VPS, Cloud, with a "fairly" consistent bandwidth - (optical fiber to the side))

This is an open question. Each idea is welcome regardless of protocol.

+5
source share
1 answer

The task of sending and receiving data over the network is multiple.

  • Network bandwidth is the most limiting factor , and at this level hardly anything can be done for this (with the exception of random data compression, and even in this case, the compression ratio determines the gain). Thus, a faster network is the first choice.

  • /. , http . -, , , chunking. HTTP , GB, .

  • FTP - , . FTP, GridFTP, bbftp, .

  • BitTorrents - , . Facebook BitTorrents ( ).

, :). , . , .

+3

More articles:


All Articles