Geek asks and answers

How to constantly connect FTP files

Ok, here is the situation ... I have an application that generates about 8 files per second. Each file is 19-24kb. This creates 10 to 11 MB per minute. This question is not related to the way ftp, because I already have this solution ... The question is how to keep up with the data stream (only 2 times the bandwidth is most often downloaded if I do not go to the client site which has a big pipe). I don’t like if ftp takes longer to transfer the stream speed, but I want to know if anyone has an idea on how to batch files move them so that when the ftp process ends, it only deletes the files that it transferred and then move on to the next batch. Here is what I thought:

A multi-threaded application, the first thread starts the application, the second thread is a timer that creates a text file every "N" minutes with all the files created for this period of time. StreamRead file and move the files in the text to another place (maybe create a temporary folder), then ftp these files, and then delete the files, folder and text file ... in the meantime, more text files and the tempo of the created folders are written . Is it possible? I will accept any suggestions that anyone can receive on the recommendation, just looking for the fastest and most reliable way.

Please do not ask to see the code, there is no reason to see it, given that we are working with hypotheses.

+3
source share
7 answers

Wihtout, , , , , , , FTP , .

. , .

One Serivce ( /, -) .

, temp, , , FTP .

, , , .

, . , , . , .

- , - , .

, , , , , , , .

, , , , , FTP- , .

, , FTP- . , , .

, fTP, .

+1

, FileSystemWatcher, System.Threading.Timer (FileSystemWatcher , , , ). , .NET 4.0. - . , , . .

http://msdn.microsoft.com/en-us/library/dd997415.aspx OnlyOnFaulted. , .

var task1 = Task.Factory.StartNew(() =>
{
    throw new MyCustomException("Task1 faulted.");
})
.ContinueWith((t) =>
    {
        Console.WriteLine("I have observed a {0}",
            t.Exception.InnerException.GetType().Name);
    },
    TaskContinuationOptions.OnlyOnFaulted);
+4

- . . , :

  • FileSystemWatcher, ,
  • , . ( ftp )
  • , ( )

:

  • ftp/
  • FileSystemWatcher . /
+1

( FTP), , . .

, , , . , .. , , FileSystemWatcher . , .

( , ), Microsoft Message Queue SQL Server Broker. .

FTP , , ( MSMQ, SQL Server Broker ), , , . , , . , .. , .

, FTP, , FTP , ftp.exe. FTP-, ..

, , . , .

0
  • , .
  • , , .
  • .
  • . , .
  • .
  • , FTP.
  • , FTP'd , .
  • .

, fancier. FTP, , - , .

. , , , . .

FTP- , , .

0

, BlockingCollections.

, FileSystemWatcher .., BlockingCollection. , .

var availableFiles = new BlockingCollection<string>();
var processedFiles = new BlockingCollection<string>();
var newFiles = new HashSet<string>();

...
lock (newFiles) {
    foreach (var file in Directory.GetFiles())
        if (!newFiles.Contains(file)) {
            availableFiles.Add(file);
            newFiles.Add(file);
        }
}

ftp ,

foreach (var file in availableFiles.GetConsumingEnumerable()) {
   SendFileOverFtp(file);
   processedFiles.Add(file);
}

,

foreach (var file in processedFiles.GetConsumingEnumerable()) {
    lock (newFiles) {
       File.Delete(file);
       newFiles.Remove(file);
    }
}

Another alternative is that the producing stream also reads files into memory and deletes them. In this case, you can skip the last step and the newFiles collection

0
source

As the owner of the FTP server in this situation, I also ask that you find a way to subscribe as much as possible.

Turning an input on / off is often more "expensive" (in terms of computing, locking configurations, etc.) than individual file transfers.

0
source

More articles:


All Articles