Geek asks and answers

SQL VS Database. Multiple Flat Files (Thousands of Small CSVs)

We are developing an update for the current system (C ++ \ CLI and C #). The system will collect small (~ 1 Mb) volumes of data from ~ 10 thousand devices (in the near future). Currently, they are used to save device data to CSV (table) and store all this data in a wide folder structure.

Data is inserted (created / added to the file, creates a folder) that are never updated / deleted. Data processing is performed by reading many CSVs into an external program (e.g. Matlab). It is mainly used for statistical analysis.

It is possible to start storing this data in an MS-SQL database. The process time (reading CSV for an external program) can be up to several minutes.

  • How can we choose which method to use?
  • Does one method use significantly more memory than the other?
  • Roughly speaking, when is reading source data from a database faster than reading CSV? (10 files, 100 files? ...)

I would be grateful for your answers, "Pros and Cons" are welcome.

Thank you for your time.

+5
source share
4 answers

Well, if you use data in one CSV to get data in another CSV, I would suggest that SQL Server will be faster than anything you come up with. I suspect that SQL Server will be faster in most cases, but I cannot say for sure. Microsoft has put in a lot of resources to make the DBMS, which does exactly what you are trying to do.

, , . , SQL Server, , , , .

:

  • ,
  • .

:

  • SQL Server .
  • , SQL Server, , Express

!

+3

.

, , CSV? (10 , 100 ?...)

. ( , ), .

, ?

, , . , ? , , , .

, ?

. . CSV , . DB. , .

, , , , DB . , - , 100 .

+1

, , . , , SQL Server, , SQL Server.

0

ms-sql, .

. . . . SQL-, , .

. , foxpro , . , .

, , , - . 10 . , .

0

More articles:


All Articles