Geek asks and answers

The preferred (or recommended) way to store large volumes of simulation configurations, triggers values ​​and end results

I work with some network simulator. Having made some extensions for this, I need to do many different simulations and tests. I need to write:

  • simulation script configuration
  • the values ​​of some parameters (e.g., buffer sizes, signal quality, position) per device per unit time t
  • final results calculated from these recorded values

Secondary data is needed to perform some visualization after performing the simulation (a simple animation showing some statistics over time).

I am using Python with matplotlib etc. for subsequent data processing and for writing the right application (now considering pyQt or Django, but this is not the topic). Now I wonder what would be the best way to store this data?

My first suggestion was to use XML files, but that might be too much overhead from the XML syntax (I mean, files can grow to very large sizes, especially for the second part of the data type). So I tried to create a database ... But it also seems wrong to me ... Maybe a combination of both?

I tried to find some tips on Google, but I didn’t find anything special. Have you ever had to store such data? How did you do that? Is there a "design pattern" for this?

+5
3

:

, (, CSV , , ) , :

  • : ( ) , , . ( ).

  • : , , 3 , . : (. ) , . , . , , (, , ), .

  • : . , , , .

:

, ( ?) , , . , , , . , , ( ..), . , , OMNeT ++ : , vec sca ( ). , ( ). , . to , CSV/MATLAB, ( ) / .

, , .

+4

, , - , " "?

CSV , ! , . - , .

+1

. SQL-/, , SQLite, ( , 3+ GB).

Else - that is, for each simulation scenario, you may need a dedicated data structure to store the results - and you do not need any SQL function, then you might be better off using a more free solution (document-oriented database, OO, file system + csv, anything).

Please note that in the second case, you can still use SQL-db, but you will have to dynamically create tables for each result set and, of course, dynamically create the corresponding SQL queries.

+1
source

More articles:


All Articles