Geek asks and answers

Storage of scientific data in a relational database

I want to store hierarchical, two-dimensional scientific data sets in a relational database (MySQL or SQLite). Each data set contains a table of numerical data with an arbitrary number of columns. In addition, each data set may have one or more children of the same type associated with a given row of the table. Each dataset typically has 1 to 100 columns and 1 to 1,000,000 rows. The database should be able to process many data sets (> 1000), and the read / write data should be fast enough.

What is the best database schema for storing such data? Is it wise to have a “main” table with the names, identifiers, and relationships of individual datasets and, in addition, one table per dataset that contains numerical values?

+3
source share
4 answers

Is it wise to have a “main” table with the names, identifiers, and relationships of individual datasets and, in addition, one table per dataset that contains numerical values?

How I would do it.

I'm not quite sure how the custom column works, because the data usually doesn't work. Despite this, it sounds like saving it, since the string, col, val can work beautifully.

, (max, min ..), , - .

, , SQLite , .

, , , .

+4

, . db, , , ,

, , "master", , .

, , .

+3

, , , . , - , , , . .

, ( , ), , .

zebediah49: " ? ?"

+2

We store a bunch of data like this in our own flat file. The file header contains enough information (timestamp, number of rows / columns, etc.) so that it can be read. Then metadata about this data is in the database. At a minimum, this is the location of the file, but may contain other information about the data. For example, we combine data into proxy variables that summarize details at a high level. Typically, this summary is good enough, but if necessary, we can read the file for all the details.

+2
source

More articles:


All Articles