Geek asks and answers

Efficient storage and reading of large data files

I am working on a project where I have large input files that come from pdes numerical solutions. The data format is as follows.

x \t y \t f(x,y)

For each y value, we have several x values ​​and a function value estimated at each point. The size of the data I'm dealing [-3, 5]x[-3, 5]with is in increments 0.01of each dimension, so the original data file is quite large (near the 640,000records). Reading it into memory is also very important, because the tools I'm working on must read several raw data files of this type at the same time.

I am using Python.

Is there a way to store and read data like this in Python? The idea is to incorporate a tool that masses these raw data files into something that can be read more efficiently. I am currently working on data interpolation and storing some coefficients (substantially replacing memory due to computational time), but I am sure that there should be an easier way that helps both memory and time.

Thanks SOCommunity!

PS: I saw related questions in Java. I am fully working on Python here.

+5
source share
2 answers

If you are using numpy (and you should probably be), numpy.save/ numpy.savezand numpy.loadshould be able to handle this pretty easily.

For instance:

import numpy as np
xs = np.linspace(-3, 5, 800)
ys = np.linspace(-3, 5, 800)
f_vals = np.random.normal(size=(xs.size, ys.size))
np.savez('the_file.npz', xs=xs, ys=ys, f=f_vals)

, 5 .

+3

, , Python?

, Sqlite, SQL- .

0

More articles:


All Articles