What is the most efficient (but flexible enough) way to store multidimensional data of variable length?

I would like to know what is the best practice for efficient storage (and subsequent access) of sets of multidimensional data arrays with variable lengths. The focus is on performance, but I also need to be able to handle changing the length of an individual dataset at runtime without undue cost.

Note. I know this is a rather lengthy question, but I looked around a lot and could not find a solution or an example that describes the problem at hand with sufficient accuracy.

Background

Context is a computational fluid dynamics algorithm (CFD) based on the Galerkin discontinuous spectral element method (DGSEM) (see Kopriva (2009), Introducing spectral methods for partial differential equations). For simplicity, suppose a 2D data layout (this is actually in three dimensions, but expanding from 2D to 3D should be simple).

I have a grid consisting of Ksquare elements K( k = 0,...,K-1), which can be of different (physical) sizes. Inside each grid element (or "cell") KI have N_k^2data points. N_k- the number of data points in each dimension and may vary between different grid cells.

At each data point n_k,i(where i = 0,...,N_k^2-1) I need to save an array of solution values ​​that is the same length nVarsin the entire domain (i.e. everywhere) and which does not change at run time.

Dimensions and changes

The number of grid cells Kfrom O(10^5)to O(10^6)and can vary at run time.
The number of data points N_kin each grid cell is between 2and 8and can change at run time (and may be different for different cells).
The number of variables nVarsstored at each grid point is around 5to 10and cannot change at run time (it is the same for each grid cell).

Requirements

. ( ). , K N_k , , , .

(.. , ) . N_k, , , .

. , , , . , , , , .

, , , , .