Perl: problem with storing a huge hash on disk?

I am working on a Perl project that involves creating a hash with approximately 17 million keys. This is too large to be stored in memory (about 10 million keys will be stored on my laptop memory). I know that the solution is to store data on disk, but I am having problems with this in practice. Here is what I tried:

DB_File

use strict;
use DB_File;
my $libfile = shift;
my %library;
tie %library, "DB_File", "$libfile";
for (my $a = 1; $a < 17000000; a++) {
    # Some code to generate key and value #
    $library{$key} = $value;
}

This gives me a segmentation error: 11 part of the way through the loop, for reasons that I don't understand.

Berkeleydb

use strict; 
use BerkeleyDB;
my $libfile = shift;
my $library = new BerkeleyDB::Hash
    -Filename => $libfile,
    -Flags => DB_CREATE;

for (my $a = 1; $a < 17000000; a++) {
    # Some code to generate key and value #
    $library->db_put($key, $value);
}

, 15 , , , . , ; , ( ~ 4 ), , , 15 . , , , BerkeleyDB ~ 15 ?

DBM:: Deep

use strict; 
use DBM::Deep;
my $libfile = shift;
my $library = new DBM::Deep $libfile;

for (my $a = 1; $a < 17000000; a++) {
    # Some code to generate key and value #
    $library->put($key => $value);
}

, , , : 5 ~ 22 . , .

.

UPDATE

+3
2

btree HUGE BerkeleyDB, " ". - .

: : comp.mail.sendmail , HUGE BerkeleyDB 20 btree "key sorted". , soft, SQL, SQL BerkeleyDB. (virtusertable, sendmail- > postfix)

+2

PostgreSQL.

, , varchar ,

, , Pg:: BulkCopy .

100 , , COPY FAIL, PostgreSQL , , , VACUUM FULL . ( 5 , , .)

ps: DBD:: Pg: https://metacpan.org/pod/DBD::Pg#COPY-support

, , Redis memcached MAXMEMORY POLICY

0

All Articles