Performing an intensive batch process in PHP and preventing memory exhaustion

I have several thousand records (stored in a table in a MYSQL table) that I need for a "batch process". All entries contain large JSON. In some cases, JSON exceeds 1 MB (yes, my database is much larger than 1 GB).

I have a function that captures a record, decodes JSON, modifies some data, transcodes the PHP array back to JSON and saves it back to db. Pretty simple. FWIW, this is in the context of the CakePHP application.

Given an array of identifiers, I'm trying to do something like this (a very simple code layout):

foreach ($ids as $id) {
    $this->Model->id = $id;
    $data = $this->Model->read();
    $newData = processData($data);
    $this->Model->save($newData);
}

The problem is that PHP runs out of memory very quickly. When starting foreach like this, it is almost as if PHP was moving from one record to another without freeing up the memory needed for previous operations.

Do I still need to start the loop in such a way as to free up memory before moving on to the next iteration of the loop so that I can process a huge amount of data?

Edit: Add more code. This function accepts my JSON, converts it to a PHP array, performs some manipulations (namely, reconfigures the data based on what is present in another array) and replaces the values ​​in the original array. JSON is a lot of layers, therefore very long foreach loops.

function processData($theData) {
    $toConvert = json_decode($theData['Program']['data'], $assoc = true);
    foreach($toConvert['cycles'] as $cycle => $val) {
        foreach($toConvert['cycles'][$cycle]['days'] as $day => $val) {
            foreach($toConvert['cycles'][$cycle]['days'][$day]['sections'] as $section => $val) {
                foreach($toConvert['cycles'][$cycle]['days'][$day]['sections'] as $section => $val) {
                    foreach($toConvert['cycles'][$cycle]['days'][$day]['sections'][$section]['exercises'] as $exercise => $val) {
                        if (isset($toConvert['cycles'][$cycle]['days'][$day]['sections'][$section]['exercises'][$exercise]['selectedFolder'])) {
                            $folderName = $toConvert['cycles'][$cycle]['days'][$day]['sections'][$section]['exercises'][$exercise]['selectedFolder']['folderName'];
                            if ( isset($newFolderList['Folders'][$folderName]) ) {
                                $toConvert['cycles'][$cycle]['days'][$day]['sections'][$section]['exercises'][$exercise]['selectedFolder'] = $newFolderList['Folders'][$folderName]['id'];
                            }
                        }
                        if (isset($toConvert['cycles'][$cycle]['days'][$day]['sections'][$section]['exercises'][$exercise]['selectedFile'])) {
                            $fileName = basename($toConvert['cycles'][$cycle]['days'][$day]['sections'][$section]['exercises'][$exercise]['selectedFile']['fileURL']);
                            if ( isset($newFolderList['Exercises'][$fileName]) ) {
                                $toConvert['cycles'][$cycle]['days'][$day]['sections'][$section]['exercises'][$exercise]['selectedFile'] = $newFolderList['Exercises'][$fileName]['id'];
                            }
                        }
                    }
                }
            }
        }
    }
    return $toConvert;
}

Model- > read() Cake db . , , - , .

+5
1

, , , .

,

foreach ($ids as $id) {
processData($data);
}

function processData(&$d){}

http://php.net/manual/en/language.references.pass.php

+2