You just need to change the values so that each point has four coordinates on the same line:
result = [X_old(:) Y_old(:) Z_old(:) ones(numel(X_old),1)] * M;
Each line resultgives new coordinates for each point.
, , A times matrix B A B.
, (- ),
result = X_old(:)*M(1,:) + Y_old(:)*M(2,:) + Z_old(:)*M(3,:) + ones(numel(X_old),1)*M(4,:);
, , , X_old, Y_old, Z_old, :
s = size(X_old);
X_new = reshape(result(:,1), s);
Y_new = reshape(result(:,2), s);
Z_new = reshape(result(:,3), s);