According to the OpenGL Red Book, Appendix F, to calculate the action on a normal vector, you can use the regular 3D transformation matrix, which:
normalTransformed = transpose(inverse(M)) * normal
However, although the orthogonal plane associated with the transformed normal is actually parallel to the transformed surface, it can happen that the transformed normal vector itself points in the opposite direction to what I would expect, i.e. “in” the surface and not “outside” the surface.
If I want normalTransformed to point in the right direction (i.e., in the same direction in which it points when the surface to which it is attached does not transform), how can I do it mathematically?
Example
Suppose my normal surface is (0,0,1), and my transformation is a shift of 10 in the Z direction. Then the transformation matrix M:
1 0 0 0
0 1 0 0
0 0 1 10
0 0 0 1
The transposition (reverse (M)) is as follows:
1 0 0 0
0 1 0 0
0 0 1 0
0 0 -10 1
As applied to the surface normal (0,0,1), i.e. (0,0,1,1) in homogeneous coordinates, this gives:
normalTransformed = (0, 0, 1, -9)
Return from uniform coordinates:
(0, 0, -1/9)
Normalization to length 1:
(0, 0, -1)
Which indicates the opposite direction compared to the original normal vector (0, 0, 1).