The blender performs both selection and deactivation of indentation, based on dynamic AABB acceleration structures from the Bullet physics library. Occluders and objects are represented by their bounding volume (frame aligned along the axis).
Viewing the burr truncation simply traverses the AABB tree given the camera truncation.
The occlusion hole is based on the occlusion buffer, a kind of depth buffer that is initialized using a very simple tool for rendering programs: a basic ray tracer that uses dynamic AABB tree acceleration structures. You can choose the resolution of the occlusion buffer to trade with accuracy.
See also the documentation http://wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.49/Game_Engine#Performance
Blender Blender:
blender\source\gameengine\\Bullet\CcdPhysicsEnvironment.cpp
bool CcdPhysicsEnvironment:: cullingTest ( PHY_CullingCallback, void * userData, PHY__Vector4 *, int nplanes, int occlusionRes)
SDK Bullet Physics ++ Bullet/Extras/CDTestFramework.