If you are talking about X11, you can write a Composite Manager for this. Your composite manager will probably want to implement these effects using OpenGL, as texture_from_pixmap plays an important role.
You can start with these two keywords. Perhaps this option is not to write everything, but to write Compig instead. Another keyword :)