contains()will return true if the object is a descendant of the caller. This will return the truth to indirect descendants, children of children, etc.
Determines whether the specified display object is a child of the DisplayObjectContainer instance or the instance itself. The search includes the entire display list, including this DisplayObjectContainer instance. Grandchildren, great-grandchildren, etc. All are true.
http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/display/DisplayObjectContainer.html#contains%28%29
You can check the parent:
if(modalMessage && modalMessage.parent && modalMessage.parent == this)
Or, for a general purpose solution:
if(modalMessage) {
if(modalMessage.parent) DisplayObjectContainer(modalMessage.parent).removeChild(modalMessage);
modalMessage = null;
}