Transfer decrypted data from CryptoStream to SecureString

I am trying to pass decrypted data to System.Security.SecureString from System.Security.Cryptography.CryptoStream. If you look at the IL code for CryptoStream, it seems to be buffering the converted data like any other System.Buffer. I could read the data in large chunks and then add it to my SecureString; The buffered data is short-lived (inside the method), but SecureString itself may hang for a long time.

Does it make sense to associate in-memory buffered data with a GCHandle to reduce the attack surface by preventing copies of the GC, or is it completely unnecessary? Also, is there a better way to put converted data from CryptoStream to SecureString?