5cannot be implicitly converted to long doublein your case. You need to change it to 5.0to make it long double or to explicitly call the function yourself to implicitly convert to work:
int m = 5.0_tryit;
OR
int n = operator"" _tryit(5);
(verified as with clang version 3.1 (trunk) (llvm/trunk 155821))
This SO question has a good explanation of the rules.
( , abarnert, , -std=c++11 ).