A sloppy camera usually does not include zooming. The image rotation component will also be very small and can probably be ignored. You can probably get enough results with only 2D translation.
What you probably should do is determine your shake path in time — the amount of motion the image from the original static video for each frame — and then shift each frame by that amount.
You might want to crop your video a bit to hide any empty parts near the border of the image, the remaining empty areas can be filled by drawing. This path should be relatively smooth and not completely random jitter, as you imitate the movement of a physical arm.
To make the effect more convincing, you should also add motion-blur.
The direction of this blur coincides with the direction of shaking, and the amount depends on the current speed of shaking.