It's completely normal, and I think it's really just because a Jeep isn't built like a car. There is a fair amount of give in the rubber isolators where the body mounts to the frame, and they're designed that way to help prevent damage while you're offroading. The transmission (including the shifter) is actually mounted to the frame, so what you're actually seeing is the body bouncing slightly on the frame when you hit a bump. It's nothing to worry about.
The only time I've had a problem shifting with my manual is when it's at least -25° and the engine hasn't been given enough time to warm up... and even then, it really only seems not to like 2nd gear.
