Casey,
Compressor's DVD presets make a 23.976p MPEG-2 (with NTSC resolution) from either 24p or 23.976p (regardless of its resolution). If you use DVD Studio Pro to make the DVD files from the MPEG-2, it will preserve the 23.976p. So the unsmoothness you're seeing must be due to the player, not the MPEG-2 itself. The player could be adding pulldown (changing the frame rate) or it could be skimping in decoding the MPEG-2. Your description suggests the latter.
Professional DVDs use better encoders than Compressor's, requiring skillful guidance. Best compression involves constant compromise between motion smoothness, detail, and color depth. Different scenes, different shots, and even different portions of the frames warrant different compromises. The hardware based CINEMA CRAFT Xtream Encoder is one example of what professional studios use.
Back to our semi-pro universe; experiments you might try, just to see where the problem lies:
...Try
reducing the bitrate from Compressor's preset values.
...Try changing the GOP size to 6, and from closed to open.
...Try using a completely different player.
Dennis Couzin
Berlin, Germany