Do you still have the DVD? Because I would try re-ripping the video straight to DV directly off the disc. Handbrake doesn't offer this though, you'll need to use something like MPEG Streamclip or DVDxDV. And this is a non-commercial DVD, right?
MP4s are so heavily compressed, they're not going to upscale well no matter what you try. What you've done is squeeze your original video down to a postage stamp, and you're trying to blow it back up into a poster. Ain't gonna be pretty...
JK
_______________________________________
SCQT! Self-contained QuickTime ? pass it on!