there are a LOT of questions here... maybe your ram IS bad, what brand is it?
there is a LOT of logic to NOT upgrading ANYTHING mid project. but in your case i dont see that you have much choice.
here is my take on your problems.
major problem. i have heard that older versions of fcp (3x) is not compatible with qt7/tiger. SUGGESTION, install 5
second problem. insufficient ram. suggestion - you simply must get more! other suggestion. many macs require upgrading ram in matched pairs. is yours one of them and if so, did you?
third problem. your FCP AND DVDSP are way outdated. install new versions of both apps.
to be honest, if i were you, id zero my boot drive and reinstall with the new apps. then try it all again.