Good question, and it pertains to FCP7 too. Generally effects are what get rendered. Those four options under the Video Processing tab determine how the effects are calculated, so they affect the quality of the effects in the final QuickTime export. They incidentally affect whether unrendered real time playback is possible.
A further question is what the four options mean.
.....1. Always Render in RGB
.....2. Render in 8-bit YUV
.....3. Render 10-bit material in high-precision YUV
.....4. Render all YUV material in high-precision YUV
What Apple means by them is not exactly what they say. For example "high-precision" YUV reasonably means 32-bit floating point precision. But for Apple it also means that extended-range YUV which includes super-whites and sub-blacks will not be clipped (if the effects themselves allow them). They are clipped with option 2. Who knows Apple's exact meanings for the four options?
Dennis Couzin
Berlin, Germany