Flaw in ProRes in FCP7

Part I. Experiment
Try this simple experiment in FCP7.

1. Make a 1920x1080 progressive sequence in ProRes HQ codec; framerate doesn't matter. The default FCP7 render option for ProRes is "high precision YUV".
2. Insert 100 frames of black slug.
3. Export to a .mov file. The file is 2.2 MB.
4. Return that .mov file to the sequence, replacing the slug.
5. Apply the FCP7 gamma filter with gamma = 2.
6. Export the filtered clip to a .mov file. The file is 83.8 MB!

The FCP7 gamma filter is 10-bit capable and should not modify a video black picture at all. What has gotten into the file to make it 38 times as large?

OK, old FCP7 has a problem with its implementation of ProRes codecs. An explication is given in Part II below. But is it just a problem in FCP7? Hopefully others will try the experiment in newer video editing and processing softwares to find out. Caution: the FCP7 gamma filter at gamma = 2 is what others might call gamma = 1/2; it lightens the darker tones. Also various FCP7 concepts -- "sequence settings", "slug", "render", "export", etc. -- might require creative re-interpretations in the newer softwares.

* * *

Part II: Explication
You can see with your eyes what has gotten into the bloated ProRes file. In the FCP7 canvas, when examined closely, it isn't really black: it has very many very dark grey horizontal lines. The luma waveform videoscope shows a thickened fuzz over the "black" level. Things get more obvious with gamma = 5 filtering in place of gamma = 2. The fine horizontal lines then even show colors: magentaish with greenish zigzags between them. The videoscopes are on fire.

The line pattern repeats every 8 pixels. The different flavors of ProRes each show a somewhat different 8x8 pixel block. Here it is for ProRes XQ with gamma = 5.



Every codec is an encoder plus a decoder. The ProRes encoding of a black movie is very efficient -- making the tiny export file -- but the high precision ProRes decoding of that file is a pixel-by-pixel mess. The FCP7 gamma filter (>1) applied to the mess of values, all very near to black, exaggerates their deviations from black.

The DCT (discrete cosine transform) underlying the ProRes codecs is mathematically inelegant (pathetic, really) for producing disparate values on each 8x8 block of pixels even when the input values are uniform.

In our case, a full frame of black pixels, all 64, 512, 512 in 10-bit YUV, went into the ProRes encoder and they all should have come out of the ProRes decoder as 64, 512, 512. I estimate that for ProRes XQ the decoder gave luma values between 64 and 64.25, and chroma values between 511.8 and 512.2. A proper implementation would have cleansed these values to 64, 512, 512, simply because it is a decoding of a 10-bit video. Instead FCP7 passes the high precision YUV values, duly transformed to high precision RGB, to its gamma filter which then exaggerates the deviations, yielding the magenta and green and grey pixels in what should be all black.

High precision handling of 10-bit material is a good thing during image processing. FCP7's error was in not knowing when to round.

Dennis Couzin
Berlin, Germany