Thin green edge around frame of MPEG2 output

Hello,

I'm creating a DVD using outputs from Compressor. I have an hour long program, plus DVD extras that are short 1-minute pieces or so. I notice that when I playback the DVD on my laptop, which shows the edges of the frame, I can see a thin green edge to many shots...but not all. E.g. it doesn't show up in all black background title credits, but then shows up in the first shot of the film, then disappears, then shows up again in other shots. Most of my film was shot on HDV, with some SD uprezed to Apple ProRes. I don't see a correlation between the green edge in either type of footage.

In the DVD extras, it shows up clearly in a couple of them, but not others. I didn't use the exact same settings to output these 'extras', but they're not terribly different from each other.

Any ideas?

Thanks
Greg

Quick update on the above situation. The green edge appears throughout the entire clip, not just on certain shots. It's harder to see in certain shots because they're brighter. Also, it's not a solid green line, but a green cast to the image in the last few pixels all the way around the frame.

I'm finding that this occurs when I in fact have Frame Controls on. I was only able to get rid of the lines when I follow these steps:

1. Output film as Quicktime (self contained using full HD render)
2. Generate an SD 720x480 (16:9 pixel aspect) QT downconversion from the HD QT WITH FRAME CONTROLS OFF
3. Make MPEG2 from the SD 720x480

Ideally and from past experience you want the frame controls ON for the downconversion, but this generates the green edge. I cannot get rid of it otherwise. Still running other tests...

Why are you changing the GOP length and style from the default values? Only under very specific circumstances should you do that (and if you know a lot about compression).

Take a few seconds of your QT output from FCP (you can trim it in QT Player 7, if you want). Take that trimmed clip into Compressor and just select one of the best quality VBR DVD presets. Don't make any changes to the settings. What does the result look like?


-Dave

I'll try your suggestion later tonight. I think I have tried it actually, but I've tried so many tests it's hard to remember. And yes I'm not outputting the whole film for each test, just a minute or two sample.

I generate my own MPEG2 settings based on how I understand them (although not being a trained compressionist) in trying to achieve the best quality I can. My feeling is that with a lower GOP number you naturally have twice as many I-frames in a given second, which translates in my mind to better quality in the long run than two I-frames for every second. I'm not so much concerned with a smaller total file size (it's under an hour after all) as with highest quality.

Kingg33 Wrote:
-------------------------------------------------------
> I'll try your suggestion later tonight. I think I
> have tried it actually, but I've tried so many
> tests it's hard to remember. And yes I'm not
> outputting the whole film for each test, just a
> minute or two sample.
>
> I generate my own MPEG2 settings based on how I
> understand them (although not being a trained
> compressionist) in trying to achieve the best
> quality I can. My feeling is that with a lower GOP
> number you naturally have twice as many I-frames
> in a given second, which translates in my mind to
> better quality in the long run than two I-frames
> for every second. I'm not so much concerned with a
> smaller total file size (it's under an hour after
> all) as with highest quality.

Do you know that when you choose a DVD MPEG-2 preset in Compressor and you look at the Frame Controls tab the little "Automatic" button is gray by default? That means that the frame controls are set to automatic ( even though the pop-up menu just to the left is grayed out and shows "Off" ).

Also, your choice of GOP length and structure may be playing havoc with Compressor and is a likely cause of your "green fringe" issue.

GOP lengths above 4 yield very little quality improvement. The most sensitive factor is bit rate. All you gain by using a smaller GOP length is increased output file size. Stick with the defaults (Closed GOP IBBP with 15 for NTSC or 12 for PAL). If you stray from the default GOP settings and go too high on the bit rate, you may decrease the playability of the resulting MPEG-2 file, as well.

For the best quality, I would suggest that you drop your HDV sequence material into a ProRes 422 (HQ or not) sequence with full raster frame size in FCP. Render that and do any filtering and color correction in the ProRes sequence. Then, export a self-contained QT movie.

Now, you may get better results if you first downconvert your exported ProRes QT movie file to SD dimensions, then do a separate MPEG-2 encoding.

You may also get a better quality MPEG-2 file by performing very careful denoising and chroma smoothing at various stages (I won't go into this further, as it's an advanced topic). I mention this to show you that there are many things that can directly affect the quality of your final output, but messing around with GOP settings is not one of them.


-Dave

P.S. - You can search the Internet for more information, such as this paper from Tektronix: "An Analysis of MPEG Encoding Techniques on Picture Quality," which is the one of the sources of my statement above about GOP length and structure.

Another option is to use another encoder, such as BitVice or Cinema Craft, both of which can do adaptive GOP length and structure determination automatically (as opposed to manually specifying those parameters in Compressor). The Cinema Craft encoder is probably the best software encoder you can get (it comes from their original, hardware-based system that has been used for the vast majority of high-end DVD encoding).

Hi Dave,

Thanks for the info. I have tried 15 GOP settings before and still the edge appears, but I'll try one of the presets.

The project is already in a ProRes 422 timeline, fully rendered and ready to go (although the pixel dimensions are still set on HD 1440x1080). Are you saying I should nest it all and drop it into a square/1920x1080 sequence?

I'm not afraid of any 'advanced' info like de-noising if you have the time to share it. It's really important to me to make this piece look as good as I can, barring just sending it to a posthouse and paying out the nose for an encode. I purchased the Neat Video filter and have been improving several shots already, but I'm sure you mean the filtering present in Compressor itself. If it's too much to ask of your time, then any links or leads are much appreciated. Thank you!

Greg

PS Yeah, I have been trying an HD to SD conversion first, prior to encode, based on a conversation with a posthouse. It got rid of the green edge (as long as frame controls were OFF), and looks pretty good, although there is an overall difference in contrast and chromaticity in the resulting MPEG2 (less contrast, less saturation) than the file I had previously output direct from FCP and the HD timeline.

PPS Lastly, I shouldn't do a reference movie QT for Compressor to crunch (as a test option), instead of a self-contained? I've read here and there that since a self-contained is re-compressed, it's better to go Reference Movie, but I have yet to try that.

Greg,

Re-read the end of my previous post (I just added a couple of bits of info).

Yes, by using one of the default presets, we can try to determine why the fringing is occurring. This is separate from the quality optimization aspect of the workflow.

I am not a fan of nesting sequences, except under certain conditions. You could simply create a new sequence at 1920 x 1080 (square pixels) and paste all of the tracks/clips into it. I always use a full raster HD sequence because it usually helps to preserve the highest quality text and graphics overlays on export (and the video imagery itself).

Denoising may help on certain source clips, as well as just prior to MPEG-2 encoding. MPEG encoders do much better with a bit of noise reduction applied beforehand (or maybe better put, MPEG encoders are less efficient when dealing with noisy source material). I don't use any of the filters in Compressor. Neat Video is nice, as are Video Purifier (Innobits) and DeNoise (RE:Vision Effects). We are talking about very slight amounts of filtering. Chroma smoothing can help with a lot of 8-bit source footage, as well (earlier on in editing).

The best quality output from FCP to Compressor, according to Apple, is to use the "Send to Compressor" command inside of FCP. This sends each frame to Compressor where it is rendered and processed (higher quality than rendering in FCP). But, it ties up FCP and probably takes longer than exporting (as a full render is being done regardless of the render status in FCP).

Hopefully, we can figure out why you're getting the green fringing and eliminate that. Then, you can focus on optimizing the output quality of your MPEG-2 encoding. The best quality may require the use of another encoder like Cinema Craft. There are limits when going to highly compressed formats.

Does this info help?


-Dave

Yes, this is very helpful, thank you! It's just frustrating because I've used Compressor for ages and haven't had this issue before. I'm going to look at some old outputs of other projects that I burned to DVD and see if it's there too. I would notice this though, so it's perplexing.

I'd love to use Cinema Craft but I can't afford it ($700 for software plugin to Compressor). I haven't heard of Bitvice so I'll look it up. The posthouse I've talked to would charge $400 for downconversion, then encode to MPEG2 via Cinema Craft, a two step process. Can't afford that either really, but I might bite the bullet and give it a shot, given what people say about that program.

That's interesting info about Sending to Compressor as a better render process than FCP itself... Like I think I said, that's where I started in all this, but since I'm a do-it-myself kind of guy, I never actually even tried one of the Compressor presets!

And yes that makes sense to me too to not nest and just copy/paste. Just making sure.

So drop the 'color smoothing 4:1:1' filter across the board into all clips in FCP? Is that what you meant? Or is there a third party plugin filter you mean?

Thank you!

Greg

PS I also don't think in terms of using filters in Compressor, but I did find in one of my workflows that putting an HD-to-SD downconverted QT through the Sharpen Edges filter (just a touch at 5.0%) when making the MPEG2 helped with the softening of the image I was getting in the downconversion... Just food for thought

Quote
Greg
And yes that makes sense to me too to not nest and just copy/paste. Just making sure.

So drop the 'color smoothing 4:1:1' filter across the board into all clips in FCP? Is that what you meant? Or is there a third party plugin filter you mean?

Nesting is fine if used properly. In this case, it's probably better to copy and paste everything from one sequence into another.

I've only used the filters from Graeme Nattress's Big Box of Tricks (for chroma smoothing). I don't know how well the Color Smoothing filters in FCP work. I haven't ever applied these filters across the board, either. I only use them when needed on specific clips.

The denoising tools are nice to use just prior to MPEG encoding. A lot of this stuff requires testing and is somewhat subjective.


-Dave

Great, thanks so much for all this! I'm familiar with Nattress' stuff but don't own it. I appreciate your time in answering this issue.

Greg

Hi Dave

I copied and pasted my edit into a 1920x1080 square pixel ProRes 422 HQ timeline (with high precision YUV render), "Sent it" to Compressor, chose the preset DVD Best Quality 90 minutes... and the green edge is as prominent as ever. Ugh. This is driving me nuts. I took some screen grabs of it, but I don't know how to attach to this forum, in case you want to see it.

So of all the methods I've tried, the only files I have without the green edges are when I export the film as a self-contained movie file, downconvert that HD file to an SD (ProRes as the codec but 720x480 16:9 pixel ratio) QT?with Frame Controls OFF (although I haven't tried or looked for a preset, not sure if there is one for downconversion)?and then encode the MPEG2 from that SD clip, again with Frame Controls off, but you don't need it at that point anyway. The resulting MPEG2 is okay-looking, but I feel like it could be better. I still need to actually burn it to a DVD and watch on my various monitoring options...

Any other ideas? I'm all ears. Thanks!

Greg

BTW, any chance you're a fan of Run DMC? I couldn't help but think of DMC with your pseudonym...

Oh, also, in case this info is of any use, I noticed something odd today in my outputs that I've never seen before either.

I output a self-contained QT of five minutes of the film, and used it to generate an SD downconversion QT, as I mentioned above, but I did so a half dozen times with a half dozen slightly different settings in Compressor. Some had Frame Controls on, some off, minor sharpening, etc. What I noticed in comparing them all was that one?just one?actually showed more of the original image area than the others, like the others were cropped just slightly, by 5-10 pixels on each side. BUT, the weird thing is, when I opened up the original HD QT I was using to make those downconverted clips with...it was cropped too. Bizarre! The 'special one' clip showed more image than the clip it came from... This makes no sense to me, and I triple checked it. It's like the Quicktime app/viewer doesn't actually show all pixels in my HD excerpt QT or something. Maybe there's something happening where it's those overlapped pixels that are giving me the green edge ultimately. I don't know. Naturally, I checked the settings of that 'special one' clip, to see how it differed from the others. Aside from a slight change in sharpness settings, it differed only this these ways: 1) I had checked OFF the "Add clean aperture information" in the video/audio settings pane in the inspector, and 2) I had the little star button to the right of Frame Controls activated, BUT the actual controls set to OFF (rather than grayed out). That's it.

?

Greg,

So, when exactly do the green fringes show up?

If you could export a few seconds of your original sequence and post it on Dropbox or one of the free file sharing services, it would help (I'd do some quick testing). This would be your HD ProRes export from FCP.

Then, copy your settings summary from Compressor for the step that results in the green fringing (or maybe save your batch, i.e., the .compressor file). That way, we can see your settings.

At least we can try to reproduce the behavior...


-Dave

P.S. - I don't mind Run DMC, but it doesn't have anything to do with my silly site username...

Quote
Greg
It's like the Quicktime app/viewer doesn't actually show all pixels in my HD excerpt QT or something.

QuickTime has a weird (non-obvious) way of presenting frames during playback. For analog video, it may make sense to slightly crop some of the "noise" at the edge of the frame (and analog TVs usually cropped the "overscan" area).

Quote
Greg
had the little star button to the right of Frame Controls activated, BUT the actual controls set to OFF (rather than grayed out).

For Frame Controls to be on automatic, the "star" button should be a darker gray color and the pop-up menu to its left grayed out (dull gray), or inactive, showing "Off". This is confusing as heck. If the "star" button is selected (lighter gray) and the pop-up menu made active, you can select "Off" or the other settings.

-----

If you can upload a sample QT movie clip and a text file or .compressor file of your various settings, or a document describing everything, it would make it easy for us to help you troubleshoot this issue.


-Dave

P.S. - You can only post links or embed items here that are hosted elsewhere...

Great, I'll do that. I have a lot of work on another job tomorrow, but I'll try to get it done. If not, it'll have to be Monday. Thank you so much!

Greg

Hi Dave,

It's been some time since I have dealt with this, but I thought you'd be interested to know where things are at.

I got sick of trying things through Compressor and being unhappy with the encode with or without the green edge being resolved, so I farmed it out to Cine-o-matic in Minneapolis. I sent them a self-contained Quicktime (Apple ProRes 422 HQ), and they used Cinema Craft to do the encode (after first downconverting the master QT to a ProRes SD file).

At first, they sent me a check disc and, although the encode as a whole looked better than what Compressor was giving me at it's highest quality settings, it was still generating a green edge all the way around. I brought it up with them and they didn't know why it was there. However, as a stab in the dark, the tech person changed the output workflow a little. Instead of downconverting the Master ProRes QT file to SD ProRes, he downconverted it to an Uncompressed SD file, then used that for the encode...and voilá, the edge disappeared. The resulting encode looks amazing; I'm glad I spent the extra dollars on it.

This leads me to think that there is a basic problem with the ProRes codec, perhaps in how it handles (originally) HDV material, or the green signal. What do you think?

Best,
Greg