Which is better- HDV transcode or re-capture?

Two possible workflows for client -

Transcode previously captured HDV footage into ProRes422 using Media Manager...

Or re-capture HDV footage directly into ProRes422.

Which yields better quality? Better time savings?

- Loren
Today's FCP keytip:

Play from In to Out with Shift - \ (Backslash) !

Final Cut Studio 2 KeyGuide? Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

> Better time savings?

Compressor should be much much faster if you're on an octo.

Haven't compared the quality, though I'd capture straight as ProRes if there is enough space. Else, if it's already captured, transcoding may yield marginally better results, or you could just render in ProRes.



www.strypesinpost.com

[ transcoding may yield marginally better results, or you could just render in ProRes.]

Much thanks, Gerard-- I guess I'm asking for edit purposes. I don't want to constantly have to render effects, which would be the case stacking up tracks or applying effects in HDV native. It's just not an edit codec. So we're out of there one way or another.

And here's another consideration-- possible film out. Now, I've seen tests of film out from MiniDV, SD, HDCAM and RED. Client for this piece is targeting for either broadcast or theatrical for his Sony Z1U HDV footage.

Compressor certainly makes sense if we choose to transcode existing footage. Yes assume an octopus with 4TB storage using a CalDigit RAID card.

I'm interested in ProRez. What do you think of downrezzing from HDV 1080i to say, ProRes422 HQ at 720 X 1280? That's only 3/4 the storage of ProRes 1080 (SQ). What's the quality hit?

Any other comments? I would like to hear from folks who've compared quality working one method over the other.

We certainly have options we didn't have a couple years ago.

- Loren
Today's FCP keytip:

Play from In to Out with Shift - \ (Backslash) !

Final Cut Studio 2 KeyGuide? Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

I have much the same question regarding quality - but think I'm going to go the transcode route on my current project (we are mixing ProRes 1080 SQ and some client supplied HDV). There's not a lot of the HDV and it's a short project.

The giggle was trying to convince the client that redoing graphics from our last project with them (SD NTSC) from 2004 was going to be necessary. That frame size comparison sheet from Jude (http://www.lafcpug.org/phorum/read.php?1,212388,212388#msg-212388) sure did the trick. It's the first time in a while I've heard "Ooooh - never mind." to a budget item.

Ian

I wouldn't do either one of those things, Loren.

I work in HDV a lot, and I have a very particular workflow. I won't say it's the RIGHT way, but it works flawlessly for me.

I capture all my footage as straight HDV, right off the tape, over Firewire. It's coming in off an XL H1, which has an HDSDI output on it, so if I wanted I could capture through my Kona card as whatever I wanted. But I bring it all in over Firewire instead, as pure HDV.

Then I cut that footage into a 1920x1080 ProRes timeline. On my Mac, I can play HDV in a ProRes timeline at full quality in real time without rendering, so it's just like working with native footage. But when I do render, the system effectively transcodes my whole timeline to ProRes, upscaling it from 1440x1080 to 1920x1080. The only difference is that it only transcodes the bits of footage I use in my timeline. Those bits I didn't cut in never get converted, which saves tons of time and disk space.

On projects where I'm going out to SD and where I don't have any on-screen text ? titles, lower thirds, whatever ? sometimes I'll choose to go with a 1440x1080 timeline instead of 1920x1080. This saves Final Cut from having to do an upscale. But most of the time I'm mastering to full-raster 1920x1080 anyway, so I just go with that format.

As for questions of quality, I really don't think you're going to see a difference at all. The ProRes Quicktime component is what actually does the transcoding, and it's the same whether you're capturing to ProRes with "capture now," or transcoding with Compressor, or rendering out of Final Cut. A fun little experiment would be to handle the same little bit of footage in all three ways, then pull difference mattes. My guess would be that the footage comes out bit-for-bit identical, since it's the same computer code in all three instances. But I'm not willing to put money on it or anything. ;-)

I'm not sure what the mathematical differences are, but I find it hard to spot much difference between ProRes and ProRes HQ. So you could possibly save even more space by going ProRes 'standard edition' as well. Try some comparisons - I'd be interested in the results.

I've done exactly the same thing as Jeff Harrell (above) on two features also shot on an XL H1, but we had a Firestore hard drive on the camer and used the internal tape for backup.

I just dropped the 24p files from the camera into a ProRez Hi quality timeline and it all seems to work fine.

Harry

Harry Bromley-Davenport.

Jeff writes-
[I capture all my footage as straight HDV, right off the tape, over Firewire. It's coming in off an XL H1, which has an HDSDI output on it, so if I wanted I could capture through my Kona card as whatever I wanted. But I bring it all in over Firewire instead, as pure HDV. ]

Very interesting choice! If I had SDI I'd be using that in a heartbeat.

You didn't mention your hardware-- an Intel octopus?

[Then I cut that footage into a 1920x1080 ProRes timeline. On my Mac, I can play HDV in a ProRes timeline at full quality in real time without rendering, so it's just like working with native footage.]

Then it's no longer HDV, right? It's unrendered ProRes playing realtime? Are you applying effects, titles, color correction, et al?

Illuminating, folks.

Is there a white paper somewhere on how the image looks coming from HDV to ProRes or HQ to large projection? Pro's at the post-NAB SMPTE meeting I attended a year ago were cooling off on HDV as "looking like crap" on anything larger than 42".

- Loren
Today's FCP keytip:

Play from In to Out with Shift - \ (Backslash) !

Final Cut Studio 2 KeyGuide? Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

>downrezzing from HDV 1080i to say, ProRes422 HQ at 720 X 1280? That's only 3/4 the
>storage of ProRes 1080 (SQ).

That doesn't sound right. You're effectively introducing subsampling on pixels if you eventually need to go out full 1080. The render pipeline doesn't up or downscale images, from what i know, it only decompresses and recompresses images and calculates in 4:4:4 YUV space.

HD-SDI outs may transport at full 1920x1080 frame size, but it also does other stuff like chroma smooth, etc. So the results may be different. But naturally, you'd want to be able to do a batch capture should anything happen to the footage (or maybe i'm just harbouring traditional editing ideas).

>The ProRes Quicktime component is what actually does the transcoding, and it's the same
>whether you're capturing to ProRes with "capture now," or transcoding with Compressor,
>or rendering out of Final Cut.

I won't put money on that at all. The export functions all have different 'looks'. Export with QT conversion, render and export a QT movie, rendering via Compressor, exporting with QT conversion pre render and post render... Native FCP rendering and exporting via Compressor seems to yield the best results.

Even transcoding to a compressed H.264 at different bit rates via QT pro vs Compressor vs QT conversion yields different results. With QT pro providing marginally better results than Compressor (but I won't trust QT pro for huge conversions or deinterlacing), and QT Conversion yielding the worst result of the three. Can't run differential mattes, as there is just too much white count due to the compression used, and white spots seem to vary. I suspect that QT conversion renders at 8 bits. And of course, nothing beats Compressor at pure speed when you set it out on an Octopus. I'd love to see FCP rip when it finally manages to use all 8 of those cores.

Else, specs for how exactly ProRes does its spatial compression is not available (it's more likely DCT than wavelets).



www.strypesinpost.com

Quote

Very interesting choice! If I had SDI I'd be using that in a heartbeat.

What for? HDV is HDV, and it'll never get better than that. Once it's hit the tape, coming out of the camera or deck via the SDI spigot won't make it look any better than coming out over Firewire.

Quote

You didn't mention your hardware-- an Intel octopus?

Yeah, it's a relatively modest system for doing HD: An 8-proc 2.8 with 8 GB of RAM. My 16 gig upgrade is literally in the mail. I'm just using two striped SATA drives as my framestore right now, because my G-Speed ES is also literally in the mail.

Quote

Then it's no longer HDV, right? It's unrendered ProRes playing realtime? Are you applying effects, titles, color correction, et al?

You know, Andy Mees and I have had a really interest email exchange about that very thing, and we can't figure out precisely how FCP 6 handles this scenario. Is it transcoding HDV to ProRes in real time when it plays back unrendered, or is it just decoding the HDV to uncompressed and playing it back that way? I honestly don't know.

The only test I've been able to imagine to find out is to play out an unrendered ProRes timeline over my Kona card and record it on HDCAM SR, then render it and record it again, then capture them both as uncompressed and pull a difference matte. Since I don't have access to an SR deck, I haven't had a chance to try it. Also, um, it kinda doesn't matter. I can't tell the difference by looking, and that's all that matters in my scenario.

My work is mostly industrial, so I'm not doing any serious visual effects. I do lower thirds, but those come out of After Effects as stills with alpha that get dropped into V2. I believe FCP plays them back at lower quality unless I render, but again, it's not a visually obvious difference. I haven't looked closely to see.

I do all my color correction in Color; HDV in via a Final Cut-generated XML, ProRes 422 HQ out via another XML.

Quote

Is there a white paper somewhere on how the image looks coming from HDV to ProRes or HQ to large projection? Pro's at the post-NAB SMPTE meeting I attended a year ago were cooling off on HDV as "looking like crap" on anything larger than 42".

It's entirely possible. HDV is highly compressed. But it's hardly unique in that. Yes, you can see the compression easily on a freeze-frame, but it's far less obvious in motion. There are particular shooting scenarios in which HDV will fall apart more quickly than most other HD formats, but for stuff like interviews and doco, it's aesthetically fine. I wouldn't want to shoot a car commercial on it, but then again I wouldn't want to pay thousands of dollars in rental fees just to shoot a short film on the weekend for fun.

I highly recommend you check out a short film called "White Red Panic." It's on Vimeo. It was shot on an HV20 with, if I remember correctly, no add-ons at all. Would it hold up for theatrical presentation? Probably not. But the point is that because gear like the HV20 is available, that guy was able to make the movie he wanted. I can't imagine that being a bad thing.

> Is it transcoding HDV to ProRes in real time when it plays back unrendered, or is it just
>decoding the HDV to uncompressed and playing it back that way?

Sounds like a very interesting workflow... Does it do a conform when you hit render? If not, pretty much, this would beat the GOP issues AND save on drive space AND time!



www.strypesinpost.com

That's part of the beauty of it: there is no conform. (Blah blah dumb movie allusion re: spoons.)

For those who are following along at home, the last step in editing an HDV timeline is "conforming." It's annoying as hell that Apple uses this term, since "conforming" isn't the same thing as conforming your edit.

Compressed video ? and virtually all video is compressed these days, in one way or another ? comes in two varieties: Some kinds are compressed as if they were a sequence of still images. Imagine exporting your footage as a TIFF sequence and then JPEG-compressing each one. That's called I-frame compression.

The other kind of compression is called GOP, for group-of-pictures. GOP-compressed footage deals with a bunch of frames at a time. How many exactly depends on the format, but for 60 Hz HDV, it's 15 frames. The first frame in a GOP is compressed all by itself, then the second frame is described in terms of what's different from the previous frame. Frames inside a GOP literally cannot be decoded without first decoding the previous frames in that GOP.

Final Cut doesn't require you to make your cuts only on GOP boundaries. For obvious reasons; can you imagine trying to edit if you can only make a cut on the half-second? The consequence, though, is that an HDV timeline ends up being a random mishmash of frames taking from the insides of GOPs that weren't entirely included in the edit.

So before you can export from Final Cut in HDV format, whether you're going out as a Quicktime or to HDV tape, the system has to go through and decode all the footage, then re-encode it in a regular GOP structure. Final Cut calls this "conforming," and it sucks for two reasons: First, it takes forever. And second, your footage takes a big compression hit. This compression hit is inevitable if you're laying back to HDV tape, but who does that? HDV is an acquisition format, not a delivery format.

ProRes, by contrast, is an I-frame format. Every frame of ProRes footage is compressed as a standalone image. So the system doesn't care if you take 40 frames of this clip, then cut to 110 frames of that clip, then 29 frames of some other clip. It's all the same.

So when you cut HDV footage (or XDCAM, or in fact any kind of footage at all) into a ProRes timeline, Final Cut never has to "conform." What happens technically is that Final Cut decodes the source footage from whatever format ? HDV, in this case ? and then encodes it as ProRes. But ProRes was created specifically to be really fast to encode and decode, especially on multi-processor systems.

Of course, just like HDV is only an acquisition format, ProRes is really only an intermediate format. Nobody delivers in ProRes-encoded Quicktimes, although I guess technically they could. Odds are you're gonna deliver on HDCAM or something like that. But if you live in the HDV world while you're shooting, then live in the ProRes world while you're in post, you can live in the uncompressed world or the HDCAM world or whatever when it's time to deliver.

(Sorry for being long-winded.)

Ahem ...

You said: " What for? HDV is HDV, and it'll never get better than that. Once it's hit the tape, coming out of the camera or deck via the SDI spigot won't make it look any better than coming out over Firewire."

You know, I believe that the SDI output from the Canon XLH1 is NOT HDV. It doesn't hit the tape and the results that I have seen are better in colorspace and sharpness when viewed on a 20 foot projection screen - which I have witnessed. What I saw had been recorded directly into a Mac from the SDI port.

Harry

Harry Bromley-Davenport.

Oh, right, duh. Sorry, it was really early in the morning. I thought you were talking about capturing from tape via SDI. Once your footage has hit the tape, it'll never get better than what you get over Firewire. But if you're operating on a tether and taking the output from the camera straight into a computer or recorder, it does in fact bypass the HDV encoding bits and gives you uncompressed 4:2:2.

I watch Pro-Rezzed HDV every day on a 62 inch screen and it looks excellent.

I also saw a side by side test of HDV off a Canon XL H1 and DVCProHD off the Panasonic HVX200 projected onto about a 15 foot by 8 ft screen and I personally found (subjectively) the Canon image at least as good as the HDV.

So I don't agree that HDV sucks when shown at sizes over 42".

It's when you get up to the 20 ft wide screen that you really can see a difference.

I think that there's a lot of invalid macho tech snobbery about HDV because of its lowly origins as "semi-pro".

I've shot and sold three small features internationally got through USA and foreign QC just fine without any adverse comments relating to the films' humble HDV origins. I think I could have told them it was shot on a Viper and they would have nodded their heads sagely and muttered favorably about 444 colorspace being the only way to go.

Harry.

Harry Bromley-Davenport.

>it does in fact bypass the HDV encoding bits and gives you uncompressed 4:2:2.

Same goes with the EX1.

>That's part of the beauty of it: there is no conform.

Thanks for the confirmation, Jeff. Yea, I know the GOP bit, but everyone here has to learn... Just wanted to make sure if hitting render on HDV footage edited into a pro res sequence causes FCP to "conform" those cut points (since you're 15 times more likely to cut on a B or P frame rather than an I frame, I was wondering if FCP will reorganize the GOP before the transcoding). Good that it doesn't...



www.strypesinpost.com

I've also seen a feature shot on HDV and it looked great. But whenever I've used it myself I've had GOP problems, so I personally feel it's really fragile and annoying.

Hmm... Jude, did you happen to try dropping the HDV sequence into a ProRes timeline? The not conforming part may actually solve a lot of render issues...



www.strypesinpost.com

Actually, I think there is a bug with HDV in 6.0.1. I did a lot of tests with Ken Stone because he has a workflow for exporting HDV as ProRes for DVD which works really nicely, except in 6.0.1. I'm on 6.0.4 now and the problem has gone away.

But my HDV hiccups - losing GOPS, TC breaks and so on, were all native, and it's made me wary. It's fine to drop 15 frames in GVs (I suppose), but what about in the middle of a sporting event or an important interview? Those moments get junked up and you're up that proverbial creek.

At least with drama you know you'll probably have several more options to choose from, I guess. It's the 'happening now, don't miss it' stuff that worries me.

I'm sorry you've had bad experiences with the dreaded HDV. Nevertheless, as I said before, I've shot three features now on it. The first we immediately transcoded from HDV to DVCProHD because there was no other way FCP handling Canon's proprietory "24f", and two additional pictures using ProRez.

I've obviously been lucky. Even using cameras in 10 degrees F of cold produced no problems other than short (and I mean SHORT) battery life. Also occasional dead pixels.

Oh, and sometimes (intermittently) a sudden light level change will freak the camera out and it will produce blocky stuff for a few frames - but that's on the firewire output to camera hard drive only and doesn't go to tape which we use as backup.

Tried to get it fixed but they can't figure it out. Ah well, I guess these things aren't exactly built to military spec.

The only other problems I've had are to do with losing things with my FCP logging non-system.

Best

Harry

Harry Bromley-Davenport.

Posted by: strypes (IP Logged)
Date: August 07, 2008 07:43AM

Hmm... Jude, did you happen to try dropping the HDV sequence into a ProRes timeline? The not conforming part may actually solve a lot of render issues...

That's exactly what I do and it woks just fine.


H.

Harry Bromley-Davenport.

And why on earth are we all awake so early?

H.

Harry Bromley-Davenport.

What essentially has me avoiding HDV (especially in post), in general, are those render error bits you brought up... If the editor was unaware and pumped that stuff out for broadcast... bam. Though the checks in place usually prevent that, it will still be a nasty surprise...

Rearranging GOPs never used to be part of NLE workflows. Editing codecs have always been spatial until the rise of HDV. Calculations get more complicated and render errors are more likely to occur as a result. I should run some tests on Jeff's workflow when i get my hands on some HDV footage...



www.strypesinpost.com

>>And why on earth are we all awake so early? <<

It's 11pm here in Back 'O 'Bourke

But my HDV hiccups - losing GOPS, TC breaks and so on, were all native, and it's made me wary. It's fine to drop 15 frames in GVs (I suppose), but what about in the middle of a sporting event or an important interview? Those moments get junked up and you're up that proverbial creek.

At least with drama you know you'll probably have several more options to choose from, I guess. It's the 'happening now, don't miss it' stuff that worries me.

You quite possibly have a camera head misalignment issue, or dirty heads, or something.

I repeat: three features, using two cameras on two of them. At a rough guess, a total of maybe over 120 hours of recording - and not a single dropout or GOP issue.

Sorry to sound so holy and know-it-all. But it's an efficient format for me to shoot pictures costing under a million $s and I tend to get defensive about it because of the somwhat snooty and annoying term "pro-sumer" which is tossed around when referring to HDV.

Happy Gopping.

H.

Harry Bromley-Davenport.

Sorry to piggy back into the conversation, but I work mostly with HDV, the finished product ends up on standard definition DVD.

But if I wanted to make a master to HDV tape am I going to take a hit?
What would you guys recommend as a proper workflow going back to an HDV tape on a JVC BR-HD50?

Thanks for any and all input.

Haha.. It's 11pm here in sunny singapore.. err... not so sunny now... where the hell are you, harry?



www.strypesinpost.com

Interesting thread.

Jeff

Thanks for the Vimeo White/Red/Panic short film link.





Excellent stuff! and shot on the mighty HV20!

Essential viewing for the HDV naysayers.

>But if I wanted to make a master to HDV tape am I going to take a hit?

Well, if you render, you take a hit (lose a generation). That's for almost any codec (except for uncompressed). With HDV, when it has to reorganize your GOP structure (eg. when you make a cut), technically, you also take a hit, as B frames now becomes I frames, etc... and the entire GOP structure is rearranged.

But generally, you save conforming time if footage has already been conformed to the GOP structure in an HDV sequence.



www.strypesinpost.com

Sorry Harry, I'm not trying to run it down. If it works for you, or anyone, fantastic. It's SO much more accessible for film making than traditional workflows.

My problems happened on three different cameras, at two different events. None of the losses were critical. All three were hire cameras, one from a place where I wouldn't be surprised to get dirty heads, one from a place where I would be surprised and one was almost brand spanking new - only one session before mine, by the owner.

Wish I had some stuff to show you - one of the worst ones we just got blue for about 15 frames in the middle of a shot, for no reason whatsoever. No button off, no battery removal, no nothing. Scared me.

And then I had the problems in 6.0.1 .. actually, I think maybe some of the tests are still online - let me see - here's one - just three shots back to back. It's not an effect I've added!

HDV conformed Prores in 6.0.1

And here's another one from another machine running 6.0.1 - again just three shots back to back. The red flash comes out of nowhere. It is not added or intended. It's a shot from elsewhere on the timeline. Another example