Batch Converting DNxHD to Prores

Hi everyone.

this is a follow-on from my previous post about porting a feature from Avid to FCP,
i thought this more specific subject was better on in a new thread.

turns out the Avid project is using media that was synced during transcode.
so, on transcode from Prores 4444 to DNxHD, any audio with matching TC was found, and married; as in baked in with the picture.

what this means is that the number of audio tracks varies from file to file, depending on the original audio source.
so the simplest approach for me is to get a ProRes copy of those files WITH the corresponding number of audio tracks.

what would be the best tool to do this conversion?

Compressor, of course has the "Audio Pass-Through" option,
which is good, BUT i have found compressor tends to fall over on really big jobs,
and not so big jobs, too: i would be wary of a job with more than 10 files, (based on my experience)

working in avid is still not entirely out of the question, but unlikely,
and i don't want to paint myself into a corner there, so trasncoding will start ASAP.


thanks for any advice,
nick

Wait, so these files were ProRes4444 to begin with? (Alexa?) What did they use to convert/sync them to DNxHD and was that a project based app? The great thing about a well managed RedCineX or Resolve sync and conversion is that you can save the projects and re-render with new output settings without too much bother. Assuming there was a well maintained project to begin with.

However:

I've done Compressor jobs of large projects. Not to DNxHD though.
Setting up a Qmaster simple cluster on a MacPro with 8+ cores can make a real difference as long as you have the temp folder on a drive with TONS of space.

For ease of render queue management its always better to submit the files in smaller Compressor projects. I've broken it up by reel, camera, shoot day or whatever.
Save the Compressor projects and be good about naming the batch when you submit it.

To save yourself from hassle get the Digital Rebellion Compressor Repair utility to reset Compressor if it acts up. I do this once every 3 or 4 months. You get familiar with the Qmaster settings by rebuilding them after each reset and maybe you can dig around and learn what all the other settings are.

Leaving a whole bunch of batches overnight and being rewarded by seeing the last progress bar disappearing just as you hit the seat the next morning is a satisfying feeling.

One gotcha I have run into was while fixing a doc project where everything had been captured as full length HDV tapes/files and they wanted matching ProResHQ files for an edit about a week away from lock. I have learned that if you set up a simple cluster with file segmenting, the file will get broken down into chunks and rendered. Then, as a last step, recombined into the final file. This is trivial on H.264 or most other normal size files. However, for 45-60minute ProResHQ files that last step took a VERY long time and it turned out to be faster to just render the files unsegmented but at least be doing them 8 at a time.

ak
Sleeplings, AWAKE!

I have DR compressor repair utility, and it does help but...my experience with compressor and Qmaster has been WAY way more bad then good. I'm sure its my machine, but I haven't for years now been able to make it work consistently or reliably.

Nick,

My concern is that AFAIK, Media Composer doesn't keep DNxHD MXF audio and video in a single file. So for one clip with three audio tracks, there's four individual MXF files at the Finder level. So that would mean transcoding all that DNxHD, then bringing in the audio and re-syncing! Unless of course there's some alternate AMA workflow going on, and the DNxHD was wrapped in QTs. In that case you could proceed using Compressor, making sure it's bringing the reel info along for the (long) ride. Oh, and Compressor won't grok Avid MXF files. Maybe if you buy MXF4mac? Squeeze will (slowly).

Andrew Kines Wrote:
-------------------------------------------------------
> Wait, so these files were ProRes4444 to begin
> with? (Alexa?) What did they use to convert/sync
> them to DNxHD and was that a project based app?
> The great thing about a well managed RedCineX or
> Resolve sync and conversion is that you can save
> the projects and re-render with new output
> settings without too much bother. Assuming there
> was a well maintained project to begin with.
>


That would be another way to go -- re-render from Resolve (or ColorFront or whatever, RedCine only works with Red files) to a lighter ProRes using the original dailies project(s) from the DIT. Resolve renders pretty fast in my experience, on a beefy system.

Or just batch export PR QTs from the MC bins (sloooow).

Good luck. Sounds like a pretty massive undertaking.

- Justin Barham -

thanks, Andrew.

sorry, i left out a few details

camera originals are Prores 4444

transcoded to DNxHD36 MXFs using Colorfront Express Dailies.

yes, we could re-transcode the 4444 to ProResLT, it is an option.
i wil talk w the data wrangler about it,
it just seems there would be room for error...?

hard to say right now, i only just got the media files today,
and will hopefully be able to examine them tomorrow.


i have never had any luck with multi-threadeding compressor on my octo-core Mac-Pro.
no matter how i set it up it is the same.
i was advised that some machines "just don't work" and that's all there is to it.
not sure about that, but eh.
i may well put it though on a few machines: mine, and some that really work!

METADATA
this is a biggie:
the Metadata (TC & Reel#) is stored in a separate ALE file.
how will that work?
i can live with losing reel#, but if i have to replace TC that'd be a real drag.


nick

sad news is the Data Wrangler deleted the project files a couple of weeks into the edit.
DOP isn't fussed about the look so much, so long as it's close,
i am worried about constancy of audio tracks.

Quote
Justin
So for one clip with three audio tracks, there's four individual MXF files at the Finder level

looking at it now, that's exactly the situation
1st clip i see in day 02 has:

a FOLDER named A902C001_121009_R2DU. deep inside this there is an XML file

then
A902C001_121009_R2DU_A01.mxf
A902C001_121009_R2DU_A02.mxf
A902C001_121009_R2DU_A03.mxf
A902C001_121009_R2DU_A04.mxf
A902C001_121009_R2DU.aaf (a very small file)
A902C001_121009_R2DU.mxf (the video file)

no way to view any of them: they wont open in QT, compressor, or even VLC!



Resolve does a similar thing to Colofront, in that it can sync the audio via TC.
i have access to that, on fast systems, at a very helpful facility.

need some time to test it all.

thanks for the ideas,

nick

I'm not sure I'm getting all the details, but it sounds like you'd be much better off in DaVinci Resolve v9 than compressor. You would at least be able to synch audio, do conversions, maintain file names, and render out to individual clips.

Are you up and running enough in DaVinci? Or if not, then bring someone in for a hour or so to show you how to wrangle clips, audio and set up the delivery page.



hth,
Clay

UPDATE

Avid installed, projects opened.
exported AAF from an edited timeline,
AutoDuck into FCP, creating new media.

nice, simple.

chosen import sequence settings didn't match, but now i have made a new sequence preset, so next time will be even smoother.
Auto Duck has re-wrapped the Vid files, and made self contained copies of the Audio files, all in QT wrappers

re-wrapped files now open in QT, compressor

QUESTION
with this smallish 12 minute sequence, the DNxHD is well behaved.
can i keep on using it?

i am now Media Managing and recompressing to ProResLT
will take more than an hour for 1hr 12 minutes of rushes.
let's say real time for now.

(jump back down to working with one 4 minute clip)

compressor is faster: no audio files to create, (hard not to create them using MM)
multi-trheaded compressor will be faster, and i can avoid creating yet another copy of the audio



WEIRD
using MM, i noticed some weirdness with the resulting file names.
all have -v on the end, and no files extension
R-Name addresses both, and files now appear to be well behaved ProRes LT QuickTime files

compressor creates a simple .mov


PROBLEM
in FCP, none of the clips are linked!
does this reflect a state in avid i can fix easily?



ANOTHER POSSIBLE PROBLEM
switching off the Auto DUck option to "use existing media" (really means "create re-wrapped reference files" gives me an offline sequence in FCP.
that is expected, BUT the files have no "memory" of their original file path,
they don't want to connect to "A007C003_121017_R2DU"
they want to connect to "1/40/1 CU to Ms window James & Zoe"

so if i AAF out of Avid Sc1 cut1, then Sc1 cut2,
i need to create TWO SETS of the re-wrapped reference files

any way around this?
seems very wasteful


eventful evening,
enough for now!



nick

thanks, Clay.

i can get on a fast Davinci in at work.
and have the option of running it all weekend, which would churn though most of the rushes.

i have to run some tests next couple of days, and see if it's the right way to go.

current thinking is Avid AAF > AutoDuck > FCP.
this creates QT wrapped reference files.
they can then be batched though compressor.


nick

>WEIRD

>using MM, i noticed some weirdness with the resulting file names.
>all have -v on the end, and no files extension
>R-Name addresses both, and files now appear to be well behaved ProRes LT QuickTime files

>compressor creates a simple .mov


>PROBLEM
>in FCP, none of the clips are linked!
>does this reflect a state in avid i can fix easily?

Yes. With Automatic Duck, you do have the issue of it not creating files with proper extensions. I believe this is an FCP "bug". Yes, do media manage/transcode the files to ProRes if you want FCP to be stable.


Now, since they are ProRes 4444 files, you can download a trial of Adobe Premiere Pro. Adobe Premiere Pro can import AAF files that are linked to QTs. I've done this quite a few times to send Avid projects to FCP projects (because the house works in FCP and wants FCP prjs). I've never worked with dual system sound with this workflow, though.

To get the picture cut to FCP via PPro, it's darn simple:

- re-connect back to your source ProRes4444 QT rushes via AMA.
- export AAF files
- import AAF files into PPro
- export XML to FCP

Done. No transcoding. No creating mixed media.

If you want to transcode, you can transcode it in Compressor and re-link the edit once you have ported it over to FCP. The rushes will then be able to preserve linkage back to it's source files, so quality will not be compromised.

I'm not sure how the audio will work, since the audio is probably batch imported into Avid to preserve the timecode. If the audio is AMA-ed, it should follow the same workflow as the one I mentioned earlier.



www.strypesinpost.com

Nick Meyers Wrote:
-------------------------------------------------------
>
> a FOLDER named A902C001_121009_R2DU. deep inside
> this there is an XML file
>
> then
> A902C001_121009_R2DU_A01.mxf
> A902C001_121009_R2DU_A02.mxf
> A902C001_121009_R2DU_A03.mxf
> A902C001_121009_R2DU_A04.mxf
> A902C001_121009_R2DU.aaf (a very small
> file)
> A902C001_121009_R2DU.mxf (the video file)
>
> no way to view any of them: they wont open in QT,
> compressor, or even VLC!

Yeah that's the way Avid does it. That AAF is the clip basically. So in Avid you import that after your media essence files are copied over and it just sees one clip with four audio tracks.

>
> Resolve does a similar thing to Colofront, in that
> it can sync the audio via TC.
> i have access to that, on fast systems, at a very
> helpful facility.

That might be the way to go. There are some tuts around for Resolve 9 for dailies. You'll auto sync in there by TC and add a LUT for Alexa log.

Get a few days ready and then work on them while the rest are churning away in the BG.

> need some time to test it all.
>
> thanks for the ideas,
>

No worries!

- Justin Barham -

ok, so due to the sound being in separate files, i realise audio is NOT an issue.

i can use any old thing to transcode, anything that is Fast and that works.


also, i now realise that there is a 2.35 mask that has been added in the transcode,
i prefer to add my own (makes for simpler re-frames etc)

so i may be going back to the 4444,
setting a look, and doing a batch transcode over the weekend.

i think (hope) i can work with the re-wrapped DNxHD in FCP until i have my new ProRes files.

has anyone worked with the AutoDuck re-wraps?
they seem stable in my veery small test.


Strypes, i might check out the Premiere options, but forgive me, i am a bit dim, what is the advantage over AutoDuck?


nick