Suggestions please regarding oddball workflow

I'm going to have to develop a pretty unusual workflow over the next few weeks, and I'd really appreciate some suggestions or feedback about it.

My client has thousands of hours of archival footage in MPEG-4 format sitting on a server. Each MPEG-4 file is one to three hours of footage. The footage doesn't exist anywhere else, so there are no videotapes to go back to. I have to work with what we have.

Client asks me to cut together a show using this archival footage, from a loose script. Script calls for three or four seconds from this MPEG-4 file, ten from that one, six from another one, and so on. Maybe forty or fifty distinct archive files per show, for anywhere from fifty to a hundred hours of original source footage.

Just importing the MPEG-4 footage straight into Final Cut works fine, surprisingly enough. I load them all into a bin, open them in the viewer, set ins and outs and insert-edit into my timeline. (My timeline is ProRes 422.)

Because this archival footage is all MPEG-4 and heavily compressed, a lot of work has to be done to deblock and just generally make the show look like it was made by grown-ups. So no problem, I'll just use Pro Import AE to take my timeline into After Effects when I'm done editing. Except After Effects can't use the MPEG-4 files found in my client's archive! They simply don't work there. Which is hardly surprising, but there it is.

Obviously I have the option of transcoding all the source material for a given show into ProRes before I start editing. But what would be even better is if I could transcode just my edited footage plus handles, then relink my timeline clips to the new media files before going to After Effects. Sort of like using the MPEG-4 as you would an offline, then using the transcoded footage as you would the telecine'd selects. What I'm missing is the neg cutter.

I tried using the Media Manager to trim my source media by checking the "delete unused media" and use handles" boxes, but FCP just ignores those settings when it comes to this source footage. I guess it's an artifact of how it's encoded.

The only other option I can think of is to take an XML of my edited timeline out, and somehow process it to turn it into a Compressor batch file. At least in theory, I think it should be possible to add handles that way, then send the batch through Compressor to have it transcode just the selects from my timeline. Then I'd need a roundtrip XML file to go back to Final Cut, recreating my original timeline with links to the new, transcoded media files.

The problem there is that I can't seem to find such a tool in Google, and I'm not enough of a gearhead to create one myself.

Has anybody out there run into a situation like this before? Does anyone have any suggestions or pointers? Or even flat-out criticisms of my intelligence or overall worth as a human being?

> Each MPEG-4 file is one to three hours of footage.

Two problems right there. Firstly, MPEG-4 is not an editing-friendly format. Secondly, a three-hour clip would likely have been very awkward to work with in FCP.

If you hadn't already edited, I would have suggested breaking each clip into 20-minute chunks. Or, if your client were certain s/he didn't want more than, say, five minutes out of each clip, I would have used MPEG Streamclip to hit In-Out points on the MPEG-4 source files, then exported them into a better editing format, such as DV NTSC or Uncompressed SD.

What about just exporting your timeline at its current settings? Not being an After Effects user, I'm only assuming that you're planning on processing the footage in AE to spiff up the look, since your source couldn't get any better from the original MPEG-4s. So what about just exporting a movie file of your edited timeline? Do you really need the handles?

Never use web-delivery formats (MPEG-1, MPEG-4, Sorenson 3, etc.) to edit with.


www.derekmok.com

Thanks very much for the reply. And you're totally preaching to the choir about GOP formats for editing. It's a nightmare. But it's my nightmare, because literally nothing can be done about it now. That's what I was given to work with.

Batch-processing all the footage I'll use for a given show is certainly an option, and right now it's the most attractive one. The only reason I don't love it is because my client has a habit of saying, in a supervised session, "Oh, can we just insert such-n-such from such-n-such piece of archival footage which you don't have on your framestore yet?" Asking the client to wait while that piece of media is transcoded is certainly something I can do; I just want to be sure it's the best answer first.

The thing about handles isn't an absolute either, frankly. Maybe I'm a lousy finishing editor, but I often find myself wanting to make tiny changes to my edits after the graphics have been put on or the color correction/effects added. I'm talking about rolling a frame, or slipping two frames, tiny stuff like that just to perfect the finished work. So having handles inside my finishing session would be really helpful. Not essential; just really helpful.

After pondering this situation for a few weeks and trying different things out, I've come to the tentative conclusion that there is no perfect answer, just a small set of answers that are sub-optimal in different ways. That's why I wanted to ask, to see whether unbiased outsiders would agree.

Thanks again.

Media Manager like files with timecode. I suspect that the MPEG-4's don't have it. Chose a file in the browser and add a reel number to the metadata. (Modify Timecode). This forces a timecode onto the Quicktime file. It should then trim when you media manage the sequence.

This behavior seems to be purposeful so that you don't throw out media that you don't have reference to recapture. Not an issue in your case.


-Vance

I agree. The footage probably doesn;t have reel numbers, which is why Media Manager is ignoring it.

Here's Nick Meyer's FAQ on this

From Nick Meyers

Sometimes you go to ?Delete Unused? in the Media Manager and the Green Bar indicating your new Media size doesn?t get any shorter. Or as short as you?d like it, anyway. Here's some reasons for this, and suggestions for how to deal with it:


NO TIMECODE TRACK

* Media Manager WILL NOT trim files that don?t have a proper timecode track.
* Sound files such as AIFFs won't trim, but will be moved or copied in their entirety.
* Also if you get Motion Graphics from a designer, or that you make your self, they probably wont have a proper timecode track, and wont be trimmed either.
* Same goes for exports from FCP that you bring back in.


The thinking here is that if there is no proper timecode, the media didn?t come from a tape, and therefore may be THE ONLY EXISTING COPY IN THE UNIVERSE!!! So Media Manager doesn?t want to take responsibility for deleting that media. This can catch you out. In some instances you could have very large files without proper timecode. The good news is that it's easy to add TC to a video clip in FCP. All you have to do is give the clip a reel number, and you can do that in the FCP Browser. This will activate a proper timecode track in the file, and NOW it will be trimmed by Media Manager.


TECHNIQUE:

* To find all clips in your project without reel numbers, do a search in your browser for: "Type" Contains: "Clip". Add another filter of "Tracks" Contain: "V" to only get video clips

* Sort the 'Find' results by 'Reel'.

* Add an obvious Reel number to one clip, like "NO REEL"

* Then select all the other clips with no reel number

* Right or control click in the reel column, and you should get a list of all reel numbers used in the project.(If you can select and click on the clip you just modified, its new Reel number will be at the top of the list)

* Select the new NO REEL reel, and it will be applied to all selected clips


If there are clips in the sequence that aren't in the browser, select the sequence in the Browser, then go to Tools Menu > Create Master Clips. In general, there?s no need to trim audio files, as they are relatively small.


A few things to (un)check in the Media Manager before trimming:

* DESELECT ?Include Master Clips Outside Selection?
* DESELECT ?Include Affiliate Clips Outside Selection?
* DESELECT "Include Non-active Multiclip Angles"


Even after all this, FCP may be hanging onto some clips due to corrupt Master / Affiliate relationships. (This is fairly old behaviour, going back to FCP version 4 and 4.5. It isn't likely you'll have this in FCP version 5 or 6.) There are two ways to fix this:

* 1. Copy the sequence you are attempting to manage into a new temporary project. This will ?wrench? the sequence away from any master clips in the original project. (This doesn't have to be a permanent project. it only has to live long enough to perform the Media Management.)

* 2. Control or Right click on the sequence you are attempting to manage and choose ?Make Sequence Clips Independent?. NOTE: If you do this, you will lose the ability to Matchframe back to a bin! If you are not ready to put the project to bed, this might not be a good idea, so I recommend the "copy into a new project" method

but by cutting the MPEGs into a PrRes Sequnce, and rendering, you are creating a ProRes versions of all your segs.

as well as creating a ProRes version of your show.

you could pretty easily render all, and export.
then use an EDL if you need to take it somewhere for grading.

do you think you'll get a lot better quality by not doing the convert in FCP?


nick

> as well as creating a ProRes version of your show.
> you could pretty easily render all, and export.
> then use an EDL if you need to take it somewhere for grading.

That was my thought as well. So the only fly in the ointment is if he wants to keep trimming as he goes. Frankly, though, once your picture is locked, you should tie your hands behind your back and not keep tweaking. If you're gonna keep tweaking, don't lock picture and don't online yet. If you can stick to this, then just exporting a full-quality movie file sounds like your best best, Mucking with Media Manager with untimecoded clips -- especially three-hour-long ones -- sounds dubious to me.


www.derekmok.com

out of curiosity i just tried adding a reel# to an mpeg clip,
and MM still refused to trim it.

i'd say work in FCP.
if you are having to scale the clips up to fit the timeline,
make sure you have your sequence settings all set to the highest quality motion scaling
that;s A/V Settings > Sequence Presets > open a preset and go to Video Processing for all future sequences
and Sequence Settings > Video Processing on all existing sequences

If you really hate FCP's scaling,
then you could set your sequence frame size to match the mpeg,
and do all the scaling elsewhere.


cheers,
nick

Bummer. Nick, since we are upgrading the FAQ right now, should I add that this won't work for MPEG sources?

i guess we should get to the bottom of it fist.

it's probably that some sorts of files just wont take a TC track,
but im just guessing,
and i wouldn't want to go on record saying that.

probably need someone like Andreas or Tom to conform that.

I'm sorry to be terse, but I'm still finishing up a very, very long-running session tonight.

Something I neglected to call out emphatically enough, I think, is that these are not QuickTime movies, but rather MPEG-4 movies ? .mp4 files as opposed to .movs. I think that might have something to do with the fact that they're not taking a timecode track.

You folks have been amazingly helpful so far. I don't really have a perfect solution yet, but your input has been fantastic. I'm going to finish up here, take Sunday off and then ponder this a little more early next week. I'll post a detailed update as soon as I can.

Thanks for all the help.

you were pretty clear all along, Jeff.

we're just going off on a tangent.

nick

Which almost never happens here.


pffft.

[Asking the client to wait while that piece of media is transcoded is certainly something I can do; I just want to be sure it's the best answer first. ]

That's good-- a patient client!

Definitely get your selects into Prores 422. I like the philosophy of getting all selected footage into the same codec even if it involves extra steps. Now... what are they?

[ a lot of work has to be done to deblock and just generally make the show look like it was made by grown-ups.]

So MM transcoding may not be the answer. Maybe the footage can be ripped a la DVD rippers, maybe it can be played out through a fancy card, right into ProRes (I'm thinking of the AJA IO HD box actually, but I have no seat time using this item).

Could you get a better result if you played the mp4 footage out digital to analog to a capture deck using perhaps a DVCPro codec? (Could the MXO2 box do this? I dunno.) But that gives you-

1) a fresh original of selects without entire source;
2) easily captured into project-native codec;
3) timecode and reel number needed for management.

I don't envy you your task but like many here I'm interested in how it goes. Hope you'll report back.

- Loren
Today's FCP keytip:

Toggle your Timeline Filters bar with Option T !

Final Cut Studio 2 KeyGuide? Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

Again, this is all excellent stuff. I really appreciate the ideas.

Lemme tell you a little bit about how yesterday's marathon seventeen-hour session went. Not to commiserate ? okay, maybe a little ? but also to see if any more brilliant suggestions emerge after hearing about the process.

First, this is a corporate client, not an industry client. So the ground rules are a little different. Basically nobody has ever heard of pre-production, and having twenty-four hours from this-is-the-first-I've-heard-of-this to here-is-your-finished-show isn't uncommon.

Yesterday I started with a bin full of about 50 of those MPEG-4 movie files. They'd been pulled for me from the archive server by an assistant, who'd given me a paper edit of ins-and-outs. Blah blah assembly edit, et cetera.

Client came in around mid-afternoon, and we spent some time cutting the show. Extend this shot, trim this one, reorder these. Entirely standard stuff.

After dinner, we settled down to cut the third act. It was one of those "let's raze it to the ground and start over" situations. Client recalled additional footage in the archive that hadn't been included on the original list, so I had to delve into the archive-management system to get additional MPEG-4s. This happened a couple of times. We were on deadline, so at that moment I would have been more interested in the fastest way to get the footage into my timeline above any other choices.

Finally we got the show to a good place and agreed to lock picture so I could move on to doing the graphics ? some intertitles and a few light motion-graphics shots in act three ? and mixing the sound.

As I was working my way through the graphics, around 11:30, I got a call from the client. "Can we extend that one intertitle by about 15 frames?" That required a little bit of slipping and rolling of footage down the timeline; nothing difficult, but something that would have been a real pain if I'd been in a workflow stage where I'd committed my edits and thrown away my heads and tails.

Finished the graphics, did the most halfassed color-correction pass you've ever seen in your entire life and mixed the sound. Home by 2:30 a.m.

This isn't all that unusual a workflow for me, unfortunately. It's getting better all the time, as the client and I educate each other about how this stuff is supposed to be done. But right now, it's really messy. That's why I'm leaning toward using After Effects as a poor-man's finishing system with Pro Import AE; it seems like it'd give me a huge degree of flexibility for doing things like extending that intertitle by 15 frames when the call comes and I'm in the middle of final color correction. That flexibility would make the client happy, and the client's money makes me happy, so there it is.

Now, lest I sound like I'm just complaining about my client over the Internet here ? hi, client ? the good part about all this is that the sky's really the limit, workflow-wise. There are some constraints, like the fact that the archival footage has no timecode on it. (The archive was constructed and is maintained by a computer guy who doesn't know from TV.) But for the most part, I'm free to explore all the various options to find a workflow that makes this client happy and me sane and well-rested.

So thanks again, very much, for all your suggestions. Monday afternoon I'm going to try some things based on this discussion. First, I'm going to try using Automator to change these MPEG-4 files to QuickTime movies without actually transcoding any of the footage, to see if I can then add a timecode track that will make Media Manager happy. Failing that, I'm going to do a test where I just batch-convert all 50-odd hours of archival footage from MPEG-4 to ProRes through Compressor, to see first how long it takes, and second how bad the storage requirements really are. If one of those turns out to be practical, I'll just unilaterally declare it "part of the workflow" and have done.

Thanks, everybody.

cool.
love to hear how it works out

50hrs of Standard Def ProRes HQ = 1.3 TB

according to this widget:
[www.digital-heaven.co.uk]



nick

Hi all,

Nick asked me to give some comment - here a quick one without any check.

My knowledge is that MM works with "QT files" only - not with "QT compatible files", that's a big difference!
In my opinion the (missing) timecode doesn't seem to be the problem. But even if there is a timecode a missing "Reel" can cause a problem. As always -- I could be wrong.
So you might put just a simple "QT wrapper" around those "QT compatible files" and make them "QT files" as referenced movies. In theory you can take a simple Apple Script or Automator Script to save those files as "QT files" in a referenced movie without converting them. This won't generate not too much additional work time and disk space, but you have to keep your file management very clean.

But as said I haven't tried yet (since I don't like MM and never use it) - I'll do on Monday or Tuesday.

Regards
Andreas

Some workflow tools for FCP [www.spherico.com]
TitleExchange -- juggle titles within FCS, FCPX and many other apps.
[www.spherico.com]

I promised to report back, so I am.

As everybody already knew but I just learned, Final Cut doesn't love MPEG-4 files. It'll work with them, more or less, but it doesn't love them. That's the bad news.

The good news is that a QuickTime reference movie that refers to an MPEG-4 file apparently works perfectly. I was kinda surprised to find that to be the case, but it was. It works like a charm.

So now my challenge is to find a way to batch-create a bunch of QuickTime reference movies. Does anybody have any suggestions on that front?

Good to hear that it works now - that's what I meant with those "QT files" versus "QT compatible files".
So I don't have to try - saves me some time.
What you can try is to "Batch Export" your files from FCP. Or to write a simple AppleScript or Automator Action to that.

Regards
Andreas

Some workflow tools for FCP [www.spherico.com]
TitleExchange -- juggle titles within FCS, FCPX and many other apps.
[www.spherico.com]

reference movie?

how did you make those?
are you sure you haven't just exported your movie, transcoding it to another codec on the way?

i cant make a ref movie of an mpg from FCP

and when i use QuickTime, FCP's Media Manager wont trim it.



nick

Yeah, plain old reference movies. I opened one of the MPEG-4s and did a "save as," checking the "make reference movie" radio button. The resulting .mov file went into Final Cut and worked exactly as everyone predicted. Giving it a reel number provoked the standard "you're changing the original file on disk" alert, and once that was done, Media Manager trimmed and recompressed my timeline clips to ProRes. Worked perfectly.

So that's the solution. Before cutting any of these MPEG-4s, I'll just make reference movies of them and tell Final Cut to refer to those instead. Easy.

What's trickier is that I've still not found a way to automate that process. I spent about an hour messing around with Automator actions and AppleScript this evening, but failed to find any good way to tell QuickTime Player to make a reference movie. I tried using Automator's "record" functionality, but it was unreliable. The workflow I created wouldn't always play back the recorded steps correctly, for reasons I'm not sufficiently educated about to understand yet.

For the next show, it won't be all that onerous just to include "make a reference movie in QuickTime Player before importing into FCP" a part of my workflow. But I'd love to find an easy fix for the 50-odd MPEG-4s already on my framestore. One that doesn't involve me putting on a podcast or something and just muscling through it. I'm lazy. ;-)

Any suggestions on that front?

> For the next show, it won't be all that onerous just to include "make a reference movie in
> QuickTime Player before importing into FCP" a part of my workflow.

For the next show, I'd still say you should be going out of the MPEG-4 format. The process you described above only results in a QuickTime movie with an MPEG-4 codec (or H.264 -- whatever your original MPEG-4s were). I'd say get out of those non-editing codecs altogether.


www.derekmok.com

I agree totally, Derek. But it's just not practical. I did some poking around this morning, and these guys have an archive going back to 2005 consisting of more than 20,000 hours of footage, all of it stored in MPEG-4 format. The only way out of it is to transcode whole hour- and multi-hour-long segments before cutting it, and that would be prohibitive in this environment.

I'm not defending it. I think it's bull, frankly. But it's what's there, and I'm getting paid to find a way to live with it.