Back Up (What's your Work Flow)

A few weeks back there was a guy on this forum trying to retreive some data for a couple of guys off of some HD's that had "crashed" and they hadn't backed up their data.

First, if that guy reads this post, let us know how it worked out.

Second, after reading some of the comments on the post it really got me wondering. What's your work flow for back-up? I thought I was doing a good job backing up (using Media Manager and such) my projects as I completed them. But there are some guys who are seemingly backing up on a regular weekly (and maybe even daily) basis.

How do some of you guys go about doing that? Is there a software package that you use? Do you set aside time and just go through media manager every week and re-back up your files (deleting all of the previously saved stuff)? Or do you simply "drag and drop" ALL of your files onto an external and "overwrite" the previously saved stuff?

Inquiring minds want to know!

Steven Gladney



Sometimes the obvious is hidden in plain view.

I backup my project files, and keep a backup of the music and pictures I use in a project.

But that's it.

I never back up my media. Why? Well, I have it backed up already. I have the source tapes, or in the case of footage shot with the HVX-200, I have the P2 media backed up on hard drives on my shelf. If I lose a media drive, then I simply recapture, or re-import.

Final outputs, I keep a copy of the master tape. Although I have toyed with the idea of exporting a self contained file and dropping that onto external drives, but hard drives don't store as well as tapes do, so I haven't done it yet.

I simply drag and drop and overwrite the project files.

That was me - and no, I had no joy getting the data off the drive. I'm blaming it on the the fact the drives were set up in RAID 0 for speed, which seemed to throw off all of the backup programs - I'm happy to be corrected about this.

As for point 2, I think my backup process is a little extreme, but I like it

I set up FCP to save everything to one directory for each different project (ie render/capture files etc etc) - I do still leave the autosave fault to its default location (Home directory from memory).

While working on the project, I copy the directory to an external drive at the end o every night/session, and on completion (client happy, I've been paid), I back EVERYTHING up to DVD.

I know this is overkill in regards to media, but my memory and notes are not great, and re-capturing footage is a REAL pain IMO.

This is just my process. Hope this helps. I think I'm a rarity backing up media files to DVD.

Justin.

How many DVDs does it take? And is it really any faster than recapturing footage?

Hi Shane,

Do the math on the DVDs I guess - 4.2 gig a DVD, so really depends on how large your project is.

My current project of 22.5 minute tv shows seem to take somewhere between 6-8 DVDs to backup.

Takes about 8-9 minutes to burn full DVDs at 4x (safely), so its a process.

However, for example, on the last ep I completed, the footage came from 21 different tapes/sources (yes I know), so i'd much rather have the complete project lying around on 6-8 DVDs.

As for moving the footage back to the G5, I have 2 external DVD drives, as well as my internal, so yeah, for me it is faster than capturing the footage again.

Justin.

I back up my project files to a harddisc.
The source media comes from tape and those are put in our archive.
My edited programs are mastered on digital betacam.
Besides this I record an international version of the same film,
or sometimes only from the relevant part(s) on a second Betacam.

Chris

Backing up the actual media tends to be incredibly painful. I would never use DVDs to backup clips unless I really had very little media (for example, less than 20 minutes of DV NTSC clips). It just takes too long and isn't practical.

If all your media is batch-listed, logged well from timecoded tapes, then you won't ever be stuck losing edits. If you have a *lot* of such clips, say more than 40 hours, then backing up the clips to a mirror drive would probably be valuable. I believe this is the method recommended by Nick Meyers. The advantage is that if you have a complete set of media duplicated on a drive, you can immediately resume editing should your original drive fail.

I usually don't do that, though, because if you do have a lot of clips, the storage can get expensive.

Principles of backups:

1. The project file is king. Back it up at least once a day. I actually do it every two hours or so.

2. Never, ever rely on project-file backups that are going to the same drive as the media or the original, active project file. If the drive dies, you lose both your original and your backups -- which defeats the purpose of having backups in the first place. That's why you should never set the Autosave Vault to go to the same drive where the rest of the files are.

3. I also recommend that the backup location *not* be a drive/disk location that is active and hooked up to the computer at the same time as you edit. Use external media -- I love USB flash drives for this purpose, and data CDs and DVDs work fine for project files as well. Only insert said medium into the system during the backup process, and remove it once the process is done. That way, if lightning hits your computer and everything connected to it is fried, your backups are safe.

4. If your project is comprised of non-batch-recapturable media (eg. Capture Now clips, graphics, stills), then backing up the project file isn't enough -- you must also back up all of the non-recapturable media. With a non-timecoded clip, such as an AIFF imported from a music CD, if you lose the file, even if you take the same CD and import the whole song with the same settings again, the edits probably won't reconnect properly. So you need to back this up as well.

5. Put a date stamp on backups (if it's media, then put a date on the backup folder; if it's a project file, put the date on the project names), but not on your active current project file. I've seen a lot of cases where people put "Airhead.project 05.06.05 edit" as their main project file name, and then keep editing with it. To me, that's bad form, because if you edit with this project file on 05-08-2005, then the date is no longer accurate. With my archives, I can tell by eye, just looking in the Finder, which project file came from what time of the day on which date: "Airhead.proj (05-18-05 1800)" -- that's the project file from May 18, 2005, at 6PM (army time, or 24-hour clock).

6. Never, ever leave a backup file with the same name as your main active project file. If you ever drag a file the wrong direction during backup, you could overwrite your new project file with your old one, and as far as I've seen, that can't be undone.

7. Don't count on Autosave Vault. It's designed for a short-term save, such as the last two hours of work you did before the computer crashed. But Autosave Vault has been known as an unreliable, unpredictable backup mechanism for days, weeks and months of work. There have been reports of Autosave not backing up as often demanded by User Preferences, not backing up at all, and the backed up files being corrupt.

It's my opinion that all that's really important is the project file, the render files (if you don't want to have to re-render) and the audio files. Now, when using the P2 workflow, just moving them onto one drive scares me. Tapes can be a comforting thing but counting on one drive (especially Firewire) for your source material is like putting a gun to your head with one bullet in the chamber.

The real answer to backing up large media file is coming soon with BluRay data storage. 25 or something actually less, GB of disc is much better that 4.3 GB.

I love the idea of capturing video to hard drives but long term storage on a rotating disk drive is asking for trouble.(Especially if it's Firewire)!!!!

External SATA drives are much more reliable to mount every time after an OS upgrade.

For most client projects - we work into the price the cost of two external DV Hard drives -- most projects we do max out about 150 to 200 gigs...so a 250 gig drive can be had for about 250 to 300 dollars....adds 600 dollars to the cost....

We edit everything on G-Raids....and overnight media manage the project to make a clone of everything to one back up hard drive......when we're done - we do it again with the finished files....(and keep a separate folder of graphics - animation template - AE files - Motion Files etc) and move that to the back up drive -- then clone that drive to a second back up on another computer while we keep working on other things.

The odds of BOTH drives crashing or failing are very high....and they will just sit on the shelf waiting for client to update or change in the future.

It's inexpensive insurance against them having to pay to reload everything and rebuild the wheel the next time they want to change their project

MOST OF THIS IS DONE WHILE WE SLEEP...

we tried the DVD method -- and it works for smaller projects if you only archive just what is used in time line -- Roxio Toast Titanium's DATA spaning for DVD's works really well if you need more than one DVD -- and you can drag your entire project and related files to the TOAST window...and it will span the data across however many DVD's you need.

Andy

Cool,

Thanks guys for your feedback.

Jusrus, yeah dude, you are a little hard core with all of DVD's! But, I'm kinda with you with the tapes. I'm not a "Master Logger", yet. So right now, it's just easier for me to back-up my data to an external HD than trying to go back and recapture footage (but I absolutely keep hold of all of my original tapes just in case. Store in a filing cabinet. I want to get one of those Fire Resistant ones one day). Anyway, I kinda do what Andy does. I just make sure I work the cost of an external HD into the final cost for my clients project.

But tell me if you guys think this is OK. I use an external USB enclosure (I don't work from it, just use it for back-up purposes ONLY). Basically, when I fill that up with one clients job (or several clients smaller jobs), I pop out the old HD, store it in one of those static free bags (like what you get from a shipping place) and store it on a shelf until (if ever) I need it. Then I pop in a new one when I'm ready to go to a new project. Would like to know if any of you guys see fault in that.

Derek, I love how you name and date your back-up project files. I really hadn't been doing that on a regular basis, but I'm definatly going to incorporate your method into my workflow.

Any more input would be great.

Steven Gladney



Sometimes the obvious is hidden in plain view.

I'm with Shane. Backing up media when you have access to the source tapes is a waste of time. If I stored every show I did on a HD, I'd be broke by now. Client wants a change after picture lock? They must pay to recapture the media. They are well aware of this, so when picture is locked - I really mean it. Making changes weeks later is a pretty rare occurrence, for me anyway.

Plus, generally, clients have the master tapes anyway. Like Shane, I do keep a master of the show for reel purposes.

As Derek says, the project file IS king. Guard that with all ya got. I have a .mac account with Backup which automatically puts a copy of my auto-save vault and FCP projects folder in my iDisk nightly. I don't even think about it.

If you don't know how to accurately log and capture tapes, well then, that is a vital skill you simply must learn. No pro editor I know uses Capture Now unless they have to or are in a huge hurry and/or don't care about recapturing. For me, I think it's foolhardy to not have a project that is easily recapturable (if that is a word, anyway).



Kevin Monahan
Social Support Lead, DV Products
Adobe
Adobe After Effects
Adobe Premiere Pro
Adobe After Effects and Premiere Pro Community Blog
Follow Me on Twitter!

Kevin,

Don't get me wrong. I do log and capture. Very rarely use capture now (sometimes have to when capturing from an uncontrolable devise like analogue from a VCR or something). My issues are more with reel names and making sure I mark my tapes correctly. I think I'm doing it right. But, quite frankly, I've never had to go back and recapture footage to test it. So if I happen to be doing it wrong, I'm screwed. But, I log and capture only what I need to my HD, name my clips and takes accordingly, and online everything pretty much immediately. So I try to keep what clips I do have to back-up to a minimum.

Steven Gladney



Sometimes the obvious is hidden in plain view.

> If I stored every show I did on a HD, I'd be broke by now. Client wants a
> change after picture lock? They must pay to recapture the media. They are
> well aware of this, so when picture is locked - I really mean it.

I completely agree with Kevin on this one. Clients must realize that being wishy-washy, such as saying picture is locked when it isn't, comes with a price tag. I generally also demand that a client get his/her own drive so that at the end of the project, I give him/her the drive and tapes and I'm not responsible for their safekeeping.

Just last week a major electronics company told one of my employers they wanted to revisit a marketing highlights reel we'd edited for them. This was for the CES...the edit was done at Christmas! Almost seven months ago! To add to the joy, most of their media was delivered to us on raggedy WMV (!) files, video DVDs and the like, and needless to say, we had to use MPEG Streamclip to convert them and it's not timecoded or batch-recapturable. Since the company never told us it was going to recut this thing, and since I wasn't the one doing the backup, the original clips were gone. Even though I had dozens of backup project files,they didn't do any good because they had no media to relink to. So I had to recut the piece using the final, full-quality movie file. No quality loss, but definitely lots of handicaps like not being able to get rid of sound overlaps, and not having access to original graphics files and clips. However, I told this to the client outright -- they dug their own graves by not thinking ahead, and they paid the price.

> If you don't know how to accurately log and capture tapes, well then, that is
> a vital skill you simply must learn. No pro editor I know uses Capture Now
> unless they have to or are in a huge hurry and/or don't care about
> recapturing. For me, I think it's foolhardy to not have a project that is easily
> recapturable (if that is a word, anyway).

To me, any project whose post-production life is in months rather than weeks should be logged in the most meticulous way possible. If it's a narrative film, then it should be shot-by-shot with accurate file names matching shot numbers and camera slates; if it's a documentary, then it should be clips no more than 10 minutes in length. Those couple of hours spent breaking down the tape into short logs will allow your file management to be pristine down the road. And you save time crying about the online, about dead drives and missing media, about overstuffed drives. There was a long thread in here wto or three weeks ago where a guy kept complaining about problems reconnecting media at the online stage -- as it turned out, he captured 43-minute clips and used subclipping, two things that can wreak havoc on your online process.

All of a project's media (video, stills, music, titles) is kept in one central location, its Capture Scratch folder. This exists on an external, removable serial ATA drive (Firmtek). When a project is completed, I place the removable media drive inside a box and store it on a shelf.

The FCP project file is routinely backed up from the system drive to two volumes: a second internal drive and the removable serial ATA drive.

> All of a project's media (video, stills, music, titles) is kept in one central
> location, its Capture Scratch folder.

I'd take issue with that. Batch-listed media -- ie. media that can be recaptured -- should not mingle with non-capturable media like graphics, stills and music. And graphics should be in one folder, music in another. When I was chief editor on a reality show, I had the joyous task of dealing with the file management of about eight different freelance editors, and the worst kind of file management I saw was one involving 300 files in one folder, all arranged alpha-numerically, with only Finder icons telling the difference. I would also put Capture Now media, or any clips not captured from tape (eg. extracted from DVD using MPEG Streamclip, or converted from files given directly to editorial in the form of files) in a place separate from the batch-listed clips.

The rationale? Capture Scratch is the folder created when you set a Scratch Disk, which is where media captured from tape goes. I only ever put "expendable" media there. That way, if you want to archive and you want to get rid of clips that can be recaptured for sure, you don't even have to look -- ditch everything in Capture Scratch. If you mix and match, you'll have to either use Media Manager (unreliable), or do it by hand and avoid all the other non-expendable files that are in your way. Select the wrong file and you'll end up with non-recoverable media being destroyed.

Please put Derek's excellent post (#7) in the FAQ. Project management and Backup are things that would get by a self-trained newbie, for sure.



Kevin Monahan
Social Support Lead, DV Products
Adobe
Adobe After Effects
Adobe Premiere Pro
Adobe After Effects and Premiere Pro Community Blog
Follow Me on Twitter!

Good idea. I've grabbed this thread for the FAQ.