It's been six days, I'm at my wits' end...

Hello, All... Long time lurker, occasional poster... Some of you may remember me...

I am having a nightmare of a problem that actually began weeks ago, and culminated in upgrading machines, OSs, some Apps, media drives, and almost every single file but my clients' media, and I still don't have any answers. I am hoping & praying that this is where the forum system shines...

It started in PremierePro 5.5. (we'll get to FCP) After 11 successful exports, the last approved version of the program just refused to export. I tried multiple codecs and changed settings myriad times. This was on a 2009 MPB running Snow Leopard with 8 GB of RAM, an eSATA RAID connected via Express/34 card. All connceted drives have significantly more than 10% space available.

The export would usually get to about 26% and PrPro would crash, and Force Quit was the only way out. Sometimes it was 2%, once it was 98%... but most of the time it was 26%. I was able to get it out in two halves and stitch it together in QTPro7. I thought the problem was PrPro, so after exhausting all the usual remedies of general houskeeping, etc., reinstalling became the next viable troubleshooting method. I opted to upgrade to CS6, since I was reinstalling anyway (CrCloud is too big of a jump for me right now, so let's just leave that be). After the Uninstall/Install Adobe dance, it was exactly the same problem. In all respects. Not a whit of difference.

I made my delvierables from the QT7 stitched file and moved on to another project. This time in FCP7. Everything was going fine for the first two versions. I get to the third version, and FCP crashed during the render. It kept crashing. And crashing. I was able to render it in small chunks and get it out, but the project is far from done. I thought perhaps it was failing RAM (2-4GB SIMMs from crucial.com). It passed the TechTool test & Rember found nothing after running overnight, however. So I made a Genius Bar appointment. They found nothing on the bench test, but wanted to send it out to the out-state facility for a stress test. I was all for that, but I needed to get my backup plan in motion because I can't not work for the next 5-7 days. I then thought to put the old 2-2GB Apple RAM chips back in to rule in/out bad RAM. Same problems. Just slower.

Everything had been working fine on that system for at least a year. Since that MBP was running Snow Leopard, I hadn't updated anything for quite awhile. It had been working. Until it didn't. The only thing I'd installed was the XDCam Transfer utility and other XDCam software required to run the U1 drive. That was months ago, and that project ended before any of this started.

So it became clear it was time to invoke the backup/upgrade plan. I migrated my edit system to my husband's i7 MPB that is one version newer than my MBP. It is running Mountain Lion. Fresh installs all around for FCP7, CS6, and I haven't even tried Avid 6.5 yet, although I will need that on Tuesday next week.

Cut to the chase... Everything has changed, yet nothing has changed. New computer, new OS, fresh installs. I needed a 2TB portable drive for a traveling project at the end of the month, so I even was able to rule out the RAID & the Express/34 card by copying the media to the FW800 external drive. The only things that were the same between the computers were a handful of clients' fonts that I have now deactivated, some FCP plugins that I deleted except for the one I am using (TMTS windblurcross... they like it...) and my Digital Juice library that I just nuked off the new MBP. I also found three .png photos that were slightly not-quite-optimized, so I adjusted them to be no larger than 4000px 72dpi RGB 8bit files, as well.

I just tried the PrPro project to see if perhaps I was chasing a new FCP-on-Mountain-Lion problem, but it crashed at pretty much the same spot it usually does. I've even tried opening up the next most-recent project I did in PrPro because the media was still on the RAID and tried to export that. It exported fine when I finished that project the first week of May. It refused to export the same way the other PrPro project does today.

My most current attempt was to make a new FCP project (second time I've tried this, but I'm running out of ideas) and that is rendering now. It crashed the same way as I was proofreading this.

I've lost six days to this, and I have deadlines that are looming fast, not to mention new projects I should be starting next week. And I'm supposed to take my system to a live event and edit an overnight project for presentation the next morning at the end of this month. I need a system I can trust!

Can anybody tease anything out of this?!? I have a smattering of Console & Crash Logs if anyone thinks they could ferret something out from those... I've tried the Digital Rebellion Crash Analyzer, but it gives me a non-sensical error that can't be traced to any particular piece of media. And I find it hard to believe the one piece of media it complains about is causing problems in 2 different NLEs on three different projects... I don't think any of them share any media... But perhaps that is what I should check in the morning...

I am flummoxed and otherwise running out of ideas. I just trashed the Digital Juice Juicer app from Applications and removed the files from Library>Application Support>DigitalJuice. Could it really be the TMTS plugin causing all this mayhem?

Many, many, many thanks in advance,

debe

Holy cow! This is indeed a job for Jon Chappell. He often checks in so hopefully he will have an answer soon. Crash Analyzer wont solve the problem but maybe Jon can decipher it.

Michael Horton
-------------------

Deb, you haven't given us the specs for your media. The fact that you were having problems across multiple applications, multiple software reinstalls, multiple editing systems and even multiple media drives -- added to the fact that the system was working just fine prior to this -- suggests to me that there's something inherently wrong with your media. Have you performed surgery on the media? One method I use for something like this is to put everything out of reach of the system (not just offline, but to put the media onto drives which aren't connected to the system), and then re-copy them over one small group at a time. In your case, because you've already tried so many times to fix it, I would reconnect one group and then try the export.

How big is the project? Did you consider re-ingesting/re-capturing the master clips in batches? Did you try creating test projects where you take out any "exotic" file (graphics, stills, nests) or use flattened versions of them?


www.derekmok.com

Yea, it sounds like corrupt media. Have you tried rendering your edit in parts to find the problematic clip?

Jon Chappell has a Corrupt Clip Finder utility in his Pro Maintenance Tools that could help. There's a free trial.

[www.digitalrebellion.com]



www.strypesinpost.com

Since it crashes at around the same spot, and you mentioned around 26%, that would probably be the location of the corrupt clip.



www.strypesinpost.com

Also, try a simple Media Manager trim (create new media). That may create new media that doesn't contain the corruption. Corruptions often carry over to resaved versions, but this would be a fast, relatively painless test that has a chance of vaulting you over the deadline. If it works, you can get the show out and figure out the underlying problem later.


www.derekmok.com

One project is just over 1 TB of media. That's the PrPro one. The current FCP project is just over 244 GB. They are, strangely enough, both history projects. Both are also standard definition timelines because so much of the original media is 4x3 SD and neither was interested in the extra cost to upres. The third project I pulled out of the PrPro mothballs is 117 GB and that one was always 1920x1080.

I am not against the idea of it being corrupt media, but would reason dictate that it has to be media that is common to all the projects? I used a basic full-screen, slightly-better-than-a-white-pop transition in all of them, but each project has it's own copy of the animation file. It's a not-too-flashy flashy transition that just seems to work for all of them. I also have a full screen white PNG that I use for a white-pop when that is needed, but they are actualy different white PNGs. They come from the same Elements file on my Drobo, but they are differnet copies of each in each project. Two use the HD frame size white PNG and the other uses the SD frame sized one.

I can start pulling that out of the current project and see what happens. Even though they all crash a lot at the same spot, it's not always the same spot... And if I get them past the part where I think they are cranky by rendering or exporting in pieces, it doesn't crash either PrPro or FCP. If it was corrupt media, would I never be able to get past the bad media?

As far as making the media offline and bringing it back in small chunks... If I can already render successfully in small chunks, how will I know when I've found the corrupt media? (I don't mean any of this to sound snarky... I really hope there are answers that I don't see since I'm too close...It's hard to discern tone in a forum post, so I'm just coming straight out and telling you that I am earnestly asking these questions. They are not meant to sound sarcastic or snarky... If they do, please chalk it up to six days of utter frustration.)

For ease of archiving later, I do keep all the original photos (many of them are large, CMYK scans that I converted) & the edited/color corrected/cropped RGB PSDs in the same GFX folder with the PNGs I export to use in the projects. Even if they aren't in my timeline, would them being in the folder and in the project because I imported the whole Photos file instead of just the PNG Photos file? They aren't being used in the timeline, but they are in the project.

The PrPro project is the history of all the major events at a local extremely larger retail establishment. About 30% of that footage is from tape, BSP, DVCam, MiniDV. Some of it was captured by me in 2010 the first time I did this progject. The rest was captured this year. I have 3 real tape decks and a Blackmagic card for capturing in my aging G5. The remiander is a hodge-podge of tapeless. Most of it was XDCam EX that I ingested via XDCam Transfer. They are .mov files, but in the XDCam codec. There were some formats I didn't recognize, but MPEG Streamclip did. I turned those into ProRes. There was one AVCHD that came in late. I just left that one, since PrPro is supposed to be so good with "throw anything at the timeline". When I ended up XML-ing this to FCP to get a OMF the ProTools guy could use, that event, of course, was unreadable by FCP. Luckily it was uninteresting audio that we wouldn't have used anyway...

The current one is largely files pulled off of DVDs. They are the only sources they had for most of their historical events over the years, as they recently purged several banker's boxes of BSP & DVCam tapes because "hey, why would we ever need these?!?". There is a smattering of VHS from earlier years, and some DVCam that I had in my personal archive for this client (been working with them since 2004) & (yes, I have a functioning VHS deck...) A few digital files came from the PGA and (with permission) from YouTube. (At least this client gets the fact that just becasue you can download it doesn't mean you can use it...)

The one I re-opened just for fun was photos from an event and HDV tapes sources that I captured and converted to ProRes. The project was partially an AE Template from Digital Juice.

I'm going to make the edits today to the current project, render in small batches, and get something to the client today becasue I have to show progress. TheMr (my husband) did go over the Console & Crash Logs this morning and came up with some other ideas, too, that I will also try. But he's in Denver on a show and isn't available until later in the evenings, so I am reaching out to as many resources as I can to try to ferret this problem out.

Thanks for the ideas, Derek, Strypes & Mike. Please let me know what you think of my questions above!

deb

> would reason dictate that it has to be media that is common to all the projects?

Not definitively.

> If I can already render successfully in small chunks, how will I know when I've found the corrupt media?

If you keep being able to export successfully piecemeal, then either there's no corrupt media, or the problem will resurface at some point when you put back in a specific chunk. Either way, it's something that needs to be ruled out.

I also assume you've ruled out human error -- such as garbage at the 10-hour mark on your timeline? Ten per cent is also on the low side for your drive space, so you may want to look at your system drive to see what you can take off. Just stick them onto a large external drive (even USB would be fine) until you're done whacking away at the global problem.

> Even if they aren't in my timeline, would them being in the folder and in the project because I imported
> the whole Photos file instead of just the PNG Photos file?

I think it's possible. CMYK files and overlarge graphics usually wreak their havoc in a timeline, but I've seen just the act of importing or opening them (into the Viewer) slow down system performance and cause hiccups. For the sake of troubleshooting, I'd eliminate them from your bins. If you're not editing with them, you can always re-import them later (master-affiliate relationships are really pretty irrelevant when it comes to stills), though I don't really see why you imported the original, misformatted versions in the first place. It's just wasted RAM and wasted project-file size.

I'd also consider converting some if not all of the XDCam material -- only the clips used -- to ProRes and then re-conforming the new ProRes clips to the edit. It's just another system-taxing X factor, in my view.

> The current one is largely files pulled off of DVDs.

I've also had clips ripped from DVDs, usually with MPEG Streamclip, behave in funny ways. That's yet another variable you may have to check out.


www.derekmok.com

It is significantly more than 10% on all drives. I just mentioned 10% because in the old days, that was a common number we tossed out there: There's 2.4 TB available on a 6 TB, 580 GB on a 2 TB, and 350 GB on a 1 TB. System drive has 316 GB available of 437. There is a Boot Camp partition of 62 GB that has 28 GB free.

It turns out the CMYK photos aren't actually in my project. They were in a separate folder in the Photos folder on the media drive, but I did not pull the whole Photos folder in. It was just the PNG folder. I just pulled everything but the PNGs out of that Photos folder for the time being. There was one PNG that the Corrupt Clip Finder kept bringing up. I remade it 4 times & CCF kept complaining about it. Then I sent the original JPEG scan to anohter PS user to see if he could remake the PNG... That hasn't come back yet.

Of course, I plucked sections from the DVDs because I didn't need the whole DVD... so I pulled the :20,:30 I needed. That'll make for a fun thing replacing the clips...

The XDCam EX material will have to wait. That's the project I've already delivered. But it will come back for updates over the years, so I'll have to sort it out eventually...

Thanks! I'll see what happens when I get that new PNG back...

deb

"> would reason dictate that it has to be media that is common to all the projects?

Not definitively

> If I can already render successfully in small chunks, how will I know when I've found the corrupt media?

If you keep being able to export successfully piecemeal, then either there's no corrupt media, or the problem will resurface at some point when you put back in a specific chunk. Either way, it's something that needs to be ruled out."

Oh, and on these ones. I did move just the media for the current project to that external 2TB drive, cutting off all the other media on my RAID. It crashed the same way. Is it reasonable for me to assume that if the problem is corrupt media, that the problem is in that media?? That would definitely help narrow it down... from somewhere within 5.6 TB to just 240 GB...

deb

> I did move just the media for the current project to that external 2TB drive, cutting off all the other media on my RAID.

Did you dismount the RAID? What other peripherals do you have hooked up? Are you on a network?


www.derekmok.com

I powered the RAID down, removed the eSATA Express/34 card, and used media via FW800 from the external drive.

Other peripherals are Wacom Intuous 3 tablet, USB external Apple Keyboard, second Apple monitor connected via the Mini-Display port & USB (from the monitor's power, Mini-Display & USB multi-cable), and external Monsoon computer speakers. On the back of the Apple monitor are a slew of pigtails hanging out to connect various drives. Nothing is connected to them. They are just waiting for drives & printers & whatnot.

I did disconnect the monitor once (thereby removing the pigtails and the external KB from the situation). Same thing happened. That leaves the Monsoon speakers and the Wacom tablet? Should I try disconnecting everything but the RAID?

I disconnected the internet this morning. That was one of the things TheMr suggested.

deb

>I powered the RAID down, removed the eSATA Express/34 card, and used media via FW800 from the external
>drive.

Good suggestion. I have actually retired my eSATA Express/34 card. eSATA is ridiculously problematic on portable devices. I contacted Sonnet about it and they suggested that most eSATA issues are caused by bad connection. The Sonnet eSATA cables actually have a thumbscrew to secure the connectors, but on a portable device it's pointless.



www.strypesinpost.com

Btw, can you upload some of the crash logs?



www.strypesinpost.com

[www.dropbox.com]

Crash logs are on my Dropbox for your viewing pleasure. I tried to name them with something that shows what I had done on some of the more recent ones.

There are several in there. The Console log, several FCP crash reports, and one brand-new Motion one. I had Motion open on the old MBP because I was looking for the name of a font I had used that wasn't on the new MBP... when I reconnected the new machine to the network, I got the dueling apps warning. Both Motion & FCP closed on the new machine, but allowed me to save. I didn't need to save as I had just done so, but after that, the apps closed correctly. On the old machine, Motion crashed hard. The Crash log for that is in the file as well.

The eSATA RAIDs were great on my G5. Although the laptop doesn't leave my desk all that often, I have never felt great about the Sonnet card, either... Waiting for my new MacPro tower before I upgrade storage solutions. So that makes today a good day, at least! :-D

After I'd ruled out it likely wasn't bad RAM in the old computer, I was starting to lean towards perhaps it was the eSATA card acting up. That's when I tried the FW800 test... I was really hoping that was the answer... <sigh>

Thanks for sticking in with me. I did get a new project v1 out the door without any crashes, which does help point the finger back to the other media.... although both the offending projects started out behaving... let's see where we are on this one after I do a graphics pass...

deb

If you can export in chunks, can you export a self contained quicktime for each chunk, then reimport it to a fresh project and see if that functions ok? This will create all new media (albeit stitched together unless you go through and add handles) and although it will be inconvenient for the parts that need changing, this far into your edit you're probably done changing quite a lot of the cut anyway.

The parts that need changes you can cut out and replace with newly imported media, and if this goes haywire, you'll know which clips were the problem.

Of course this is an emergency-recovery-deadline-looming method. Not a recommended method for anyone who is not already in some kind of trouble.

That is a good alternate solution, Jude!

Of course, since this is a history project, they want a full archive. Why can't stuff like this happen on one-off projects that will never see thelight of day again??

Once I've delivered, I will sit down and sort it out. You've all given me good ideas!

deb

It looks like a corrupt clip to me from the logs.

My software:
Pro Maintenance Tools - Tools to keep Final Cut Studio, Final Cut Pro X, Avid Media Composer and Adobe Premiere Pro running smoothly and fix problems when they arise
Pro Media Tools - Edit QuickTime chapters and metadata, detect gamma shifts, edit markers, watch renders and more
More tools...

Here are a couple of guesses at the problem:

1. One or a few corrupt clips. This could be a single problematic still that you happen to have in common.

2. Storage corrupting media
This is a swipe at a bigger underlying problem and this is based on my bad experience with 2 different eSATA expresscards (and I'm talking 2 different chipsets, not just manufacturers), I would take the eSATA expresscard out of the works for a few months to see if things stabilize. Have you come across any issues such as dmg files not verifying on mounting or failing the verification process on mounting?



www.strypesinpost.com

I could probably take the eSATA card out of the equation and port things to FW800 drives per project, once all this is over... I have lots of TB of active media at the moment, however...

But my gut has been telling me it's the card...

I don't have any .dmgs on the RAID. I keep active media on it only. I can drag some from my Drobo and do some tests later.

John, thanks for confirming. The Corrupt Clip Finder (which I found eventually... stress...) calls out only one .PNG, which I have made and remade 4 times. I even sent it out to someone else to rule out something I keep doing. I sure hope that is the thing! (and when I pay off the other 6 TB of storage I just bought, I shall be purchasing the Pro Maintnance Tools... Indespensible!)

Thanks again, all!!

deb

Have you tried turning all the PNGs into TIFFs or JPEGs? PNGs are supposed to work fine in FCP, but I've had properly profiled RGB TIFFs fail until I resaved them as PSDs or JPEGs.


www.derekmok.com

Derek: I really don't WANT to turn the PNGs into JPEGs or PSDs... but I suppose I can write an action for that... :-D

Strypes: When you bailed on your eSATA Express/34 storage, what did you replace it with?

Tip: hover over the file in Corrupt Clip Finder to see why it was flagged up. Also, you may get better results from scanning the media directory than the project file.

My software:
Pro Maintenance Tools - Tools to keep Final Cut Studio, Final Cut Pro X, Avid Media Composer and Adobe Premiere Pro running smoothly and fix problems when they arise
Pro Media Tools - Edit QuickTime chapters and metadata, detect gamma shifts, edit markers, watch renders and more
More tools...

I did discover the media directory tip. Very resourceful!

I will try to hover over the clip, to!

Thanks, Jon!

debe Wrote:
-------------------------------------------------------
> Strypes: When you bailed on your eSATA Express/34
> storage, what did you replace it with?


At home I run RAID 1 drive arrays, since the work I do at home isn't much and those drives come with quad interfaces. Now, I haven't had much issues with eSATA on desktops. What would I replace eSATA with if i needed a fast connection? Thunderbolt.



www.strypesinpost.com