system choking on ProRes

Hello out there...

please help...

do others have success playing '2k' ProRes files off an external, non RAID hard drive?

Here is my situation:
My new Mac Pro is choking on playback of ProRes HQ files (2048 X 1024 - they were transcoded from RED R3Ds using Red Rushes)

Please see my system specs below in my signature.

In FCP, my sequence is Unlimited RT, Playback Video Quality Low, Playback Frame Rate Dynamic. It chokes when playing a clip in the Viewer or on a single video layer sequence.

I sadly suspect the culprits are the three external drives I'm running the footage from... two are running off eSATA and one is FW800.

they are OWC 1.0TB Mercury Elite AL-pro. They have either a Hitachi or Seagate Barracuda inside. They are not RAIDed.

All my tests, research and consulting with a Post house led me to believe that my system would be fast enough to accomodate files of these size off external drives. Was I gravely mistaken?

Oddly, things were running fine for past 2 weeks until today (off these same drives). But today, FCP started choking (able to play a few seconds here and there before the spinning beach ball).

Thanks for your troubleshooting ideas....!
(I've done the usual - ie. disk permissions, restart...)

ALSO ---
So what are my options? Buy faster RAIDed drives?
OR
Downconvert these large ProRes files and make a smaller 'offline' sub-masters? .... I've already Merged all my footage (30 hours!) with Sub-clipped audio files. I hope and pray things will be able to link up to the new sub-masters...

If the downconversion is required, I'm leaning toward DVCPro HD 720.

thanks,
Susan

Susan E. Kim
Mac Pro 2 X 3 ghz quad-core,
8 gb ram,
NVIDIA GeForce 8800 GT,
FCP 6.0.5

Quote

do others have success playing '2k' ProRes files off an external, non RAID hard drive?

...no way. A RAID array is a MUST for video performance. You are playing Russian Roulette with straight drives.

When life gives you dilemmas...make dilemmanade.

Nope. Wanna work with high data rates and RED, gotta get RAIDED drives.


www.shanerosseditor.com

Listen to THE EDIT BAY Podcast on iTunes
[itunes.apple.com]

Just guessing, did you add a lot of footage to the drives today? It's easy to forget that hd performance drops as the drive gets more full. Maybe you were just squeaking by before, and when the drives got more full and slower you reached the limit of non-raid performance.

ProRes should be OK at lower res (1080i or 720p) on non-RAID eSata, if that is an option.

>All my tests, research and consulting with a Post house led me to believe that my system would
>be fast enough to accomodate files of these size off external drives. Was I gravely mistaken?

I'm not sure who you talked to, but no one I know of will recommend a FW800 interface for anything higher than single stream ProRes HQ 1080/ Uncompressed SD editing. It's too much of a stretch, and definitely not a solitary drive for what you are doing either. It's out of the question.

I believe you're looking at somewhere around 60 MB/s for sustainable single stream data rates (can't really find a calculator for this right now, so these are my rough estimates so give and take 10 MBs). And that doesn't include multiple streams and overheads... You'll need RAIDs, and a RAID 5 (probably 4-5 disks) off at least eSATA.

>If the downconversion is required, I'm leaning toward DVCPro HD 720.

Yes, if you can't afford to get bigger, meaner drives. DvcproHD 720p edits alright on a FW800 drive. Recompress, then online it on a bigger badder machine.



www.strypesinpost.com

I think you're overestimating just slightly, Strypes. When I first came to the facility where I work now, I spent several months cutting on a system that had only a two-disk internal stripe for a framestore. I don't remember the numbers exactly, but I'd be surprised if my I/O bandwidth topped a hundred megabytes a second. I was able to play back multiple streams of ProRes HQ 1080i60 with no problem at all. Of course, 2K 2:1 is a bigger frame than 1080, but it's also a lower frame rate, so I doubt the I/O requirements are that huge.

Susan, Final Cut will tell you the data rate of your clips; it's one of the columns in the browser. You can also select one and hit command-9, I think it is. What's the data rate of one of your 2K 2:1 clips?

I don't think it's necessarily that a single Firewire drive can't play back one of these files. I think it's more likely that, for whatever reason, your single Firewire drive can't play back these files right now.

I think I've got some R3Ds at work. If I find a few minutes today, I'll render out some 2K 2:1 ProRes files with RedRushes and see what happens on my system.

First things first! It is certainly not your Mac... even in high quality playback.

Susan - how full are the HDDs that you are playing from?

Press Command+i on the HDD icons on the desktop and give us the capacities and the available space.

If its over 50% full then your HDDs performance will start to drop and as you approach 10% free space then weird things start to happens and performance will be awful.

As a rule of thumb 30% (20% as a minimum) free on an individual HDD should be ok.

Academy 2K in ProRes HQ is going to be about 14MBps to 20MBps with a Stereo Audio track - add more realtime tracks or try to playback realtime VFX and you are going to have problems.

A single HDD via Firewire might give you about 20-30MBps Read/Write if its not too full, so you really are pushing it via a single drive if you have filled it up!

Maybe delete some render files that you don't need to free up space.


Honestly you should certainly be looking at a minimum of two Drive RAID 0 - preferably via eSATA but FW800 should do.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

>I think you're overestimating just slightly

By golly. You are right... I was calculating rates proportionally to data rates at 2000 vs 1000 pixels... Just noticed that it was horizontal pixels. My bad. A solitary FW800 is okay for single streams, ensure sufficient storage...



www.strypesinpost.com

Yeah, the thing to keep an eye on is that "2K" off a Red is almost always 2048x1024, fitting with their 2K 2:1 format. It's a lot smaller than Academy 2K.

I believe there's also a 2K 16:9 format (which comes out, I think, to 2048x1152), but for some reason I haven't seen a lot of footage in that format. Not that I've seen a LOT of Red footage at all, but as a fraction, I've seen very little 2K 16:9.

Even so... the HDDs are usually the problem when it comes to playback as they are often the bottle neck.

A single HDD can playback HD/2K ProRes but not if it isn't performing/functioning at its peak, hell, even DV on a very full HDD can be jumpy!

Thats why we buy RAIDs - you must have overheads in terms of performance so that your minimum performance meets the basic data for your chosen format.

Anyway - lets not digress to much.


Susan

What are the capacities and available space on each of your media drives?



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

First - Thank you everyone for taking the time to read and respond to my post!!! Amazing help.

Jeff,
From spot checking a handful of clips, in FCP the data rates range from 17.9 - 22.6 MB/sec.

Again, they are 2048 x 1024, 23.98 fps, 2:1 clips


Ben,
Yes... seems I'm guilty of filling too much to capacity. Yikes! I guess I'd thought 10% free was okay and then I really pushed it... bad susan.

Drive 1: via FW800, capacity 931.39 GB (as formatted, was '1 TB'), 38.41 GB available
Drive 2: via eSATA, capacity 931.39 GB, 105.55 GB available
Drive 3: via eSATA, capacity 931.39 GB, 81.22 GB available

Does the 20 - 30% free space rule apply to the full 1TB or the actual capacity after formatting? (ie. 1 TB or 931 GB?)

The documentation with my drives claims max transfer rates as follows:
eSATA - 1.5 Gbit/s (or 150 MB/sec)
FireWire 800 - 800Mbps (or 100MB/sec)

I suppose actual use doesn't come near those rates, huh? Esp w/drives filled to max capacity....

So I must ask...
Is there any chance I could get by using the HDDs I have, leaving at least 30% free space on each? I don't totally understand why clips with 22 MB/sec rates shouldn't function on a drive that can get up to 150 MB/sec? Forgive my ignorance...

Anyway, I'll try making more space on my drives and see how that affects performance. Will also start researching RAID 0 options.

thanks again for your insights....!
Susan

Susan E. Kim
Mac Pro 2 X 3 ghz quad-core,
8 gb ram,
NVIDIA GeForce 8800 GT,
FCP 6.0.5

>FireWire 800 - 800Mbps (or 100MB/sec)

These are max rates. In practice, they do not come even close. Not to mention, Firewire is a very chatty protocol. The eSATA drives should be fine, if they are not at 90% full (then it's hard to gauge how far the performance will drop, since drive performance varies).

You can run this to test on a drive that is 90% full.

[www.lafcpug.org]



www.strypesinpost.com

Quote

Does the 20 - 30% free space rule apply to the full 1TB or the actual capacity after formatting? (ie. 1 TB or 931 GB?)

the 1TB refers to the capacity in Decimal Bytes which is what HDD manufacturers use to make their things sound bigger... (probably some male sales rep with an inferiority complex came up with the idea of using the decimal!)

Formatted capacity refers to the actual Binary Bytes used by computers.

If you want to understand more read this: [en.wikipedia.org]


As far as performance as you even from 100% as you fill a HDD its performance starts to diminish - go over the 50% point and your average SATA HDD (don't confuse this with the connection protocol like eSATA/FW) will start to show a more obvious decrease in performance. So going to 30% free or even 20% free is quite likely to drop frames.

Quote

The documentation with my drives claims max transfer rates as follows:
eSATA - 1.5 Gbit/s (or 150 MB/sec)
FireWire 800 - 800Mbps (or 100MB/sec)

This is the connection protocol data-rate NOT the HDD data-rate!

Quote

Is there any chance I could get by using the HDDs I have, leaving at least 30% free space on
each?

Maybe - there are such a variety of HDDs and performance that I wouldn't like to say for sure - the only thing would be to test them.

I would seriously look at a RAID 0 though...



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Quote

Will also start researching RAID 0 options

Cheapest options:

You could put 2x or 3x 1TB HDD inside your Mac and RAID them - or even the Seagate 1.5TB HDDs!!

Keep the external HDDs as backups for footage and for any files you make or need to keep!



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Ben King Wrote:
> You could put 2x or 3x 1TB HDD inside your Mac and
> RAID them - or even the Seagate 1.5TB HDDs!!


Thanks, Ben, for all the info. Really appreciate it.

In addition to my 500 GB HDD from which I run my OS,

I currently have two (2) - 1TB HDDs in my Mac Pro. I could easily get one more internal HDD!

Is the following correct? I would:

1. Add 1TB more for total of three (3) '1 TB' HDDs
2. Move off ALL material currently on the internal HDDs (ie. onto external drives)
3. Use Disk Utility
4. Create ONE "RAID set" by selecting all 3 of my internal HDDs from the left column
5. RAID them as a "Striped RAID Set" (aka RAID 0)... ?
6. Get back to working with all media linked to internal RAID 0 HDDs!

Would the full capacity of these individual HDDs after making them a "Striped RAID Set" still be 931.5 GB each?

I have a total of 2,543 GB of footage...

So 3 TB (in reality, 2,795 GB) would leave me a little short of space if I wanted to keep my headroom of 30%...

Do internal Mac Pro HDDs max out at 1.5 TB?

Susan

strypes Wrote:
> You can run this to test on a drive that is 90%
> full.
>
> [www.lafcpug.org]
> Getting_Started_FAQs#How_do_I_check_the_read.2Fwri
> te_speed_of_my_drives.3F


Hey strypes,
cool, thanks for this info and the link. The AJA test and data rate calculator are very helpful. I'm checkin out the System Test now...

cheers,
susan

Susan E. Kim
Mac Pro 2 X 3 ghz quad-core,
8 gb ram,
NVIDIA GeForce 8800 GT,
FCP 6.0.5

whoops just realized I was digressing too much. Sorry 'bout that.

I just started a new thread in which I propose my RAID-0 drive solution.

it's called... "RAID-0 drives"

thanks, and all my best,
Susan

editor08,

Keep the "RAID 0" chat in this thread...please do not start another of the same subject. It creates forum clutter & will be deleted. Everyone is still answering in here so no reason to split the thread.

Peace.

When life gives you dilemmas...make dilemmanade.

cool, ok reposting this back in this thread...

For my 2.5 TB of footage, I am considering the following RAID-0 solution: (I'd keep all footage backed up multiple times on other external HDDs)

Does anyone work with something similar to this?
Is it fast enough for my 2048 x 1024 footage? Stable?

-- Burly 4 Bay SATA Hot eSwap 4 Bay Enclosure
-- 4 Burly Standard SATA Drive Trays
-- 4 drives totalling 6 TB ---- Seagate ST31500341AS 1.5TB Barracuda SATA 7200 RPM 32MB
Cache 8.5MS Hard Drive
-- eSATA connection only

----- runs about $1,200

Any problems with just having eSATA connection, no FW, down the road? (ie. for Color correction, layoffs, finishing)

cheers,
Susan

Susan E. Kim
Mac Pro 2 X 3 ghz quad-core,
8 gb ram,
NVIDIA GeForce 8800 GT,
FCP 6.0.5

We've been cutting a Red show in ProRes Hq for some months now. The system has 3 Internal Terabyte Sata drives and One External. The footage plays great. The system is a bit unstable though. The G5 Mac Pro Quad Core 8 gig of ram system crashes 5-10 times a day.

Very occasionally a particular clip locks up, and anything more then a basic effect won't render. Anything like a matte or speed change gets cooked, and brought back into the Project.

I've turned off all the automatic software checks, and used all of Ken Stones tips. and I'm going to re-balance the drives as per this discussion, but I'd entertain any other ideas about making the system more stable.

Quote

The footage plays great. The system is a bit unstable though. The G5 Mac Pro Quad Core 8 gig of ram system crashes 5-10 times a day.

That is ridiculous. Your drives set-up is not working, Ethan so I wouldn't recommend it.

RAID, folks. RAID...and not a SATA RAID either. Sata RAIDs are notorious for losing performance as the drives fill up. Caldigit's Firewire RAIDS were tested by Barefeats. Look at these stats:





The Empty SATA write starts out great - Full SATA write drops from 124 MBPS to 73.
The Empty FW800 starts at 69 MBPS - Full FW800 STILL writes at 69 MBP.

That's impressive...which is why I bought 2 of them.

BTW...the single drive takes a dump as it gets full. RAID PEOPLE!!

When life gives you dilemmas...make dilemmanade.

Don't understand those statistics. Sounds like the Full SATA is faster full than the Empty FW800... How is being slower, better?

It's very simple. This illustrates the difference between SATA & FW800 (CalgDigit) in the "SUSTAINED WRITE" department. FW800 handles ProRes CONSISTENTLY through the filling up of the drive. SATA drops off as it gets full. With video, you need SUSTAINED TRANSFER RATES or you will get dropped frames (even on a SATA RAID closing on "FULL". The FW800 is a bit slower, but it is plenty fast for ProRes and keeps it's SUSTAINED WRITE SPEED throughout the drive.

Understand now?

When life gives you dilemmas...make dilemmanade.

Hey again,
Don't like to leave a thread hanging. I ended up getting a 4-bay Burly enclosure with port multipler and four - 1.5 TB Seagate drives.

The Burly guys are great - knowledgable about storage, responsive - and I've personally been working with a helpful guy named Brian.

I got my Enclosure and the drives, zeroed the bits in Disk Utility for good measure, formatted them and striped them RAID-0 in Disk Utility.

After some software/hardware troubleshooting, I'm quite sure the Burly enclosure itself performs fine (I can describe my troubleshooting in detail if you're interested) --

However, that said, BEWARE those new 1.5 TB Seagates - they have a bug that can crash FCP and, subsequently, your whole system - at least, that's my experience.

I definitely attribute this crashing to the 1.5 TB Seagates because they were crashing tiny projects that worked perfectly when linked to the very same media on my internal HDDs.

I've gotten my system working to a point where the erratic crashes happen once in awhile in FCP (aka haven't had a crash in maybe about a week).

Seagate released a firmware upgrade for the 1.5 TB drives, a few weeks ago, which is supposed to resolve these video issues.

I haven't yet gone through the upgrade, but I will and will let you know how it goes. If other Mac users have gone through this upgrade, please let me know how it works for you.

best wishes for all of you out there in editing land,
Susan

Susan E. Kim
Mac Pro 2 X 3 ghz quad-core,
8 gb ram,
NVIDIA GeForce 8800 GT,
FCP 6.0.5