MacBook Pro External Hard Drive Question

I need some advise on external HDD enclosures.

I recently downscaled my iMac 27inch (Quad i7) to a late 2010 MacBook Pro 17inch (2.66 GHz Intel Core i7 with 8 GB RAM).

I only ever work with HDV tapes (1440x1080 50i) & Canon 5D MarkII files (1920x1080 25p), which I religiously capture or transcode to:

• HDV Tapes get captured as Apple ProRes 422 files
• Canon 5D get transcoded to Apple ProRes 422 (LT) files

The external drive I work with is a Taurus Dual Bay 1TB drive which is a RAID0 setup



and gets connected via the FW800 port on my MBP. The read/write speeds of this drive (using AJA) are:



I understand that Apple ProRes 422 has a data rate of 105 Mb/s, which this drive should handle.

Apple ProRes White Paper




I was hoping in finding a solution where I can scrub ProRes 422 just like Proxy.

I've uploaded a video so you can see the differences between scrubbing 422, 422 (LT) & 422 (Proxy) on my current setup.





I need to be able to edit mobile, but in order to do this currently I find myself converting ProRes 422 files to either ProRes 422 (LT) or even ProRes 422 (Proxy) when I have the time. Then when the edit is complete I delete the Proxy's and re-attach the ProRes 422 files and export to a master.

Although this solution works just fine, the problem is I sometimes need to do a same day edit and don't have the time to transcode twice, which is the reason for my post.

I guess the million dollar question is, is it possible to scrub ProRes 422 files like Proxy on a MBP? and if so what type of HDD enclosure or hardware would i need to do so?

Any help would be greatly appreciated.

My rig is a Macbook Pro with external RAIDs. I personally have never heard of "Taurus" so I do not know what type of drives are inside. I personally use (FW800) a 2 TB CalDigit VR RAID "1" (1 TB volume) for a work drive:

[www.caldigit.com]

and for Time Machine I use a NewerTech Guardian Maximus 6 TB RAID "1" (3 TB volume) that is daisy chained to the VR:

[eshop.macsales.com]

The whole system is very efficient & smokes.

I do not use "LT" or "PROXY". I have no clue why you cannot "scrub ProRes 4:2:2". I can scrub all forms of ProRes including 1080 HQ. Have you tried zooming in on your timeline and scrubbing? Are you in Dynamic / Safe RT in your timeline? Sounds like a settings thing.

When life gives you dilemmas...make dilemmanade.

Yea. I'm not sure why you can't scrub on that thing. But I haven't seen the video. I have one of those Tauruses in Raid 1, and I have full specs for that thing and on Fw800 you can get around 60 MB/s read/write depending on the drives in it. FireWire is always less responsive than a raid 5 running off esata. The things I don't like about the array is the lack of a display screen and screw mounted drives.



www.strypesinpost.com

Well it is a Mirrored Raid and not striped so it not going to be any faster then a single drive maybe slower due to mirroring? Much safer though if one drive dies on you.

------------------------
Dean

"When I see you floating down the gutter I'll give you a bottle of wine."
Captain Beefheart, Trout Mask Replica.

Mine is raid 1. The OP is running raid 0, so his data rates should belch faster than mine.



www.strypesinpost.com

BUT...the OP is saying they have a RAID "0" set-up and the Disk Utility pic he listed shows a Mirrored RAID Set. RAID "0" is NOT a "Mirrored Set". RAID "1" is Mirrored. RAID "0" is Striped. WTF??

When life gives you dilemmas...make dilemmanade.

It's configurable by setting the jumpers. I hate it that it's not too user friendly, but once you're set, you're good.

edit: I can't see the pictures as they say "domain unregistered", but if it is raided in the array, it shouldn't show up in disk utility, as it isn't a soft raid. It should only show up as a single volume. Not sure exactly what is going on since I can't see the pics, but I'm wondering if the OP is using another model which has 2 separate data outputs which he raided with disk utility. Raid 1 shouldn't be a soft raid as it uses too much processor power.



www.strypesinpost.com

Quote
grafixjoe
I personally have never heard of "Taurus" so I do not know what type of drives are inside

Opening the Taurus enclosure I found the following drives:

• x2 Barracuda ES.2 SATA 3.0-Gb/s 500-GB Hard Drives SPECIFICATIONS ON THEM HERE

And here are some detailed pictures of this enclosure.

Quote
grafixjoe
BUT...the OP is saying they have a RAID "0" set-up and the Disk Utility pic he listed shows a Mirrored RAID Set. RAID "0" is NOT a "Mirrored Set". RAID "1" is Mirrored. RAID "0" is Striped. WTF??

The Taurus HDD enclosure has a switchable pot on the back (which is set to RAID0), not sure why the disk utility shows them as being "Mirrored" but it's definitely RAID0.




Going off the specs from the Segate website, these drives should absolutely "SMOKE"

Quote
grafixjoe
Have you tried zooming in on your timeline and scrubbing? Are you in Dynamic / Safe RT in your timeline? Sounds like a settings thing.

Sure have. Whenever I scrub the timeline I always zoom all the way in, then zoom out 5 times, which gives me a 1:1 timecode.

I've also tried the following:

• Trashed Preferences
• Repaired Disk Permissions
• Set Safe RT playback and Frame/Playback to Dynamic
• I'm making sure there is at least 20% free space on the HDD

Screen snapshot of my RT settings.



The only thing I'm left to do is re-format and reinstall all Final Cut Studio..!

I'm up against the wall and really don't know what else to try, any suggestions......?

Quote
strypes
I can't see the pictures as they say "domain unregistered"

Hmmm....OK... I've re-uploaded the screen snapshots and re-edited that post. Let me know if they come up now...?

I see. It's a different model. Well, the best you're running off this drive is Firewire 800. Not sure why you don't get eSATA. 100 MB/s is the theoretical best you get off Firewire 800.



www.strypesinpost.com

That looks like an old unit. What RAID these days does not come with an eSATA port? IYou don't really NEED eSATA (I ron all flavors of ProRes on my RAIDs via FW800 without a dropped frame. The controller in this unit may not be working properly. You have to eliminate the enclosure as the problem, so either buy a new one or borrow a friend's external RAID and run the same project on that. If it is the same issue, you can bet is in your FCP / System settings.

When life gives you dilemmas...make dilemmanade.

Quote
strypes
Not sure why you don't get eSATA

Strypes have no problem in forking out some extra $$$$ to buy a suitable enclosure, I just want to make sure it will do what I'm wanting, that's scrub smoothly with 422 files.

Quote
grafixjoe
That looks like an old unit. What RAID these days does not come with an eSATA port?

I totally agree, I bought it x2 years ago when I had to cut 10bit uncompressed files for a production company I was contracting too.

Quote
grafixjoe
You have to eliminate the enclosure as the problem, so either buy a new one or borrow a friend's external RAID and run the same project on that

Funny you mention that, I took my MBP to a friend to do further testing today and found some really interesting results. I think in order for me to properly explain our findings, I'll record a video podcast in the next few days and visually run you through our tests. Trying to write it up might take weeks ;-)

Anyway standby and I'll that recorded in the next few days.

Cheers....

Video Podcast? Dude...you have gone above and beyond already...you are spending a whole lot of time and effort on simple troubleshooting. Are you trying to get answers or teach us something?

When life gives you dilemmas...make dilemmanade.

I haven't watched the video yet. Not sure what you mean by scrubbing. For playback without dropping frames, both your drives and your processor needs to be able to handle the throughput as well as the processing. The bottleneck is usually in the throughput, with the exception of some h.264/r3d streams, which requires more processing power. With regards to scrubbing through a video, the demands are much greater, and there really isn't a benchmark for that.

Your first picture isn't necessary, as you aren't using the Mac OS raid utility. The speed test is close to what would expect, especially on drives that aren't empty. The bottleneck in drive access speed isn't in the raid configuration or the drive I/O, but the interface. FireWire has a theoretical max speed of 800Mb/s. That is roughly 100MB/s. You do not expect to hit that, not as an average real world speed, except maybe in max peak readings. 80MB/s read speed is decent and expected, with occasional dips. In short, the issue could very well be processor. Your MacBook pro has only 2 cores with maybe hyperthreading. To encode and decode more than one stream of ProRes hd is likely to be a stretch for the processors. Faster data throughputs and beefed up ram and drive cache may help, but it is close to the limits of what your machine cam handle.



www.strypesinpost.com

Quote
grafixjoe
Video Podcast? Dude...you have gone above and beyond already...you are spending a whole lot of time and effort on simple troubleshooting. Are you trying to get answers or teach us something?

Guys I'm sorry If I've come across as being a little too thorough. The last thing I wan't to do is invest in a new enclosure, to only find similar results.

Quote
strypes
I haven't watched the video yet. Not sure what you mean by scrubbing.

Strypes, I'll try explain this as simple as possible.

After capturing HDV tapes as ProRes 422 files, I create a new ProRes 422 sequence and drop the captured footage on the timeline. I then "scrub" through the footage to find the shots I'll be using in the final edit. It's this "scrubbing" part i'm having problems with, when working with ProRes 422 files, I just can't scrub fluently.

ProRes 422 (Proxy) on the other hand is really sweet and fluent to scrub, and yes before you jump at me I totally understand proxy is a much, much, much lower data rate, which is why you can scrub fluently.

I guess the bottom line is, even if I switch to an eSATA connection, will that solve my problem?

There is one thing that is very unclear to me. The readout from the AJA System Test shows the Taurus enclosure to have a Read speeds of 67 "+" MB/s, and write speeds of 80 "+" MB/s, isn't that enough speed to scrub ProRes 422 file fluently...?

I'm totally confused...!

Quote
blimpmedia
After capturing HDV tapes as ProRes 422 files, I create a new ProRes 422 sequence and drop the captured footage on the timeline. I then "scrub" through the footage to find the shots I'll be using in the final edit. It's this "scrubbing" part i'm having problems with, when working with ProRes 422 files, I just can't scrub fluently.

ProRes 422 (Proxy) on the other hand is really sweet and fluent to scrub, and yes before you jump at me I totally understand proxy is a much, much, much lower data rate, which is why you can scrub fluently.

I guess the bottom line is, even if I switch to an eSATA connection, will that solve my problem?

There is one thing that is very unclear to me. The readout from the AJA System Test shows the Taurus enclosure to have a Read speeds of 67 "+" MB/s, and write speeds of 80 "+" MB/s, isn't that enough speed to scrub ProRes 422 file fluently...?

I'm totally confused...!

From the Apple ProRes whitepaper (Apple ProRes Whitepaper (July 2009)):



Roughly speaking, ProRes 422 (Proxy) is the only version of ProRes that will work smoothly with your external drive. Changing FCP's Real-Time settings for the sequence may help (though, picture quality may decrease and some frames may be skipped). Of course, the graphic shows a 1920 x 1080 frame size at 29.97 fps, so your situation may be about 20% less than the data rates shown (or less?).


-Dave

Dave, right now I feel like hiding in a dark hole I can't believe I skipped that whitepaper graph, That's a slap in the face

Dave correct me if I'm wrong, so if I go ahead and purchase an eSATA PCI card, and get a CalDigit VR enclosure, It's safe to say I'll solve my problem...?

Dave the whitepaper says Mb/s, is that Megabit or Megabyte?

blimpmedia Wrote:
-------------------------------------------------------
> Dave, right now I feel like hiding in a dark hole
> I can't believe I skipped that whitepaper
> graph, That's a slap in the face
>
> Dave correct me if I'm wrong, so if I go ahead and
> purchase an eSATA PCI card, and get a CalDigit VR
> enclosure, It's safe to say I'll solve my
> problem...?

I couldn't remember the range of values, off-hand, for the various ProRes data rates
(so I looked in the whitepaper myself).


Since you're using a MacBook Pro now, I don't know how that constrains the data rate,
if at all, when using an ExpressCard eSATA adapter (Sonnet Tech Tempo SATA Pro ExpressCard
adapter is good and probably doesn't limit the data rate).

I haven't used a CalDigit VR system, with either a desktop or MBP, so someone else (Joey?) can
answer the specifics. But, if the CalDigit performance specs on their website are correct, then
you should be able to use even the RAID-1 mode (mirrored) and get enough speed.


-Dave

I finally found a company that deals with HDD solutions (here in Sydney Australia) for video editors.

Explaining to him what hardware I have and what I'm trying to achieve, he technically explained the downfall in using the Taurus enclosure. The Taurus uses software solution to create RADE 0, as the CalDIGIT VR enclosure stripes it's RADE 0 using a hardware card.

He strongly suggested for in-house editing I get a Cal DIGIT VR 2 or 4 TB unit, and a 1 or 2 TB mini CalDIGI for on location.

He guarantees the CalDIGI VR enclosure with the CalDIGI PCI card using eSATA will run smoothly and scrub ProRes 422 without a hitch..! He also has a money back garentee if it doesn't meet my expectation. So I guess I can't go wrong Well.... I'm off to buy a CalDIGI VR and be satisfied once and for good

Glad to hear you got that sorted. You should be well-served by the CalDigit setups you mentioned. Happy shopping!

Let us know how they work out. I may need to get some new portable drives soon, as well as a new MacBook Pro.

Will do Dave, thanks again to all for your help

>Taurus uses software solution to create RADE 0, as the CalDIGIT VR enclosure stripes it's RADE 0 using a
>hardware card.

I would be curious as to his definition of software raid and hardware raid. That drive array is probably raided with the oxford chipset. You have 2 disk drives with a FireWire output, you cannot software raid the thing. Also, a raid 0 on software raid, isn't such a bad idea.

There are 2 bottlenecks here- the drive interface and the processor. You are asking for scrubbing. That requires more processing power and faster throughput. Simply getting an esata adapter may help isolate the cause, it may help the scrubbing, but the issue may very well lie in simply not having enough processor power on a 2 core MacBook pro.



www.strypesinpost.com

strypes Wrote:
-------------------------------------------------------
> >Taurus uses software solution to create RADE 0,
> as the CalDIGIT VR enclosure stripes it's RADE 0
> using a
> >hardware card.
>
> I would be curious as to his definition of
> software raid and hardware raid. That drive array
> is probably raided with the oxford chipset. You
> have 2 disk drives with a FireWire output, you
> cannot software raid the thing. Also, a raid 0 on
> software raid, isn't such a bad idea.
>
> There are 2 bottlenecks here- the drive interface
> and the processor. You are asking for scrubbing.
> That requires more processing power and faster
> throughput. Simply getting an esata adapter may
> help isolate the cause, it may help the scrubbing,
> but the issue may very well lie in simply not
> having enough processor power on a 2 core MacBook
> pro.

Those are good points, Gerard. Thanks for pointing them out.

In addition to the CPU power you mentioned, I wonder if the GPU also affects
the scrubbing performance.


-Dave

Quote
Strypes
There are 2 bottlenecks here- the drive interface and the processor. You are asking for scrubbing. That requires more processing power and faster throughput. Simply getting an esata adapter may help isolate the cause, it may help the scrubbing, but the issue may very well lie in simply not having enough processor power on a 2 core MacBook pro.

Hmmm.... ok what I might do is take my MBP with me and ask if I can test the CalDIGIT VR before making the purchase.

I'll keep you posted.

I want more visual aids. i think they're the future ;-)

- Loren

Today's FCP 7 keytip:
Play from Playhead to Out Mark with Shift-P !

Your Final Cut Studio KeyGuide? Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

Just to make sure that nobody might get confused by the speed terminology:

Your speed test says:
- Read 80.2 MB/s
- Write 67.4 MB/s

ProRes needs about 105 Mb/s

Please nobody confuse "MB" (MegaByte) with "Mb" (MegaBit)

1 MB/s (MegaByte/sec) is 8 times more than 1Mb/s (MegaBit/sec)

So the actual Mb/s speeds of the hard drive is:

- Read 641.6 Mb/s
- Write 539.2 Mb/s

This is enough overhead to handle ProRes easily.

D-Mac Wrote:
-------------------------------------------------------
> In addition to the CPU power you mentioned, I
> wonder if the GPU also affects
> the scrubbing performance.

No. The graphics card does only get what is already been read from the drive. It does not matter if you scrub or not.

One more thought:

Dropped frames can also come from a not fast enough decoding process by the CPU and/or a too slow/overloaded system bus. The speed of the drive is fast enough, but then the CPU needs to decode the compressed ProRes stream. This is much more complicated than to read an Uncompressed stream with a much higher data rate from the drive.

To the OP: Do you have a CPU monitor? (I like MenuMeters. It shows you how much your CPU is used.)

>This is enough overhead to handle ProRes easily.

Just to stress on this. That's average speed readings. As Christopher mentioned, speeds like that is good for single stream playback (while accounting for dips in drive throughput). The issue is the OP is trying to scrub, not playback. Scrubbing in a lot more taxing on the processors. Let's say you are playing back a 220 Mb/s stream at double speed. The processor has to decode 440 Mb/s, the drive array also has to handle that bitrate. This is also why you allow never get smooth playback on time-lapse clips in FCP without rendering. Scrubbing is a different ballgame and you cannot spec a machine out to scrub smoothly.



www.strypesinpost.com

Quote
Christoph Vonrhein
1 MB/s (MegaByte/sec) is 8 times more than 1Mb/s (MegaBit/sec)

So the actual Mb/s speeds of the hard drive is:

- Read 641.6 Mb/s
- Write 539.2 Mb/s

Christoph, that's why I feel so confused...! If this is true, then the Taurus enclosure shouldn't even flinch at ProRes 422 (being 105 Mb/s), but that's not what I'm experiencing. Also from what others have mentioned, the read/write speeds of this enclosure seem to be right, well pretty close to what FW 800 should be producing.

The guy I spoke to at eurekapacific here in Sydney Australia, advised me that the CalDIGIT VR enclosure will do exactly what I'm after, and specificly said "Scrubbing" ProRes 422 will work without a hitch.

He also told me that all video data rates are read as "MB" (MegaByte) not "Mb" (MegaBit), even the Apple ProRes Whitepaper. How true that is, I have no idea...!

I'm now awaiting my CalDigit VR 4TB enclosure from B&H.

What would be the best way to go:

1. Sonnet Tempo SATA Pro ExpressCard/34
2. CalDigit FASTA-1ex SATA ExpressCard