RAIDS

I have two clients who as it happens are both in the market for similar RAIDS. Both are looking for a device that offers:

4 Drives in a single enclosure
Built in RAID 5
eSata Interface

On one system the drives would be connected to an express buss card on a laptop, on the other to an iMac with a modification from OWC to externalize a Sata Port. Neither system has many options on ways to get a RAID 5 connected.

I have seen a couple of products, one from OWC and another from G-Tech that would appear to fit the bill. Does anyone have any other sources or recommendations?

-Vance

There are literally 100s on the market but OWC would be my suggestion.

[eshop.macsales.com]


You could get the drives separately if you can find a good deal elsewhere but then you really need to thoroughly test each one with diglloydtools first.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

wiebetech.com has a beautiful RTX series that you can look at. Trayless SATA array that allows you to literally purchase any media from a local store to go inside the enclosure.

Walter Biscardi, Jr.
Biscardi Creative Media
biscardicreative.com

Just last week purchased the quad interface G Speed Q 8TB (4x2TB) and really like it. Tried at first to buy OWC's version but the salesperson couldn't answer my questions, took all my info and never called me back. Have used the G Speed via FireWire 800 on both my Mac Pro and MacBook Pro and via Highpoint RocketRAID 2314 eSATA card on the Mac Pro. Solid so far.

Hey Vance!! Try MacGurus.com. I used their raids before the mac pros came out with 4 drive bays.

Joe Brown

What Ben said.

OWC took care of me - period. They have units they put together with matching firmware and rigorously tested. Nobody can guarantee that individual drives will work in a 3rd party RAID enclosure (I tried that with a Stardom SOHO Raid from the manufacturer and 4 brand new 1.5 TB Hitachi Enterprise Drives from OWC - could not get it to work correctly. Returned everything to both vendors and decided from now on to get assembled / tested / warrantied units. I'll pay more. The time I lost trying to rig my own was substantial (and TIME = $$$...and I lost $$$).

When life gives you dilemmas...make dilemmanade.

Ben, Joe, would you guys suggest going with the "enterprise" level with OWC? It costs A LOT more...

On that subject, G-Tech claims that since being bought out by Hitachi, ALL the drives in their G-Raids are enterprise level - think that's a legit claim?

Enterprise Drives are meant for heavy digital combat use - supposed to be bullet proof.

You cannot claim that which is not true. That is "False Advertising". That said, I am glad to hear that G-Tech is being absorbed by Hitachi because Hitachi drives are the only drives I have purchased and recommended for the last 6 + years. G-Technology has gone way down hill since being bought by Fabrik (IMHO) - I stopped supporting them years ago when wave after wave of bad units hit the shelves and customer service seemed to be "outsourced" = terrible. My G-RAID died mid-project - lost everything because it was RAID0 with no option for redundancy.

When life gives you dilemmas...make dilemmanade.

Basically if you use the standard desktop drives as long as they are tested properly then they should work equally as well and in recent years are actually faster than the enterprise versions.

However its really a matter of the gamble.

Hitachi Deskstar are fantastically reliable and hi-performance HDDs they come with a 3 years manufacturer's warranty and should work perfectly even in massive RAID arrays.

I personally have made RAIDs with Hitachi Deskstar HDDs of various units and capacities and even the ones over 5 years old are still working.

The Enterprise versions (of most drives) are (estimated/calculated) to have less errors over time and therefore have a 5 year manufacturer's warranty. They also are guaranteed to work well together in a RAID as each device is manufactured (read this as "enabled" to give a consistent performance on each HDD device, whereas desktop devices can vary in performance - but not considerably.

Honestly, unless you are super-paranoid or running a data-centre or your entire facilities shared storage is used by multiple users 24/7 then you would be better saving now and getting the desktop versions with a plan to upgrade the whole RAID set to the latest and greatest in 2 to 3 years.

Whichever way you choose you cannot rely on the redundancy of a RAID to be a "backup"

Anything other than RAID 1 (or derivatives of RAID 1 such as RAID 10) is NOT a backup.

You will need to make provisions for additional and separate devices for backup of the data.

I personally would use 2x RAID 5 (or RAID 6) arrays with one dedicated for backup and/or to separate HDDs.

If you are really paranoid - make 2 or more copies of the backup to separate drives.

Some recommend tape drives to backup - ignore this advice unless your insurance company demands it. It is extremely slow and the cost is crazy and the tech is rapidly becoming obsolete. Its cheaper and faster to make 3 copies onto 3 separate HDDs!


Whatever route you take budget for 3 times as much RAID space as you need.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Wow thanks for all the suggestions. Gotta love this board.

Walter the wiebetech.com drives look interesting. They have so many different arrangements that their site is a bit confusing, but one of their wizards suggested a device that seems exactly what I am looking for.

Joe Brown, I took a look at MacGurus.com, but didn't see a product that had RAID hardware internal to the drive. Because I can't install a RAID card in either situation, the enclosure has to host the controller. I do remember your JBODS, tho. As I recall they travelled well.

DSeelig, encouraging report on the GRAID. That is the system one of the clients prefers, having had good luck with a number of their other products.

Ben and Grafixjoe. Interesting to hear the support for OWC. One of the units I am shopping for is to replace and existing OWC Qx2. All my previous experiences with OWC have been good, but this unit has been a fail. Purchased in October, we just put it into heavy production last month. While it advertises "up to" 300MB/sec performance, it actually tests with the AJA tool at ~100Mb/s. If it was able to offer that performance as a sustained stream that would be great, but alas it cannot. It is spiky, and once every several minutes it just stops recording or playing back for several frames. That of course leads to dropped frames on playback, an annoyance and on record which is a disaster. We had a break this week and returned it to them to be tested, as were told that it tests to spec. They also suggested that the RAID 5 configuration was not a good choice for video production, and recommend resetting to RAID 10. That, of course, reduces usable capacity by an additional 33%.

Grafixjoe ====> "You cannot claim that which is not true. That is 'False Advertising'." You make me smile. I have done a lot of advertising work over the last too many years. Sure, you can't say what isn't true, but man you can walk right up to the line, reach over and it and shake the hand of a lie. Take the case of "up to 300MB/sec" above. Well if you measure the travel time of just one bit, sent alone to the disk, on a tuesday morning in a 68 degree room over a gold wire maybe. But real world, oh yea how about 100 MB/s.

Hey Vance

Just a few questions.

Quote

we just put it into heavy production last month.

Fill any HDD and even RAIDs over half full and you will see a massive drop in performance. How full were the drives?

Quote

it actually tests with the AJA tool at ~100Mb/s

Via what connection?

Quote

RAID 5 configuration was not a good choice for video production

RAID 5 is perfectly fine - but its better on Disk Arrays of 8 or more.

Quote

It is spiky, and once every several minutes it just stops recording or playing back for several frames

Sounds like a duff drive or another issue such as spotlight indexing or the drive spinning down. Could also point to an OS issue.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

I'm actually looking to test a raid at a friend's house. We're pretty excited as the speeds we're looking at means that we can run stereoscopic DPX finishing on features. I'll post eventually when the review is done.



www.strypesinpost.com

Ben-

Lets see if I can knock 'em off in order:

Capacity. When I backed the drive up before shipping for testing I moved ~2TB off the 8TB (6 formatted) unit.
All connections are esata via OWC recommended interfaces.
2 OWC techs tell me that RAID 5 is not a good video format. My experiences suggest otherwise.
This is on OSX 10.6. Spotlight could well be the culprit. I will look into that this weekend. I'm not sure if the array is set to not be indexed. Thanks for the tip!

-Vance

Strypes via what interface? 2x DPX streams has to be pretty high bandwidth. The only experience I have with that is on a Quantel Pablo with their own fibre attached custom arrays.

Quote

spotlight could well be the culprit

For those who didn't know - any drive attached the Mac OS will automatically be indexed - you have to opt out n the Spotlight preferences under "Privacy".

I would however let spotlight do its "thang" overnight on the drives as it does make searching for files later a lot easier.

Also make sure that your formatting is with Journalling off - as this can take the edge off the speed too.

What card are you using for the eSATA? Is it a 3Gbps capable card?

What drives did you opt for? I only recommend the Hitachi Deskstar since Samsung stopped doing the F1 as the only desktop drives for RAID. Otherwise you need to get the enterprise level drives.

Bear in mind that your average SATA II HDD will do between 50 and 100MBps sustained with the newer larger ones going up to 120 maybe 150+ MBps (in the case of the new 3TB Hitachis)

In a 4 disk RAID 0 then you might get ~300MBps

The limitations of a single eSATA are 3Gbps or 2.4 Gbps (in binary) which equates to 300 MBps (binary). So you cannot get more than 300MBps even if your array is capable of it. However any RAID that uses port-multipliers to deliver the data down 1 cable will have an additional hit in performance due to the "overheads" in managing the data down one pipe.

In a RAID 5 the parity is written across all the drives so you lose 1 disks worth of storage. Also if the RAID controller isn't particularly great then you get a performance hit there too.

Block size for RAIDs is also important - if you do audio work then you want a relatively small Block size but for video the you want a large one 256k or more.

My suggestion is if you can safely remove the data from the RAID - try formatting each drive separately and test them thoroughly to see what speed/performance you get out of them before putting them back in the RAID.

The AJA tool is only good for a rough guide - you really need to use diglloydtools

However if you read reviews of the Qx2 then you will see other people not getting much more than 160 to 220 MBps out of it either. [www.barefeats.com]

If you want more you will have to look at the 8 drive arrays with RAID controller cards built-in and connecting via PCIe to Express34

or even RAID enclosures with USB 3.0 and use the CalDigit Express34 USB3.0 card [www.caldigit.com]

I cannot personally vouch for any USB 3.0 RAID solutions.

Sorry to say Vance; It seems that you are running at the limit of your Mac Hardware and really need to consider investing in a Mac Pro and a good quality RAID card and Array.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Quote
grafixjoe
I am glad to hear that G-Tech is being absorbed by Hitachi

I just went through a drive replacement with G-Tech. I had a 1TB G-Raid that I bought over 2 years ago. Worked perfectly, ran it as a media drive over FW800. Last November it started taking a while to mount. I had always kept a SATA back up of the drive using CCC, so I ran it every day for the next few weeks. Finally it died and I thought that was that. I went to the site and plugged in the serial number and they told me that it was still under warranty and to send it back to them. The shipping destination was in Toronto's outer limits but not a company I knew. I thought they might be a service location. 2 weeks later I get calls about customs brokerage for a package from California. It turns out a new drive was being shipped from Hitachi in CA.
The replacement drive is ALMOST identical, save for the power supply which used to be a standard 3 prong lead into an AC power supply and then into the drive via a multipin jack. The new one is a bog standard wall wart with your standard tip/ring jack. It's now the back up to the previous SATA drive.

ak
Sleeplings, AWAKE!

Seem s like Hitachi's are highly recommended. How do they compare to Western Digital Scorpio Blue/Black, Seagate, and Toshiba? I'm looking for a pair for a two drive RAID setup for my Macbook Pro. Appreciate any feedback!

Get 2x 2TB Hitachi Deskstars or if the enclosure you choose support them; get the 3TB as they do a very impressive 150MBps.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Ben,

How about 2.5" size drives? I think the deskstars are 3.5"? I need them for a portable RAID for istoragepro.

In which case take a look at these

[www.seagate.com]

But check with both Seagate and iStorage Pro if they are compatible as I haven't had the chance to use the Seagates yet. Might be my next MacBook Pro HDD if not an SSD.


Out of curiosity which one are you using?

[www.istoragepro.com]



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

I'm using the Pocket View connected firewire 800 for now. I'm waiting for my Sonnet Tech Tempo Pro esata expresscard to connect this drive. and watch it scream!

Quote
Ben King
Also make sure that your formatting is with Journalling off - as this can take the edge off the speed too.

Actually, this hasn't been true for several years now. Journaling is pretty much a non-factor in terms of performance but can help keep your drives functioning properly in various instances (power outages, system crashes, etc.). I did a bunch of testing myself several months ago and found that having journaling "on" had no significant measurable impact on my RAID I/O speeds.

Quote
Ben King
Block size for RAIDs is also important - if you do audio work then you want a relatively small Block size but for video the you want a large one 256k or more.

Block size is not something you simply recommend that "larger is always better." Depending on your RAID level, whether it's HW- or SW-based, the number of drives in the RAID, etc., you may actually get better performance with other block sizes. It depends on your setup.

-----

This stuff is rather technical and testing/benchmarking is the only good way to know for sure the best settings, and whether your setup is working properly...

No RAID level is a suitable permanent backup for your data. Mirrored RAIDs can fail and destroy both (sets of) drives, in addition to single drive (or a single drive in one part of the mirrored set) failing.

Keeping backups longterm on hard drives is also not a good idea. Hard drives are made to be used regularly once they have been first used. Sitting in storage is not a good thing. Little used drives that have been kept (unused) in storage may fail to work a year, or more, into the future. The only truly good way to store important data longterm is on enterprise-level tape backups, such as LTO. Or, some people get a film print, or two, made for longterm storage.

As the FRAM oil filter TV ad used to say, "You can pay me now or pay me later."



-Dave

@ Andrew - glad you had a positive experience. I did not.

@ Ben - I have a new RAID 1 with 2 or those 3 TB drives in it - WOW is it fast. No problem working ProRes HQ / 59.94 / 1080p files in FCP via FW800 on my MBP

Quote

D-Mac says:
Mirrored RAIDs can fail and destroy both (sets of) drives, in addition to single drive (or a single drive in one part of the mirrored set) failing.

Aliens can swoop down and probe everyone in the midwest...but I have not seen that happen yet and until I do (see a Mirrored RAID - both drives fail together), RAID 1 is my favorite choice for my important backup. I use 2 of them everyday. Been that way for years.

When life gives you dilemmas...make dilemmanade.

grafixjoe Wrote:
-------------------------------------------------------
> D-Mac says:
> Mirrored RAIDs can fail and destroy both (sets of)
> drives, in addition to single drive (or a single
> drive in one part of the mirrored set) failing.
>
>
> Aliens can swoop down and probe everyone in the
> midwest...but I have not seen that happen yet and
> until I do (see a Mirrored RAID - both drives fail
> together), RAID 1 is my favorite choice for my
> important backup. I use 2 of them everyday. Been
> that way for years.

Strypes (Gerard) had his mirrored drives both fail a month, or two, ago. The controller in the enclosure went bad and "zapped" both drives, if I recall correctly. Regardless, no single system is adequate backup. So, the drives didn't fail themselves (one may have?), but rather because the controller went bad, or possibly copied bad data from one drive to another, or some combination thereof... nasty business...

It was a rare occurrence, most likely. But, if it happens to you, then it isn't rare...



-Dave

Core blimey Dave you don't arf like to rock the boat!

Quote

Block size is not something you simply recommend that "larger is always better." Depending on your RAID level, whether it's HW- or SW-based, the number of drives in the RAID, etc., you may actually get better performance with other block sizes. It depends on your setup.

It's a rule of thumb - but to a point it is true - a 256K Block setup on the same RAID5 should perform better for larger sequential file transfers such as video capture/playback than for a 32K Block size. Video by its very nature tends to use large files sizes.

But as you point out it does depend on your setup - and unfortunately the only way to tell is to test a particular setup with the sort of files you want to use.

Quote

Keeping backups longterm on hard drives is also not a good idea. Hard drives are made to be used regularly once they have been first used. Sitting in storage is not a good thing. Little used drives that have been kept (unused) in storage may fail to work a year, or more, into the future

Complete conjecture on your part - I personally have over 50 (and counting) HDDs, some from many years ago and we have had this discussion on many occasions; with people as well as myself testing VERY old HDDs (for work we had literally 100s of client drives) which in most cases had been left on a self in all seasons to gather dust - popped on the Mac and it was all still there.

I still have the first 9GB Mac Tell FirePower FireWire 400 HDD from 1999 that hadn't been used since 2000 and it was still perfectly viable after 10 years sat doing nothing, no use, no power.

Although it IS recommended to check and power them on periodically I have found that for all intents and purposes it has not been necessary. I do recommend transferring and even consolidating precious data to larger newer more reliable HDDs after a few years if possible. And factor in that cost is low and getting lower!

LTO is just not a viable backup for a lot of people. The cost of a new LTO-5 drive is $1500 to $2500 plus tax and despite the 1.5TB tapes have coming down in cost to around the $69 to $150 mark - for the same price you could buy several high quality HDDs and backup several copies.

Besides which, the performance of HDD and their reliability has improved by an amazing amount by almost all manufacturers. Plus I always test my HDDs before use.

Another important factor (putting this all in context) is that LTO is also only backward compatible ~2 generations; so you need to keep the old drives working or upgrade to new tape stock too! Tape stock also needs to be kept in a controlled environment. So LTO is not as great as people make it sound!

Like I often say; if you are super-paranoid or your data insurance requires it, get a tape back-up in addition to some HDDs...

...if you can afford it!

Quote

Or, some people get a film print, or two, made for longterm storage.

Those people who can afford a film transfer and then storage in temperature and moisture controlled vaults!

Quote
From Archive America
One new area of storage that has grown over the last few years is in the Film Archive Storage area. There are particular requirements for film archive storage that are not required in other types of box storage file storage and traditional records management. Firstly, due to the volatile nature of all types of film the environmental storage conditions must be strictly controlled. Film stock must be kept in strict environmental conditions, temperature and humidity must be controlled to ensure that there is no degradation of film stock.

Regarding Journaling

Quote

Journaling is pretty much a non-factor in terms of performance

I think the overhead on larger RAIDs is indeed negligible but on smaller RAID arrays or single media drives it tends to be the done thing - I will have to test thoroughly on the faster newer single HDDs and see if it is indeed something we can put to bed. As Journaling can help in recovering files.

Quote
From the Apple Support pages

When Should Journaling Be Used?

Journaling is best suited for servers requiring high availability, servers containing volumes with many files, and servers containing data that is backed up at infrequent intervals (nightly, for example).

If a volume contains read-only data that is not mission-critical, it may not be necessary to turn on journaling if performance is more important than safety.

If your server contains high-bandwidth usage data files, such as large video, graphics, or audio files, you may want to weigh the benefits of using journaling against the performance needed to access your data. In most cases, the impact of journaling upon data access performance are unnoticeable to users, but its implementation may not be practical for servers where data access demands outweigh its benefits.

However we definitely do agree on one thing - RAID is NOT a backup.

Quote

As the FRAM oil filter TV ad used to say, "You can pay me now or pay me later."

A very good mantra



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

>LTO is just not a viable backup for a lot of people.

It depends on what "most people" refer to. If you run a business and require a feasible amount of storage, it may be cheaper to archive onto LTO than on tape. 8 TBs of raid 5 storage, including raid controller and enclosure will cost you in the same region as an LTO drive and tape, except that LTO is more secure.

I would get an LTO setup if I did most of my editing on my own set up, but I don't.



www.strypesinpost.com

It's still cheaper to backup onto separate HDDs GB for GB



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Man if I EVER am at a party with other editors and the conversation flags I am just going to ask, "So what do you use for storage."

Ben, about 14 messages ago you asked about controller cards. The array in question is swapped between two systems, a 17" laptop and a MacPro tower. The interfaces are the ones recommended by OWC. The tower card is rated at 6 Gb/s and the express card at 3 Gb/s.

[eshop.macsales.com]
[eshop.macsales.com]

In both systems that I am dealing with, this package and the other IMac based system I am designing, I have limited interface possibilities. Both use, or will use, and IoHD so the firewire buss is spoken for. That leaves esata interfacing for the drives. Both clients are more comfortable with a RAID protection design. That means the enclosure must host the RAID card. There are other systems, some of which I use and like, that have a RAID controller in the enclosure, but require custom interface cards in the computers.

As for measurement tools, the AJA is a common bench mark for complete systems and is a good quick check for verification. I have not use the digiloyd tool, but I did purchase it yesterday morning. I am looking forward to have a couple of hours some day soon to download it and see how it does works.

As for the current project with the OWC RAID. I give up. I had the client order a good old reliable GRaid, hardware stripe 0. I have a couple of drives in the tower software striped as 0. For the month remaining I am going to cross my fingers and hold my breath. All the source material exists as video tape, and the finished work is delivered and archived on video tape each week. All a drive crash costs me at this point is time and breaking my New Years resolution about swearing.

Thanks for the help. When this project is behind us I'll get back to the RAID battle.

-Vance

"good old reliable GRaid"

That's an oxymoron in my book.

When life gives you dilemmas...make dilemmanade.

Quote

The array in question is swapped between two systems, a 17" laptop and a MacPro tower. The interfaces are the ones recommended by OWC. The tower card is rated at 6 Gb/s and the express card at 3 Gb/s.

Of course you won't get any performance increase by using 3Gbps single eSATA to 6Gbps thats a bottle neck there.

You might want to look at 6Gbps enclosures and HDDs such as the Hitachi 7K3000 range for the MacPro.

I'll keep an eye out for some solutions but you might be better making your own using a relatively low cost external PC case and mounting something like the Areca RAID controller [www.areca.com.tw] and 8 or more HDDs with or without hot-swap cages. [bit.ly]

One of my first entirely self-built external eSATA RAID used an 8x Hitachi Deskstars in a 8x 5.25" bay PC case with 3.5" shock absorbing adaptors and I installed 4x dual eSATA ports and linked to the Sonnet x8 eSATA PCI-X card via an octopus's orgy of 8 separate eSATA connections!

It is not scary or difficult to make one of your own. But often the sourcing and testing of parts puts people off.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]