OT: When RAID 5s aren't safe...

Just came across this... Anyone had any experience with RAIDs not rebuilding due to unrecoverable read errors?

[blogs.zdnet.com]



www.strypesinpost.com

Of course. I don't keep records of this stuff or anything, but my gut tells me that of all the RAID disk failures I've experienced, maybe two out of three have been recoverable. RAID 5 is more like your backup chute than a safety net; it gives you a chance to survive a hardware fault. It's not a guarantee. We use it only because the chance RAID 5 gives is better than the absolutely no chance you get with RAID 0.

RAID 5 is the absolute minimum amount of security you can squeeze out of a set of disk drives; it stores your data in such a way that you can survive exactly one failure. RAID 6 is safer, because with it you can survive two or more failures, but you have to have more redundant storage ? that is, you need more hard drives to store the same amount of data. As individual hard drives have gotten larger, and rebuild times have gotten correspondingly longer, RAID 5 has fallen out of favor simply because the window of vulnerability in the case of a single-drive failure is too long. If it takes you twelve hours to rebuild a disk in a RAID 5 set, that's twelve hours in which your data is absolutely unprotected. So you use RAID 6 instead, which means during that twelve-hour rebuild window, you're less protected but not unprotected.

Fortunately, for most of us a framestore failure is an inconvenience, not a catastrophe. Don't get me wrong; it's really inconvenient, but it's pretty rare for a framestore failure to cost you anything more than lost time. The original footage continues to exist on tape or backup drives, our project files get backed up in the Autosave Vault and so on. So we can live with the acceptable risk of losing the whole filesystem, versus the extra cost (and it's extreme) of using RAID 6 or RAID 0+1.

Alright, I read up a bit more on slashdot... Basically the article will imply that a RAID 5 with 3 TBs will fail on rebuild approximately a quarter of the time. However, that really depends on the RAID controller- how effective the ECCs are, as well as whether the RAID controller will simply mark that sector off as unreadable (lose a 512 byte sector and you can probably restore that easily from backups).

Yes, I have my editing stuff backed up, but I'll still sweat a bit if the framestore was to go down.

I wonder if Jon will chip in on this...



www.strypesinpost.com

Well I have my entire Mac and RAID 5 backed up on 14TB of DroboPro which will update overnight if its left on.

If you are serious about keeping yourself protected - as I think Jeff once said to me...

"You never have data until you have data twice"

Mind you that could have been Graeme Nattress as they both occupy the same inter-dimensional rift know only as "K7 the Technologicalisticmindmesh".



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

I didn't study the linked article line by line or anything, but I thought it was referencing the statistical likelihood of block-level unrecoverable read errors on the drives themselves, which has nothing to do with the RAID controller. And URE is a case where the drive itself does everything it's designed to do, but comes back and says, "Nope, I don't have that block for you, sorry." In normal operation, that means you lost a file. But in the context of a degraded RAID, depending on how the controller handles the case, it could mean you lose the whole filesystem, because without one piece of the data, the controller can't calculate the necessary checksum to recover the piece lost when the first drive failed.

How severe this is, of course, depends on how your data is structured. If it's just a bunch of tiny text files ? like an operating system, for instance ? then the severity of the failure depends on how important that particular file, which contained that particular block is. If your filesystem holds a giant database, however, one lost block can mean the whole database is invalid. In our world, it's somewhere in between, because a bad block could mean a lost frame (if you deal with DPX or EXR sequences) or a whole lost Quicktime, but probably not the loss of your whole framestore.

But the larger point here is that RAID is not magic. RAID filesystems can and do fail, pretty often really.

And Ben, I know I have said that line before about not having anything unless you have it twice ? I was just about to repeat it again here, matter of fact ? but I'm damned if I can remember who I stole it from. So I'm perfectly content giving Graeme the credit for it.

That's not a Graemeism, but it could be. The appropriate Graemism is "A backup in time saves the hard drive".

Just been copying my Drobos over to a bunch of hard drives, then mirroring those hard drives.... Multiple copies, multiple redundancy.

Graeme

Would that I could practice what I preach. My dream ? and this is a fantasy of almost sexual intensity for me ? is to have a whole shelf of hard drives, powered up and spinning all the time, that contain multiple copies of my entire digital corpus. Everything I've ever written, all the emails I've ever gotten, all my photos and home movies, all the music, movies and TV shows I've bought. Three or four copies, all fully redundant, all independent so when one of them fails for whatever reason I can just throw it away and replace it.

Instead, I have a single 750 GB USB drive hooked up to my AirPort base station, backing up my laptop over wireless.

I've said, and will say again, that you don't have anything unless you have it twice, but the older and more conservative I get, I'm starting to think you don't have anything unless you have it three times.

>"You never have data until you have data twice"

I think I said that myself (here), but I read it off an article somewhere and i still can't remember where it came from..

It'll send a chill down my spine if my editing RAID was to go down, even though I've been working with back ups of all critical information (P2 files, autosave vault, manual back ups of projects and gfx) on 2 separate drives.



www.strypesinpost.com

There are only 10 types of people that understand binary...

...those that do and those that don't.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

The having data twice maxim was floating around on the Avid-L back in the mid nineties IIRC and I think even then it was unattributed. I've passed it on myself as an operating and facility design principle that's important enough to bear repeating until a set of copies of the data is stored offsite.

ak
Sleeplings, AWAKE!

>There are only 10 types of people that understand binary...

With an error rate of 10^14, somewhere in between.



www.strypesinpost.com

I haven't backed up my RAID 5 solutions. Never have. Have I had a failure? Personally, no. But I have seen Unity failures, and they are more than RAID 5. We have a 24TB SAN here...no failure yet. Can't afford a backup to it, and as it changes daily (constant adding and deleting) it would be a bear to keep up.

Drives fail...it is a fact of life. We just deal.

As for my P2...I have that backed up 3 ways from Sunday. That's MASTER footage...not stuff on tape i can recapture.


www.shanerosseditor.com

Listen to THE EDIT BAY Podcast on iTunes
[itunes.apple.com]

If you don't mind, Shane, what's your favorite way of backing up P2?

- Loren
Today's FCP keytip:

Nudge a Canvas layer by subpixel with Command-Option-Arrows !

Final Cut Studio 2 KeyGuide? Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

Backing up P2?

Either a copy each on 2 solitary archival drives. Or a drobo. Or a separate RAID 1 or 5. Or on LTO with a copy on a solitary drive/NAS for online access if the project is still active.

I've used all of the above except for the Drobo. My fav is an LTO plus solitary drive. You're practically immune to corruption which can happen on hard drives.

I wonder what Shane would use...



www.strypesinpost.com

Or all three at once.

If you get the opportunity, sit down and have a chat sometime with a DIT who worked on a feature of modest budget or larger. I can't repeat them faithfully because I don't remember the details well enough, but I've heard some fascinating stories about the intricate backup procedures feature productions that shot digitally have had to go through in order to get their completion bonds. Nobody's more conscientious about backups than an actuary.