Archiving with a Drobo

Hi,

I'm working at a company who have chosen to purchase two Drobos - a Drobo FS for scheduled nightly backups via Time Machine ; and a Drobo Pro for long-term archiving purposes (simply eject a drive when it's full, stick it on a shelf, and insert another drive.) I know that hard drives don't last as long as eg. LTO tapes etc., but I'd be interested to know how effective people think this setup is as a backup/archive solution ? Are there any major problems I should be aware of?

Thanks, and I look forward to your comments!

You stated it yourself...hard drives do not last or were not meant to last on a shelf. Inerts get dried out over time and you may find that one day a few years from now when you plug that drive back in, the electricity passing through it may be too much for the old circuits and POOF. It happened to a camera I had...old hi8 camera I didn't use for years...just turned it on one day to archive tapes to DVDs and fried all the capacitors with plain old electricity. Who the hell even knows if Drobo or even SATA ports will be around in 10 years...so you will have to hang on to that Drobo unit forever (in which case you cannot be guaranteed they won't go out of business and stop supporting it altogether. I have a few 10 year old DVDs full of quicktimes that were rendered using a Targa 2000 proprietary hardware codec (you can only read the files with a Targa Board installed)...and the board is not supported anymore. S.O.L.

I have been told by people that work in Motion Picture and Broadcast Post that LTO tape is not going away (they recommend LTO 5 because the units read back 3 versions). It will probably be the only tape format left standing.

When life gives you dilemmas...make dilemmanade.

Many thanks for your reply Joey. Longevity is of course key. (I remember Hi 8, but not Targa 2000.)
I'm hoping that the archive disks will need to be re-inserted often enough (to retrieve old projects) so that they won't just be sat on a shelf untouched forever; but it's early days yet, so I can't say for sure. And even with frequent spin-up I know that one day they'll eventually go to that big drive in the sky :-)

I'm still interested to hear more comments though...

Thanks

I think your best bet is bare SATA drives, not RAIDed - just swap them out as they fill up. I spin mine up twice a year. I'm currently using 2TB drives, and I'm juuuuust getting to the point where I would like to take out my 250GB drives and copy 8 of them over to a fresh 2TB drive. The truly nervous (or people with more valuable data than I) will have a second as a backup, or even a third offsite.

Anything coming off the production lines today will still be supported for ... eh, 10 years or so. If SATA starts to die off, transfer over to whatever's next (which will probably be 10TB by that time).

Thanks for your reply Mike. Are you using the Drobo as well? Sounds like we have a similar system. Seems a shame to just be using the Drobo as an enclosure/JBOD though ...

>I have been told by people that work in Motion Picture and Broadcast Post that LTO
>tape is not going away

That's because there is a big market both in and beyond the film and tv industry. Governments and banks use it.

Robk, ok, haven't yet worked with the drobo, but I'm under the impression that you can't swap out the drives the way you intend to, because they work more like a striped array. You only swap out drives when they have failed or if you plan to expand the array. Removing a drive will only cause the array to go into a degenerated state, and when you pop in a new drive, it will rebuild the array.



www.strypesinpost.com

Thanks Gerard/Strypes - I've not worked with a Drobo before either, and my understanding was the same as yours (which is why I posted the question.) However, the IT department is claiming that the drives aren't configured/striped in any way, and so are hot swappable.
It would be good to get some clarification on this if possible...

Thanks!

A drobo is a redundant storage system, so it functions like any striped array. The benefits are that it is intelligent and adds redundant blocks to the drobo server without you having to reformat the data to add to the storage pool. So it is not a traditional striped volume in this sense, and you are not locked into the existing disk space when you configure the drobo.

The problem is that it appears as a massive drobo volume and you don't know where the data is stored, so you cannot eject a drive then pop it back in and expect the data to be still intact.

As Joey mentioned, you cannot predict the longevity of a format, especially not proprietary formats owned by one company.

LTO is more feasible as an archival format because it was designed as an archival format and has widespread usage both in and outside the film and tv industry. The problem is that the start up cost is reasonably pricey, there is a lot of hogwash around it and some people believe the bullshit without doing proper research, and next to that, it is not a real time "online" type of archival system, so you need to keep track of catalogue numbers and you need to restore entire catalogs, sometimes just for one file. Not sure if this has changed. Another issue is that not everyone has an LTO drive, so you may end up with an LTO tape and the deck has been DOA for years, and you need to find somewhere with the deck to restore the data so you can use it.



www.strypesinpost.com

Many thanks for this useful response! I'm not sure yet exactly how the Drobo has been set up (I.T. gave me the impression they were going to customise it somehow) but you're making me realise it's almost definitely the wrong archiving solution. I think they'll have to either repurpose it as a shared storage unit or replace it with something else.
I understand the plus and minus points of LTO tapes, and found Shane Ross' recent blog post very interesting! [lfhd.net]

Cache-A is cool, but pricey. But I'm curious... Why BRU and not retrospect? I remember retrospect being a lot cheaper.



www.strypesinpost.com

Last October we implemented two DroboPro-FS units for our long term and near line archive solution. At the time it was a difficult decision to make because like Grafix Joe, many years ago we archived a lot of QT files to DVD using the proprietary codec for Pinnacle Systems CineWave board if anyone remembers those capture cards. So dealing with a proprietary solution for something as crucial as our archives had to be taken very seriously.

What we really liked about the Drobo units was the ease with which the capacity of the units could be expanded. We also liked very much the Drobo's redundancy and recovery abilities. For those who wish to be able to put their archives in the vault and forget about them a disk based archive system is a poor choice for reasons already discussed and LTO tape is the only way to.

We chose to go with the Drobo for our long term archive because we keep the machine up and running 24*7. The archive unit is synched with the near line units. If a drive goes bad on either system, we know right away. A simple hot swap of the bad drive and we are golden. Because of the useful lifespan of hard drives one has to be prepared to say that every couple of years all the drives will have to be cycled out with new drives. However, as drives get larger, faster, and cheaper, this only works to our advantage. Every time you cycle through drives you can expand the capacity of your storage without losing your data or raid structure.

Worst case scenario: Protection is the two units. If the archive or near line units has a catastrophic failure, the other unit can be locked down until a new Drobo can be built and the data re-secured. In theory assuming products are in channel, 24-48 hours recovery time to bring a brand new machine online. If Data Robotics goes the way of our CineWave, duel redundant systems (we hope) will give us plenty of time to migrate to another solution.

David_S
Sr. Bottle Washer
[www.BottledVideo.com]
There is little left for shameless advertising when you are giving it all away for free

The only thing I don't like about the Drobo...well, other than it being proprietary, but that ties into this...is that I can't just plop in the one or to drives I need that have the footage I archived. See, I tend to back up projects to one drive, or two...(mirrored to another set) so that I can easily just grab the drives I need. With the DROBO, you need to have all the drives mounted...AFAIK. Plus, I like to archive projects that I then either hand to the drive to the clients (charge them a fee for the drive...but it is small as I use BARE drives) and they can get one of those $30 drive docks and access the footage whenever they want, or I bill them for the drive and keep it at the office. Access it when I need it.

I did check out archiving solutions at NAB, and Cache-A was the best at the moment (waiting for LTFS), but BRU from the Tolis Group was more affordable...and still tape. I'm hoping to review on of those solutions soon.

[lfhd.net]


www.shanerosseditor.com

Listen to THE EDIT BAY Podcast on iTunes
[itunes.apple.com]

I rand the original Drobos with their DroboShare NAS adapter for a few years. I found them slow, and very finicky about mounting direct and being shut down in the case of power outages and things like that. They were also a pain to back up.

So.... earlier in the year I was looking for a better solution, and thought the new DroboPro FS would be great. But it was rather pricey and I wanted two so I could make a clone from one to the other for backup. The more I looked into things, and the more I researched, the less I was convinced.

After a lot of searching, I ended up with some DiskStation boxes from Synology. Unlike the Drobos, they run on open source software that is very easy to use and extremely powerful. They're also a fair bit cheaper than the Drobos and have a very easily configurable cloning ability. So now I have a DS1511+ that clones it's shares to two DS411j units. The DS411j know to turn themselves on sunday morning, and the DS1511+ backs itself up across the two smaller units. Then it sends me a very polite email to tell me "it's done".

To me, the key thing for a NAS is how do you back it up. If you get that aspect sorted, and it fits in with how you work so you don't even have to remember to do it, that's what gets my vote.

Graeme

David_S Wrote:
-------------------------------------------------------
> Because of the useful
> lifespan of hard drives one has to be prepared to
> say that every couple of years all the drives will
> have to be cycled out with new drives. However, as
> drives get larger, faster, and cheaper, this only
> works to our advantage. Every time you cycle
> through drives you can expand the capacity of your
> storage without losing your data or raid
> structure.

There is a little-known RAID problem that will rear its ugly head at this point... namely if you buy five drives at the same time and they're in the same RAID at the same temperatures with the same workload, and one gives out after a few years... you pop in a new one, and then ask the other four, the twin brothers of the deceased, to start working 24/7 to copy over TBs worth of data onto the new drive. Basically, you have reached MTBF, and your drives which have been on a (comparably) easy workload for years and are near death... you're asking them to now run a marathon.

Beware.

RAIDs should not be considered long-term protection. They are great for protecting active data in coming hours, days and weeks. Separate, self-contained backup drives (or a format like LTO tapes) should be used for long-term protection.


www.derekmok.com

Thanks for all the feedback so far! I'm liking the sound of the DiskStation boxes, and I'm also looking into a v.impressive storage platform called Space from GB Labs, http://www.gblabs.co.uk/
On the LTO front I also have several choices, including the BRU (thanks Shane!), but I'm also curious about strypes' question above: isn't Retrospect a lot cheaper in comparison? I'm also thinking LTO-4 rather than LTO-5, again to keep costs down.

Cheers!

Mike Watson Wrote:
>
> There is a little-known RAID problem that will
> rear its ugly head at this point... namely if you
> buy five drives at the same time and they're in
> the same RAID at the same temperatures with the
> same workload, and one gives out after a few
> years... you pop in a new one, and then ask the
> other four, the twin brothers of the deceased, to
> start working 24/7 to copy over TBs worth of data
> onto the new drive. Basically, you have reached
> MTBF, and your drives which have been on a
> (comparably) easy workload for years and are near
> death... you're asking them to now run a
> marathon.
>
> Beware.

Mike,

Thank you. This is one of the issues we contemplated when deciding which way to go. Our plan is to begin the cycle to replace all the drives well before a drive has gone bad or the MTBF point has been reached. Further, if one drive does fail it was felt that if this occurred well before the MTBF point it could be chalked up to just a bad drive and the extra stress of repopulating with the hot swapped drive should pose little risk.

Also, while not scientific, with regards to the speed of the DroboPro-FS, we integrated the NAS via the Gigabit port on the network. Then we had five people at different machines on the network copy files between 2G-25G in size to and from the DroboPro at the same time. In the end no one complained the system was too slow. When people are not complaining things are too slow, life is good.

Clearly there is no one solution that is going to meet everyone's needs and there is no one system that is going to be 100%. Therefore our goal was to simply reduce the chances for catastrophic failure (unrecoverable data) as much as possible. Cloned DroboPros and swapping all drives before the MTBF point is reached seems to have given us what we thought we were looking for.

Thank you again for mentioning the MTBF as I should have discussed this in my first posting.

David_S
Sr. Bottle Washer
[www.BottledVideo.com]
There is little left for shameless advertising when you are giving it all away for free

David_S Wrote:
-------------------------------------------------------
> Our
> plan is to begin the cycle to replace all the
> drives well before a drive has gone bad or the
> MTBF point has been reached. Further, if one drive
> does fail it was felt that if this occurred well
> before the MTBF point it could be chalked up to
> just a bad drive and the extra stress of
> repopulating with the hot swapped drive should
> pose little risk.

I want you to know that I use RAID5 heavily, so I'm not trying to dissuade you, just trying to inform.

When you buy 5 drives, you often get serial # 1001, 1002, 1003, 1004 and 1005. Maybe 1003 craps out after 6 months. What are the odds, right? But here's the deal. LITERALLY, 1002 and 1004 were RIGHT NEXT TO IT on the production line. So if somebody sneezed on 1003 and that's why it went bad, 1004 and 1002 also got the spray and are next to go.

I understand that they cripple right headlights at the factory, so that it goes bad years before the left one. Because if they put two good ones in the right and left... they'd both blow at (approximately) the same time.

Graeme -
Following your suggestion, I've been looking at the Synology DS1511+ , ..and now my head's hurting since I'm neither an IT guy or network administrator, just an editor!?! So I wonder if you could clear up a couple of questions for me? The unit would potentially be for a small but growing post department (4 - 5 workstations running FCP) within a larger media agency.

1. You are connected to the DiskStation directly via one of the two LAN ethernet ports - right? Does one port = one workstation (ie.maximum of 2)?

2. Are you making use of iSCSI? It's a bit confusing re."virtual servers" etc., but essentially it seems to be a way of connecting to other iSCSI devices (whether on- or off-site) to gain more storage space as overspill. Is this correct?

3. My impression is that although the unit/software is Mac-compatible, it seems more skewed towards Windows users. Is that fair to say, or am I just not understanding the whole open source thing?

4. Last but not least, I couldn't see a price anywhere?!

Thanks in advance for your help in this matter!

Kind Regards,

Rob

I've said this before but if you peruse the REDuser forums there is a guy there from BRU who has some straight talk about archiving ( a big issue in the RED/ALEXA world). He's in favour of his product of course but he has a balanced perspective on what people need to be doing to archive all their media and what's best for one group isn't best for all. The pro's and con's of LTO is discussed and many are going this way. Issues of open source code, obsolescence (hello TARGA!) migration etc have come up in those discussions. It's worth reading if you want to avoid reinventing the wheel or at least learning from other peoples research.

I think a careful delineation of what archive, offline, nearline, online and all the other terms actually mean and which ones you need to concern yourself with is a good place to start.

ak
Sleeplings, AWAKE!

Rob, I'm not using the NAS for video editing, but for backup / storage of all the files I create, and obviously, that's a lot of files, and for an "office share" for working files around the office.

No, I'm not making use of iSCSI, I'm just using it as a traditional NAS. Nice thing though is you have the option to use that free with the package.

The boxes are Mac compatible, but are Linux boxes at heart and hence work well enough with most things.

The two ethernet ports are for either separate networks or redundancy should one go down. With some routers you can either use them for extra speed by bonding them together. I've not got a switch with that feature to try it out though. Because it's a network device, you can have any number of computers attached to the box for sharing. For convenience, most of my computers are wirelessly networked to the box, but some are hard-wired via a gigabit ethernet switch.

For a NSA solution for file storage, they're superb. But if you're looking at an editing solution, I'm not sure what's good on that front - maybe iSCSI would work for that on a hard-wired network and compressed files.

Graeme

Thanks for your swift response Graeme - very useful! (Love your plugins, btw!). I think the option is there to use it for editing, if desired, but confusingly there doesn't seem to be an iSCSI connection as such. DS Manager just facilitates the protocol, I think. Hmmm ....

Sorry Graeme - are you saying that you don't send video files to the NAS? If you're mostly connected wirelessly then I guess you don't, .... right?

Many thanks for your time,

Rob

I've been using the Data Robotics Drobo Pro for a couple of years now, 5 bays being used, 3 empty right now. No matter what I have on the DroboPro, it is backed up on another external drive as well. I haven't had any drives die on me yet but backing up is only common sense. To date, the only problem I have had that really can be annoying is that sometimes the Drobo Pro doesn't mount after turning it on. That can be a pain though the solution is easy enough. Turn it off and disconnect
its power cord, shut down the Mac, reattach the Drobo's cord and reboot. If you click on the 'info' button of the Drobo Dashboard, this issue of sometimes not mounting is the first thing you see so it must be a regular issue that I wish Data Robotics would take care of via some kind of update. I do not run the Drobo or any externals automatically, I turn them on when I need them and I guess there in lies some kind of issue.
Steve

steve-sharksdelight

I store video files on the NAS, I just don't edit to or from the NAS - it's storage only for me.

Graeme

Thanks for info and comments! Drobo Pro does look cool - just not suitable for offsite archiving. Think we'll use it for on-site backup and occasional editing....