OT: Who's using Blu-ray for archiving?

Posted by John K 
OT: Who's using Blu-ray for archiving?
June 29, 2010 05:16PM
Time to clean off the RAID and archive a few large projects. Was about to buy another stack of bare SATA drives but then I read Larry Jordan's warning about keeping them on the shelf for long periods of time (loss of magnetic signal). I can't afford an expensive tape backup system right now wondering if any of you are trusting your data to BR optical media...

JK

_______________________________________
SCQT! Self-contained QuickTime ? pass it on!
Re: OT: Who's using Blu-ray for archiving?
June 29, 2010 05:25PM
I don't consider BluRay to be a feasible archival format, because the storage per disc is too small. I know of a house that used to archive to DVDs a long time ago, and frankly, when I saw the stack of DVDs... Let's just say that if you had to search through that pile to find a project, it's a daunting prospect. Although BluRay stores more than 5 times the data of a DVD, you need to consider that you're moving into a tapeless world, and more storage is required as a result. So it is still not practical.

If you can't afford an LTO, I'd say go on RAID 1s.



www.strypesinpost.com
Re: OT: Who's using Blu-ray for archiving?
June 29, 2010 06:56PM
Since i do my shooting on Sony XDCAM EX at 35Mbs i use Blu-Ray BD-R 25 GB who cost about $4 each and using memory card SxS 16 GB , this fit very well my workflow.

I use also BD-RE of 25 GB and 50 GB to archive projects for testing final project too , it is all depend about your workflow and budget plus tolerance to risk since Blu-Ray is so new for many of us.

Hard-drives archiving except the expensises raid types are not 100 % reliable either and you must spin them from time to time for better lifetime archiving.

It is funny to have become tapeless and hear tape is the best archiving option even if it is true.

Now if only the big camera makers of tapeless have promote affortable tapeless archiving system it will be heaven but for now the closest thing to tape and affortable is XDCAM HD optical drive.

PRODUCTION D. GOYETTE
TOURNAGE / MONTAGE VIDÉO CORPORATIF
DISTRIBUTION DVD / BLU-RAY / WEB
[www.productiondgoyette.com]
Re: OT: Who's using Blu-ray for archiving?
June 29, 2010 07:02PM
Just a reminder: Nobody has the foggiest idea what the shelf life of a recordable Blu Ray disc is. For that matter, nobody really knows what the shelf life of DVD-R media is either. Testing by NIST and other groups has been all over the map, with accelerated aging studies coming up with numbers of anywhere from five to two hundred years.

I wouldn't trust optical media for any data I really, totally, definitely, for sure wanted to preserve indefinitely.

Re: OT: Who's using Blu-ray for archiving?
June 29, 2010 07:07PM
>Since i do my shooting on Sony XDCAM EX at 35Mbs i use Blu-Ray BD-R 25 GB

About half a year ago, I may have agreed. Thing is, I know of a place that shoots largely on the Sony EX cameras, they opted for BluRay as an archival format, and after a few months of hectic production, they have some 3-4 TBs of footage sitting on hard drives, most of them XDCAM EX card archives. Although the data on each SxS card may fit onto a BluRay, it is going to take a long time to burn all that onto discs. If you do it religiously after every shoot, and you're able to make it work, more power to you.



www.strypesinpost.com
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 12:08AM
After a few Final Cut Montreal meeting and discussions the following can be said and appear to be the best answers for the moment.

1-Tape base formats are robust and suggested for long term storage but will it be a player to read it in a few or many years.

2-Optical media cant get corrupted but long term survival is difficult to estimate so may makers with various quality but again will it be a reader for those media in a few years.

3-Hard-drives cant be beat for storage and speed but need babysitting and duplications or expensive raid for data recovery in case of crash.

But what the speaker said at an FCP Montreal meeting was ; even if all the above is working the explosion of Codec and Operating System and NLE you may hit a wall anyway.

Dont want to be too negative but our tapeless archiving solutions still in lab for the moment let hope the solutions will appear before some important archives are lost.

PRODUCTION D. GOYETTE
TOURNAGE / MONTAGE VIDÉO CORPORATIF
DISTRIBUTION DVD / BLU-RAY / WEB
[www.productiondgoyette.com]
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 01:49PM
Just to add to the points.

Optical media, and solid state drives are not considered archival formats, because in the case of optical media, storage per disc is too low. For solid state drives, the price for storage is too high. This is not taking into consideration whether these formats can preserve the integrity of your data for a substantial amount of time (eg. 20 years). Granted, even if the data is intact, you may not have the means to play them, as digital codecs may become obsolete.

Although the price of traditional hard drives may have fallen, they are simply not reliable enough, partly because they contain too many moving parts, data can become corrupt, and the hard drive may fail beyond a certain time span. We tend to assume the lifespan of a single hard drive to be between 3-5 years, and mainly because of this, whether it exists in a RAID or as a solitary drive, it cannot be considered an archival format.

An LTO deck may be pricier (too pricey for some of us), but if you have a substantial amount of data to archive (eg. >20 TBs), LTOs actually is the cheaper and more reliable option.



www.strypesinpost.com
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 02:05PM
It is the nature of modern video archiving that a) nothing is for certain and b) materials need to be updated constantly. It's no longer as simple as outputting to a Beta or DigiBeta tape and that's it. And frankly, these days, we'd probably look back on the Betacam days as an extremely lossy, insufficient format for archiving.

I remember talking to an engineer about compression and he talked about how the major studios go crazy with trying to archive their classic films. The technology changes every three years and suddenly they have a whole catalogue of films that need to be remastered to the newest, best-looking format.

Digital technology changes rapidly, but formats still stick around. H.264, after all, is now seven years old. And MPEG-1 has been around for something like 22 years. The problem, of course, is that digital formats have a hard ceiling in terms of quality. You can rescan a good film print 40 years later and it will yield new visual information to take advantage of superior technology; an MPEG-1 will not look any better if you reconverted it to a 24K hyper-HD frame size -- and will in fact look a lot worse.

As for long-term endurance, the nature of any tapeless medium is that the archives are going to have to be checked, tested and possibly transferred regularly. A drive sitting on a shelf for six months is not guaranteed to survive. And none of these new file-based archival storage media has been around long enough for us to know whether they'll really last 20 years, or 100, or just two. Plus, digital information is more likely to degrade in ways that render the whole thing unsalvageable. If you get a part of an analog video cassette warped, demagnetized or dirty, you can rescue the rest. If a digital file is corrupted, very often that's it.

Imagine the headache of maintaining a large archive of P2, R3D, or AVCHD raw media...not only would the drives need to survive, but to ensure the files are still truly intact, you might even have to run them through the conversion process periodically just to make sure the file structure is still sound, that the video and audio are as they were, and so on.


www.derekmok.com
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 02:50PM
>You can rescan a good film print 40 years later

The counter argument to that, is there are quite a number of people who have U-matic, S-VHS or Hi-8 tapes without a functioning deck. In a few good years, we can add Beta-SP to the list. Last year, I had a friend who asked me if he knew if anyone still had a DAT player so he could recapture the audio from an old project. And I didn't know who still keeps one of those machines. Film never really had competing technology for a very long time. Film editing on the other hand, has been taken over by NLEs such as FCP and Avid, so we know a flatbed editor isn't worth as much these days.

In world of audio recording, the old analog 1 inch open reel multi track recorders (eg. the Ampex from the 70s), has largely been replaced by digital open reel formats, and today, the DAW. Those machines are expensive, and used to cost between 5-6 digits. Storing audio recordings on those formats, isn't a pain free process either. Every few months, those analog tapes had to be loaded back on the open reel machine, fully re-wound to prevent the magnetic audio information from printing over.

Regarding digital formats, we talk about Mpeg 1, but that was a widely used end user format, and to some degree, I still occasionally send that out today. There are digital formats that we hardly hear of today- cinepak, sorenson.. How about the proprietary digital formats like microcosm. Red Giant acquired Microcosm from Digital Anarchy about two years ago, and today, it has still not been re-released. Will these formats survive for the next 20 years? And will you be able to run those files on OS 10.9 Turkish Angora?

The bottom line is that you never know what is going to happen, but if I was to make an educated choice, and I had some 3TBs of material to archive, short of making crayon drawings on wood, I would go for the LTO, and if that's not in the budget, I would get it archived onto a RAID 1 drive, and run systematic checks every 6 months. On a smaller scale, it would be optical media and hard drive.


>If a digital file is corrupted, very often that's it.

It depends on what kind of corruption.

[www.aeroquartet.com]

Heck, these days, Jon Chappell can get a corrupted FCP project up and running.

You have Disk Warrior, Data Rescue, etc... They aren't fool proof, because you need to know what kind of corruption occurred. Even film prints get mouldy if they are not stored properly.

I know of a company that had a D-beta player (Sony J30), and there was one day, one of the staff tried playing a digital beta tape from the archives, and the tape snapped, and the player had to be sent for servicing. From memory, the cost of servicing the player, was in the region of US$5K.



www.strypesinpost.com
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 04:49PM
Great discussion guys, thanks for the info. Strypes and Derek, are you guys both using LTO for archiving now?

JK

_______________________________________
SCQT! Self-contained QuickTime ? pass it on!
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 04:57PM
In some places that I work at, they use LTO. And that's what I always recommend if there is a substantial amount of material that will be archived.

For me, I don't have the $5K to invest into an archival system, and by the time I archive enough material to make an LTO deck more viable, we will probably be seeing an LTO-9 system. So I use RAID 1 external drives. And everything else is backed up, including my own system drive.



www.strypesinpost.com
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 05:03PM
strypes Wrote:
-------------------------------------------------------
>So I use RAID 1 external drives. And everything else is backed up, including my own system drive.


And those Raid-1s, do you just turn them on every few weeks so that the drives aren't languishing?

JK
Re: OT: Who's using Blu-ray for archiving?
June 30, 2010 05:06PM
Every now and then when I feel like checking what's on the drives, which usually happens every few months. I don't use a lot of storage, because most of what I cut are in various production houses.



www.strypesinpost.com
Re: OT: Who's using Blu-ray for archiving?
July 01, 2010 02:18AM
Many years ago (mid '90's) I was involved in high end audio restoration for a company in Burbank. The issue for us was that, with 2 audio-only restoration rooms going 16-24 hours a day, and through-putting dozens of feature length projects every month, local single mechanism tape based solutions, one attached to each system, weren't cutting it. Added to this was the reality that the most effective, safe & redundant workflow for this work (which included the most classic of classic films & TV shows) meant that for each & every project, from beginning to end - anywhere from a few days to several weeks - we'd end up with 5-9 iterations of the film on our drives. Audio doesn't take up a lot of storage, but each film we worked on was taking up to 9x it's native storage requirement. In other words, a 90 minute film would consume up to 810 minutes worth of storage on our drives. Multiply this by dozens of ongoing projects and pretty soon our locally attached Exabyte SCSI tape drives were blowing smoke.

In 1995 we went to NAB to investigate large tape-based archival systems. IBM showed us a huge system that was basically the size of a small room, an automated system. I think it was a half million dollar system. It fit the bill but the cost of media was huge, on the order of hundreds of thousands of dollars (for tape alone) to populate the system. The deal killer for us, however, was IBM's migration plan. Every 2 years, we'd have to invest in the next generation of tape, at a further cost of hundreds of thousands of dollars, for migration. They demonstrated how the system could be continuously archiving newly gathered material off the connected DAWs, while at the same time migrating old tapes to new tapes. Very slick and it had to be constantly manned.

We didn't buy the system. The financial guys decided that the cost benefit didn't make sense. What we started to do instead was backup data to other hard drives. Some things never change. This is essentially what I'm doing now for video & audio work. Constantly moving data around to multiple drives, exercising drives often, and using Wiebetech's RTX enclosures so I can use bare drives for the constant flow of data.
Re: OT: Who's using Blu-ray for archiving?
July 01, 2010 02:53AM
I already have some bare drives laying around.... maybe it's time for a Drobo. spinning smiley sticking its tongue out

_______________________________________
SCQT! Self-contained QuickTime ? pass it on!
Re: OT: Who's using Blu-ray for archiving?
July 01, 2010 10:31AM
>The deal killer for us, however, was IBM's migration plan. Every 2 years, we'd have to invest in
>the next generation of tape

If you're really obsessive about it, I would say upgrade every 2-3 generations, so compatibility is maintained.



www.strypesinpost.com
Re: OT: Who's using Blu-ray for archiving?
July 06, 2010 04:03PM
A 'magic number' I keep seeing when reading up on archiving in the digital world is to migrate to a new medium every 5 years so you don't end up w/the problems of no longer having the proper hardware/software to read your archive. Recently I was doing some spring cleaning and uncovered about a dozen ZIP disks. I really, really, really doubt there is anything worthwhile on them as I tended to burn everything onto CD or DVD but I have no way of finding out for sure unless I stumble across a ZIP drive that works.

Keeping that in mind I'm not as worried about my current archival medium being able to last 20 or 30 years as much as I am worried about the cost of perpetually migrating my archive. LTO seems to be the exception because it is time and industry tested, but even then you'll have to migrate eventually. Film still has a feather in it's cap because the machinery needed to view it is so basic.


-Andrew
Sorry, only registered users may post in this forum.

Click here to login

 


Google
  Web lafcpug.org

Web Hosting by HermosawaveHermosawave Internet


Recycle computers and electronics