archival medium ??

expatCanuck · 08-17-2008, 10:28 AM

Greetings.

Was wondering what, if any, cost-effective archival media exist for digital photo storage.

Anyone know of studies re: reliability stats of CD-ROMs? RW DVDs? ... ?

I know that tape certainly starts to degrade relatively quickly (within a year or three) and should only be considered a short-term backup solution.

Knowledge welcome.

Thanks.

- Richard

Wheatfield · 08-17-2008, 12:09 PM

The only archival storage method for digital files is multiple copies on different media. Redundancy is your friend.
The Canadian Archives tested CD's, and figure that their maximum life is roughly 20 years before there is a very good chance they will be unreadable (no guarantees, they just figure that is their maximul reliable life expectancy) I expect DVD's would have a similar life expectancy, though they may be better. I haven't researched them.
Something else to consider is file and media readability. It's all well and good to have an archical back-up solution for your files, but if you don't have software that will read them, you may as well not have bothered.
So, you probably want to stick with a proven file format if you really are in it for the long haul. For images, you want to use a non proprietary bitmap based file structure. Tiff is likely your safest, followed by PSD. Proprietary camera formats are not good, since support for them can change or be dropped.
JPEGS or other compressed files are not a good bet because if certain data points within the file are lost (headers for example), the file can become unparsable.

If you are going to use optical media, a good method is to burn three copies from the original hard drive directory, use one as a working copy, one as an onsite storage in case the working media pooches, and an offsite storage in the event a disaster takes out your house.
In five years, repeat the process with whatever media is current for the day.
In this example, people would be switching from CD to DVD, and then from DVD to whatever optical media supplants it.

It is important to take your data from old media and put it on whatever is state of the art now. There are a lot of perfectly readable 5 1/4 inch floppies out there with perfectly useless data stored on them because it in almost impossible to find drives. The more modern 3 1/4 inch floppies are going the same way. If you want a floppy drive on a computer now, you have to install it yourself as a cutom. Computers don't come with them any more.
RAID arrays need floppy drives for driver install, BTW, so if you are going to put a RAID into a computer, you may as well get a floppy drive installed as well. It makes it easier to rebuild the array when a drive fails.

The only problem with this is that you really need to keep on top of what you have in archive, which can be a royal PITA (is this OK, mods, it is a well known acronym?). I gave up on trying to juggle an increasingly impossible situation with optical stroage. A few thousand CD's became a few hundred DVDs became (for me) a few hard drives.
Hard drives are guaranteed to fail, it's not a matter of if, it is a matter of when. So redundancy becomes even more important. A 750gb drive holds about as much data as 175 DVDs. The HD takes up far less space and is easier to catalogue, but you are putting a lot of eggs into one basket. This is where RAID or some other redundant solution comes into play.
Educate yourself on the subject of RAID arrays and see if this is a good fit. In a nutshell, a RAID uses multiple hard drives acting as one drive.
Some methods are only good for increasing speed, at the expense of reliability (RAID 0). Others are very, very good for reducing the possibility of data loss by spreading multiple file copies across multiple hard drives. These types of arrays are called mirrored drives, and will have 2 or more drives with identical data written to them. If one drive fails, you replace it and the array will rebuild itself, bringing itself back to full redundancy with no data loss.
The best mirrored drives will have at least three drives, allowing 2 to fail with no data loss. One thing my computer geek technician told me recently is that while RAID is very good, you have to have an OS to retrive files from a RAID.
I didn't think this was a big deal, I don't know enough about computers to run one without an OS installed.

What I have done in my own system is to set up a mirrored RAID in my computer consisting of two drives. My working data is kept there. I also have a seperate unit (a Drobo) which is acting as my long term data storage device. It is only plugged in when I am backing up to it, thereby reducing as much as possible data loss due to lightning strike, virus, or some hacker-child breaking into my system.

Mistakes that you can make with data storage:
Using the "My Documents" directory in a stock Windows installation. This directory is on your C drive, and data will be lost if you find yourself having to format the drive and starting again.
For that matter, no data should be stored on the same drive as the OS. You are asking for trouble by doing this.
Any Windows imbedded program uses the C drive for data storage, as do most other programs that store data for future retrieval.

Put another drive, preferably at least a RAID 1, into your computer and switch your "My Documents" over to it. Do the same with your Outlook Express (if you use it) store folder.

Remember that anything in your computer is subject to loss in the event of a power supply failure. Most computer builds are woefully inadequate in this regard. OTS computers are notorious for having junk power supplies, and a supply that blows can take every component attached to it out of the game with it.
This is where the external storage comes into play. Something that can be untethered from the main computer, and unplugged from the wall is a must for 100% secure protection, and it should have redundancy in it as well.
I like the Drobo because it is idiot proof. Install it's software, plug it in and you're set. This is much easier than some of the RAID solutions I have seen, though they really aren't that hard to deal with either.
I don't like the Drobo because it is a proprietary storage solution. My fear is that if the box fails and Data Robotics goes out of business, I may have non recoverable data.
It's a comprmise I have made, and I will live with the consequenses if things go sideways.

MrA · 08-17-2008, 12:27 PM

Is there an opinion here on renting server file space? I.e. allowing a reputable company (or two) to take responsibility for the data protection, entirely offsite?

Spex · 08-17-2008, 12:57 PM

Making two or three copies on a DVD once a year could take just one hour and few $$ and it's pretty safe method. One copy for using, two for storing. And that's method for those who are phobic about loosing pictures.
Damn, I should do some copy myself...

*Wombat* · 08-27-2008, 03:15 AM

You can buy archival quality DVDs from specialist suppliers which are said to be good for 20-30 years. You might want to check what your nearest large museum or art gallery uses - they're usually on top of this area. As regards allowing someone else to be in charge of my precious photos on their server, however reputable, I wouldn't touch it with a barge pole. Too many options for disaster to my mind. Wheatfield said it - redundancy is the key.

*thirdofthree* · 09-06-2008, 03:43 PM

A relevant post I made on a field audio recording forum in which I participate and on which we've discussed this same issue...

Just a thought for some folks not wanting to mess around with RAID, or wanting to save some money: pick up multiple HDDs and do your own automated (though not real time) redundancy handling. For example:

Get two HDDs (A and B)
Store data on drive A and use drive B as backup
Install and configure DeltaCopy (Windows freeware implementation of Rsync for Unix/Linux/BSD) to copy drive A to drive B
Once set up, the initial run of DeltaCopy will copy all of drive A's contents to drive B
Future runs - which users may schedule to run automatically at the interval of their choice and which I personally run nightly - will then only update new files, those that have changed, and optionally those deleted.

In my experience, this approach:

Is easier for non-geeks than RAID
Saves money (quality RAID controllers are not inexpensive)
Reduces risk from unrecoverable RAID implementations; I've found inexpensive / "consumer" level RAID controllers unreliable and in the past have had 2 different controllers fail on me and render the array unrecoverable
Provides adequate frequency of redundancy - my critical data doesn't change frequently or sufficiently enough on a real-time basis to warrant real-time redundancy; Besides, my workflow prevents lack of real-time redundancy from posing a problem: until the new data on drive A is backed up to drive B, I still have the data on an SD card or optical media

My overall strategy, including the above - I have my data:

Stored primarily on Drive A
Redundant and easily recoverable on Drive B
Redundant, but recoverable with significant hassle, on optical media
Redundant, but not quite up-to-date off-site (really only a concern for me in the event of a home-impacting disaster, like fire)

The above approach addresses Wheatfield's key point: a single redundant / backup copy is insufficient. The general rule of thumb: data's not safe unless it's stored in at least 3 places, preferably one of which is off site.

I know this doesn't address the OP specific questions, but the reality is that no single archive medium is sufficient.