Forgot Password
Pentax Camera Forums Home
 

Reply
Show Printable Version Search this Thread
07-25-2010, 01:05 AM   #1
Forum Member




Join Date: Dec 2009
Photos: Gallery
Posts: 94
digital archiving & storage strategy

Dear all,

How should we organize the long term storage of our digital photography?

Neanderthalers from the film era like me were quite spoiled regarding archiving & storage: I still have my 30 years old negative films in perfect condition.

To me, archiving & storage poses a problem of a new magnitude in the digital age of photography for any kind of users, from P&S "sunday" amateurs, serious photographers, to professionals.

Strangely, I do not seem to find a lot of material regarding this subject in the usual forums.

Of course, we all would like nice criteria such as:
- long term resilience
- immediate availability
- searchability
- independance from external providers

In a first step I set up a straightforward procedure consisting of:

0. Naming
Each folder is named after date & thema, allowing easy searches.
1. copy all files to a portable USB hard disk
Limited to hard disk capacity (160 Gb). High risk due to failure possibility. Long term resilience low (magnetic signals "evaporate" in time). Half-time of disk formats limited in time.
2. make a second copy on DVDs, 4.7 Gb at a time
Good reputation of optical storage is contradicted by personal experience and variability of material (high "luck" factor). Dependability depends on burning speed (choose the lowest possible speed).

To me this is only a short-term solution depending on too many technological factors to be serious. However it will allow me to wait for a better long-term plan.

Now, what do you think? What is your procedure? Thank you very much for your kind input.

daniel


Last edited by danielausparis; 07-25-2010 at 04:04 AM.
07-25-2010, 02:56 AM   #2
Pentaxian
ChatMechant's Avatar

Join Date: Jun 2010
Location: Santa Cruz Mountains
Photos: Gallery
Posts: 1,003
Redundancy. Also, redundancy.
I have two mirrored hard drives in my desktop, and another one elswhere, which gets files dumped in every so often. HDD's keep getting cheaper and bigger, and seem fairly reliable. You can even get drives that come with free data recovery in the event of a malfunction.
Due to my complacency, I haven't burned a backup disc for a while, but probably should. Blu-ray looks promising- 25-50GB per disc, and scratch-resistant coating.
Once, a while after getting my first digital camera, i deleted ALL of my photos, on accident. like 60gb worth. i kid you not. luckily, super-girlfriend had backed them up on dvd, unbeknown to me. plenty paranoid now!
07-25-2010, 04:43 AM   #3
Forum Member




Join Date: Dec 2009
Photos: Gallery
Posts: 94
Original Poster
You're correct, Chat Méchant: newer technologies give us some more space. However does that solve all concerns? Like: how long will last blu-ray discs? How reliable are they? Do we have any serious figures on these questions?
When the CD came out, vendors pretended that they would last 'nearly forever'. We all know today that it was commercial junk.

Also, there are some additional questions that I kept aside in my first post:
- are our current image formats (DNG, JPEG, TIFF etc.) futureproof? Just an example to illustrate: how many of us would be able to retrieve flawlessly an MS Word document from 10 years ago? Given that DNG and JPEG depend on private companies subject to disappear/change strategies/etc. any time does not reassure me. Don't forget that even General Motors nearly collapsed not long ago.
- are our current storage technologies (DVD, blu-ray) futureproof? e.g. will it be possible, in 20-30 years from now, possible to retrieve files from current storage?

Or will we be in charge of a lifelong burden of periodical archive maintenance (renewal of formats and/or storage technology)?

It seems that I'm even much more paranoid than you :-)
07-25-2010, 08:24 AM   #4
Pentaxian




Join Date: Apr 2007
Location: Oklahoma USA
Posts: 1,626
What you have now with digital is, in almost every way, superior to film.

Unless you're throwing a lot of money at the problem, your film is deteriorating fast. If you are throwing a lot of money at it, it's deteriorating slowly. It can be lost; it can be destroyed. You have only one copy, unless you shot multiple identical originals and are storing them separately.

With digital you do have to constantly keep up with formats, and test and rewrite data periodically. Removable optical media using current technology isn't useful. External hard drives are probably the most effective technology at the moment, and probably have about as much reliable life, on average, as optical media.

Online backup would be excellent, but many of us have bandwidth such that it would take months (not an exaggeration) to upload our files to a server. In the future that may change. So for now, I use 4 (5, starting next week) hard drives with one copy each to store my files, located in two or three different physical locations. You definitely don't want to rely on just 1 or 2, or even 3, hard drives.

Paul

07-25-2010, 09:40 AM   #5
Forum Member




Join Date: Dec 2009
Photos: Gallery
Posts: 94
Original Poster
Paul, congrats, now it looks as if you are now the most paranoid :-)

Seriously, thank you for your input, and I'm with you regarding the subject. Just one thing regarding online storage (the "cloud") even if everybody gets high bandwidth: I wouldn't trust one second in the long term (i.e. 20 years) the reliability of a private company file storage - too big are chances that this company disappears, or changes strategy, or destroys your files by accident, or whatever.
Just look at the usage contracts you sign with such storage providers: always, they is a non-liability clause that puts usage risk to you exclusively. Even Microsoft has exactly such clauses in their Windows/Word/etc. contracts.

And thank you for putting clearly what I suspected and feared: our photographic passion transforms us all into storage system engineers constantly managing copies, file formats and storage media, at high monetary and time cost.
07-25-2010, 10:50 AM   #6
Pentaxian




Join Date: Apr 2007
Location: Oklahoma USA
Posts: 1,626
QuoteOriginally posted by danielausparis Quote
Paul, congrats, now it looks as if you are now the most paranoid :-)

Seriously, thank you for your input, and I'm with you regarding the subject. Just one thing regarding online storage (the "cloud") even if everybody gets high bandwidth: I wouldn't trust one second in the long term (i.e. 20 years) the reliability of a private company file storage - too big are chances that this company disappears, or changes strategy, or destroys your files by accident, or whatever.
Just look at the usage contracts you sign with such storage providers: always, they is a non-liability clause that puts usage risk to you exclusively. Even Microsoft has exactly such clauses in their Windows/Word/etc. contracts.

And thank you for putting clearly what I suspected and feared: our photographic passion transforms us all into storage system engineers constantly managing copies, file formats and storage media, at high monetary and time cost.
Absolutely you do have to keep up with storage technologies and file formats. I'd feel good about using storage companies for one copy of my data, but would probably keep a local copy too.

The cost of multiple copies is reasonable now if you have a modest space requirements. I have about 250gbytes so far, but it grows as I get opportunities to scan more film from the old days, and of course as I take more pictures. Two copies are just on my desktops, which have enough space, so no extra cost is involved. The three usb drives I use, one desktop-sized and two portables, cost about $70 each, and are only about 50% full.

Paul
07-25-2010, 10:59 AM   #7
Veteran Member




Join Date: Dec 2009
Location: Deep Forest
Posts: 630
What everybody already wrote -- multiple redundancy on reliable devices, i.e. hard drives, not flash cards, cd/dvd media, online, etc..

Also incredibly important is some form of data integrity checking such as crc; you don't want to discover years from now you've been copying corrupt data from old devices onto newer...
07-25-2010, 03:47 PM   #8
Veteran Member
Marc Sabatella's Avatar

Join Date: Sep 2006
Location: Denver, CO
Photos: Gallery
Posts: 10,686
"The DAM Book" by Peter Krogh is the absolutely definitive reference on such matters. I'd highly recommend reading it cover to cover before you even begin to think about planning out your own strategy.

07-25-2010, 06:15 PM   #9
Veteran Member
Derridale's Avatar

Join Date: Jan 2008
Location: Brisbane, Australia
Posts: 412
Let me just add my vote for redundancy x4...!

I store on the local HD on the computer, plus I have two RAID5 NAS boxes and I automatically store all RAW files on those as well. I use Aperture, so each time I complete a project of tweaks and fine-tuning, I export to both NAS boxes also.

Given that those NAS boxes are onsite, and could theoretically get stolen or the house burns down, I also have anothe NAS box in a fireproof safe in the garage which is about 30 metres away from the house, connected via ethernet in a conduit which is buried. Everything gets stored there too.

Then, just to confirm my paranoia, I also use online backup, using Dropbox and also my domain host's servers. These, in turn, as part of the package deal I have with them, automatically back up onto Amazon's S3 servers. Yes, it took almost a week of constant uploading to transfer everything for the first time, but now it happens overnight and transfers only the new stuff or altered stuff.

I don't burn anything to CD or DVD - those formats keep changing and won't be around forever (who can read 8" floppies now? Or ANY floppies, for that matter?) I also don't use proprietary disk formats like the old Zip/Jaz drive type of thing, for the same reasons.

That's about as far as I take it - paranoid enough I think....

So, to summarize: storage is on local computer HD, two NAS RAID5 boxes on site, another NAS RAID5 box in a fireproof safe in a separate building, and two online services. If that lot fails, then I'll simply switch to knitting as a hobby...!

PS: Marc - I've had "The DAM Book" on order for some time now, and Amazon are still giving a delivery date of sometime in September. Just in case others may be looking for it - it seems to be unavailable at the moment.

PPS: Just checked the Amazon page for the book - it is giving an estimated delivery as "1-3 months" from date of order.

That's for the second edition of the book, published in 2009.

Last edited by Derridale; 07-25-2010 at 07:55 PM.
07-25-2010, 08:59 PM   #10
Site Supporter
Site Supporter




Join Date: Sep 2006
Location: Kaunas
Posts: 1,456
QuoteOriginally posted by danielausparis Quote
Or will we be in charge of a lifelong burden of periodical archive maintenance (renewal of formats and/or storage technology)?
This is exactly what is going to happen.
07-25-2010, 09:28 PM   #11
Pentaxian
Wheatfield's Avatar

Join Date: Apr 2008
Location: The wheatfields of Canada
Photos: Albums
Posts: 11,249
Hard drives. Lots of hard drives.
And read that DAM book.
And get Lightroom and learn about keywords if you haven't already.
07-25-2010, 10:14 PM   #12
Pentaxian
SpecialK's Avatar

Join Date: Dec 2006
Location: So California
Photos: Gallery
Posts: 15,622
I back up PEFs and converted Jpgs to an external HD, and periodically make 2 sets of DVDs and keep one at work. I keep a copy of the jpgs on the computer for ready access.

However, with the decreasing cost of external HDs, I may just use those and skip the DVD part which, frankly, is slow and cumbersome, and has already filled a 3-ring binder...
07-25-2010, 11:02 PM   #13
Forum Member




Join Date: Dec 2009
Photos: Gallery
Posts: 94
Original Poster
Thank you all very much - my initial approach was pretty naive, I realize.

The DAM book looks interesting! I'll try to get it by "other means"... I had a look on french archiving intelligence and found some recommendations of the "National Archiving Institute" (INA). Main findings are:
- stay away from proprietary data formats (this is a severe point that, strictly seen, would eliminate DNG and JPEG!)
- optical media is not eliminated per se, however it is considered being subject to aging and random faults
- magnetic media (disks) is recognized as at least equally resilient as optical
- rotation frequency should be 5 to 10 years, any media

As a side note, I am quite surprised that for a leisure as popular and economically significant as photography, the subject of long term archiving still relies on private expertise and "cooking receipts": no simple, cheap and reliable solution is offered by the industry.
07-25-2010, 11:21 PM   #14
Veteran Member
Derridale's Avatar

Join Date: Jan 2008
Location: Brisbane, Australia
Posts: 412
I think by "proprietary formats", they would mean more the physical format, such as the Iomega Zip or Jaz drives, I refered to above.

I think PEF files are more proprietary than DNG files (Adobe's attempt to provide a NON-proprietary, cross-brand format), and JPGs are certainly not proprietary.

For what it's worth, I store all my RAW files in DNG format - the format that I shoot on my K20D bodies.
07-26-2010, 12:07 AM   #15
Forum Member




Join Date: Dec 2009
Photos: Gallery
Posts: 94
Original Poster
QuoteOriginally posted by Derridale Quote
I think PEF files are more proprietary than DNG files (Adobe's attempt to provide a NON-proprietary, cross-brand format), and JPGs are certainly not proprietary.
wikipedia saves the day, as usual :-) and yes, my mistake: JPEG is an ISO/IEC standard, so that would be a green light :-) (I got confused by GIF, which is indeed proprietary). DNG and TIFF are Adobe formats "ununcumbered by patent claims" (as for today).
Reply

Bookmarks
  • Submit Thread to Facebook Facebook
  • Submit Thread to Twitter Twitter
  • Submit Thread to Digg Digg
Tags - Make this thread easier to find by adding keywords to it!
copy, disk, gb, photography, photoshop, procedure, resilience, storage, term, time
Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Library / Archiving software expatCanuck Digital Processing, Software, and Printing 2 02-21-2010 11:19 AM
Portable Digital Storage Xcom Pentax Camera and Field Accessories 0 05-06-2009 05:03 PM
For Sale - Sold: Multiple Auto (Digital & Film) and Manual Lenses, flash + more jjdgti Sold Items 18 10-03-2008 09:30 AM
For Sale - Sold: Shepherd Polaris Digital Flash & Ambient Light Meter Dr Windmill Sold Items 3 07-09-2008 07:10 PM
Folder organization/archiving joefru Pentax DSLR Discussion 47 05-04-2008 06:41 PM



All times are GMT -7. The time now is 01:40 PM. | See also: NikonForums.com, CanonForums.com part of our network of photo forums!
  • Red (Default)
  • Green
  • Gray
  • Dark
  • Dark Yellow
  • Dark Blue
  • Old Red
  • Old Green
  • Old Gray
  • Dial-Up Style
Hello! It's great to see you back on the forum! Have you considered joining the community?
register
Creating a FREE ACCOUNT takes under a minute, removes ads, and lets you post! [Dismiss]
Top