[DavidSKAF3]

I seek advice regarding the architecture of a more advanced system for storing and backing up my growing collection of images.


At present I use a HP desktop PC with a 1 TB HDD, and a Western Digital external drive that backs up everything on my PC automatically. I just plug it into the PC USB port every so often and it does it thing. The external drive has already saved me twice when my earlier PCs crashed. But the external HDD's capacity is quickly nearing full! One thing I learned is it was making five backup copies of every file. I adjusted it to make three, and that opened up more capacity. Do I even need three on one drive?


I also use Lightroom and Photoshop. I'm stating to wonder if I buy more external drives or some other device, how will Lightroom find stored images I am looking for? How do I configure LR to read any and all additional devices? I guess am not interested in using the cloud.


Right now the PC is up to 800 GB with programs, documents, music, and pictures. The My Pictures folder alone is 300 GB.


I used to store images on data CDs and disks. Some of them became unreadable over time, and I've lost the images forever, as far as I know.


Then I graduated to the external HDD. And it looks like I'm due for an even more advanced storage system now.

Any advice is helpful. I want to design a secure storage and archival system that is fast and easy to access at any time and with an eye on even greater storage needs, of course, as time goes by. And I'd like to keep the cost as modest as possible.


Thank you,


David

---

Last edited by DavidSKAF3; 04-15-2015 at 07:56 AM.

[Hank]

I use a Synology DS+1513 NAS with a full 20 TB (5x4TB) of disk installed (after RAID, 14TB usable). $1500 from B&H.
It provides a good storage / backup solution for my needs. It is fast (I can burn a DVD from the NAS faster than from my own HD).
It is expandable, both with larger HDs and external expansion units (> 60TB max space with 2 expansion units).
It is very user friendly, and I'm very happy with it.

[dakight]

At a minimum you want something with redundancy, ie. Mirror (RAID 1) or Striping with parity (RAID 5). That way if a single drive fails you can still recover your files. It does no good to move all your files to an external device if you have no backup or fault tolerance provision. The drives in an external device are the same as the ones in the PC and they are subject to failure. Even RAID is not foolproof; in the event of physical destruction your data is still lost. Only regular and comprehensive backups to a device in a physically separate facility will ensure that you can withstand such an event.


How many copies do you need on one device? That depends on what you're trying to insure against. If it's accidental deletion then the original plus one copy should be sufficient. If you're concerned about hardware failure, if you keep the original and copies on the same device and that device fails you lose both unless you have a fault tolerant system and/or comprehensive backup


Anything is better than nothing and nothing is 100% foolproof. Determine your storage needs and your budget then see where they meet.

[08amczb]

I'am using a DLink DNS-320L NAS whith 2x3TB HDD in RAID 1. It mirrors all my files, so I can recover if one disk fails. I protect the NAS by an UPS (overvoltage protection and uniterrupted power). The NAS is acessable on the whole home network as windows network drive (SMB protocol). It is much cheaper then the Synology, but it provides less space. 3TB is far enough for me and for my wife.

[narual]

1) Delete any flat out bad shots on import (or skip importing them). Ditto for any video you've taken but don't need anymore (dunno about you, but I snap random videos with my DSLR now and then, but only once was it for actual use outside "I want to show this to someone later", and they take up way too much space to keep).
2) Rate photos. When you need space (assuming you didn't do it when you rated them) you can always wipe out the ones you'll never use.
3) Check for duplicates and redundant near-duplicate photos. Delete those, too. Teekesselchen: Home is a lightroom plugin to help find and flag duplicates. It works pretty well. You have to watch out for action sequences when you shot more than 1 photo per second, but it's not too bad about catching those

That will reduce the sheer mass of photos.

Then,
4) Use smart previews in Lightroom so it doesn't *look* like anything's missing when you're offline
5) Put all your photos you aren't likely to need soon on a network drive in your home (I archive by quarter now, sort of. I have everything from January 1st til now on my laptop and the rest is on a network drive. Sometime between May and July I'll probably push January through March off to the network drive.). That way you only have to carry around a small subset of the images, but with smart previews, lightroom will still let you do edits to them as if they're there, and you can even export them in web-quality sizes.

Beyond that, you should be backing up to the cloud. There are lots of options there, both photo specific and for all your files.
Don't use carbonite, though. Unless they've changed their policies, it will take almost a year to back up your photos (last I looked, they restrict upload speed to about 1 GB per day)
This looks like a decent overview of available options.
Backing Up Photos to the Cloud / 2014

it doesn't mention amazon prime photo, which might have come out after the article was written - that offers free photo storage to Prime customers. The interface is terrible, though.
https://www.amazon.com/clouddrive/primephotos

[Kunzite]

For backups I'm using a HDD dock. It has an on/off switch (so I only start the drives when I need them), USB 3 interface (USB 2 is way too slow), support for larger drives (many docks can't see drives larger than 2TB) and it has 2 disk slots (less insertions/removals, can copy from a drive to another).
A NAS sounds like a nice idea if you can afford a good one, but that's a storage solution - you still need to think about backup.

[Kerrowdown]

Do I even need three on one drive?

I would suggest at least two copies plus one original, but one on each of three different drives... you lose that one drive you've lost all copies no matter how many are on there.

You need to build in some redundancy... which you don't currently have.

[Caver]

+1 For any multi-disc RAID capable Synology NAS. I use (for work, daily) two older models, since many years, without any problems!

[RGlasel]

external drive that backs up everything on my PC automatically

Good, that's the best (and in my opinion, the only) thing you should do. Pros: Far more reliable than flash memory or online services; vastly faster than online backup; preserves directory structure and file properties (which is significant metadata in itself); doesn't become useless if you forget a password, use a different computer or have a crash with either the computer or network that transferred the files to backup storage; relies on physical, not virtual, security; is less susceptible to software failures; recovery is intuitive, not being dependent on you being on hand or remembering how to run the recovery, because you need to get your assistant to find something while you are thousands of miles away and no one remembers how to find the necessary documentation. Cons: every external HD comes with a free crippled back-up utility that nullifies any advantage instrinsic to using an external HD for backup. Use Windows Backup, Time Machine or any commercial program that you like; and never, ever change backup programs without doing a full backup with the old program first. If you need to expand capacity, buy another external HD. The time you save compared to pruning your backups (or uploading 500 Gb of RAW files) has to be worth something, and you should only need to recover files when disaster strikes, which by definition, can't be planned for. By the way, for your own personal use, there is no need to keep more than 1 copy of anything on the same device. The odds of individual files going rogue without you noticing before the entire system fails is too close to zero to make a difference in anyone's life. Rotate your backups to different devices and keep those backup devices in different locations for the ultimate in redundancy.

If you recover files from backups on a regular basis, you are doing something wrong. Network storage (online or local) should only be used for files that need to be accessed frequently, from multiple devices while keeping up to date with all revisions from any source. Simple, local and self-administered backups always trump other solutions.

---------- Post added 04-15-15 at 01:14 PM ----------

At a minimum you want something with redundancy, ie. Mirror (RAID 1) or Striping with parity (RAID 5).

RAID for personal backups is false redundancy and isn't a viable backup solution. If the controller fails for whatever reason, you are hooped. It's not just the card failing, an OS crash can make files unrecoverable as well. Using RAID for backup is putting all control in the hands of lowest cost hardware and software developed by people who bear no financial risk for software bugs. RAID only provides protection for HD failure; with the relatively low speeds and duty cycles of consumer HDs, it is far more likely the OS or motherboard fails before a HD does. I can't put this strongly enough; if you use RAID for backing up your files, you are unwittingly taking on extra risk for zero benefit. RAID only makes sense when you have HD failures on a regular basis, or need faster concurrent data access in a multi-user environment through striping.

[Kath]

I seek advice regarding the architecture of a more advanced system for storing and backing up my growing collection of images.


At present I use a HP desktop PC with a 1 TB HDD, and a Western Digital external drive that backs up everything on my PC automatically. I just plug it into the PC USB port every so often and it does it thing. The external drive has already saved me twice when my earlier PCs crashed. But the external HDD's capacity is quickly nearing full! One thing I learned is it was making five backup copies of every file. I adjusted it to make three, and that opened up more capacity. Do I even need three on one drive?


I also use Lightroom and Photoshop. I'm stating to wonder if I buy more external drives or some other device, how will Lightroom find stored images I am looking for? How do I configure LR to read any and all additional devices? I guess am not interested in using the cloud.


Right now the PC is up to 800 GB with programs, documents, music, and pictures. The My Pictures folder alone is 300 GB.


I used to store images on data CDs and disks. Some of them became unreadable over time, and I've lost the images forever, as far as I know.


Then I graduated to the external HDD. And it looks like I'm due for an even more advanced storage system now.

Any advice is helpful. I want to design a secure storage and archival system that is fast and easy to access at any time and with an eye on even greater storage needs, of course, as time goes by. And I'd like to keep the cost as modest as possible.


Thank you,


David

I'm in a similar situation, coming close to maxing out my working GBs as well as the backup ones. Having looked at all the expensive options, I've decided to continue to backup often to a handheld multi-TB drive, move everything older than a certain date to it, and just have sequential ones as they fill up. I think the suggestion above to make smart previews in LR is a good one because you continue to have access without having to search through your backups.

Though you indicated no interest in cloud backup, I would encourage you to consider the Amazon cloud, free with prime membership. It's unlimited storage and I use it for redundant backup for the entire collection, very useful should disaster strike my hard drives. I have it organized the same as Lightroom, so I can identify by date categories where I want to look. Uploading is easy, especially if you keep up with it.

[bdery]

I'am using a DLink DNS-320L NAS whith 2x3TB HDD in RAID 1. It mirrors all my files, so I can recover if one disk fails. I protect the NAS by an UPS (overvoltage protection and uniterrupted power). The NAS is acessable on the whole home network as windows network drive (SMB protocol). It is much cheaper then the Synology, but it provides less space. 3TB is far enough for me and for my wife.

I do exactly that. Plus I have an external drive that I use to manually backup the files from time to time, and store it in a safe (in case of fire).

Plus I upload all my "good" images to Smugmug, so they are also backed up in the cloud. My Smugmug account is cheap, it lets me build a website, and it has unlimited storage.

[mattb123]

I'm an ex-IT guy and I'm still the IT guy for my family. So I need to ensure my PC and two others are backed up.

So I put together an old small form factor PC from Dell that I traded a coworker a bottle of booze for. It was too slow to be her PC anymore so I upgraded the RAM from 2 to 6 GB and installed a new 1 TB drive for Windows Server Essentials 2012. Server Essentials replaced what was called Windows Home Server which as the name suggests is a version of Windows server designed for the home.
I added an external hard drive dock with 2 bays and put two 4TB drives in to hold all the computer backups.
After you install a client application on the computers, they get automatically backed up every night. It's smart about the backups too so once the initial one is done the differential backups go very quickly. It can wake computers up from sleep to back them up at night.
My backup (those two 4TB drives) is not on a redundant volume (like RAID) because by nature it's already a second copy of the data because it is a backup.

It was easy to set up and I think most non-IT folks could do it pretty easily.
One really nice feature is that if I lost a drive in a client PC I can make a USB drive that I can boot from and restore the PC to a new drive. It's nice for drive upgrades too.

For off-site, I just bring an old hard drive to work with me and try to remember to update it every now and then.

[MadMathMind]

If you recover files from backups on a regular basis, you are doing something wrong. Network storage (online or local) should only be used for files that need to be accessed frequently, from multiple devices while keeping up to date with all revisions from any source. Simple, local and self-administered backups always trump other solutions.


RAID for personal backups is false redundancy and isn't a viable backup solution. If the controller fails for whatever reason, you are hooped. It's not just the card failing, an OS crash can make files unrecoverable as well. Using RAID for backup is putting all control in the hands of lowest cost hardware and software developed by people who bear no financial risk for software bugs. RAID only provides protection for HD failure; with the relatively low speeds and duty cycles of consumer HDs, it is far more likely the OS or motherboard fails before a HD does. I can't put this strongly enough; if you use RAID for backing up your files, you are unwittingly taking on extra risk for zero benefit. RAID only makes sense when you have HD failures on a regular basis, or need faster concurrent data access in a multi-user environment through striping.

This is my assessment as well. Things:
1) Change the backup method to incremental, if it's not there already.
2) You don't need 4 months of old backups. Every now and then, go in and clear out the old ones. Often, the backup software will do this for you.
3) RAID isn't needed for backup drives. If both your main and backup fail simultaneously, something catastrophic happened and anything else you did probably would have failed too.

Acronis TrueImage is the software I use. It's quite good and has saved me when I accidentally overwrote a file once or twice.


If 1TB isn't enough, then get another drive. You can set good backup software to write as much as possible to one drive and when that fills, it will go to another.

[lightbox]

RAID for personal backups is false redundancy and isn't a viable backup solution. If the controller fails for whatever reason, you are hooped. It's not just the card failing, an OS crash can make files unrecoverable as well. Using RAID for backup is putting all control in the hands of lowest cost hardware and software developed by people who bear no financial risk for software bugs. RAID only provides protection for HD failure; with the relatively low speeds and duty cycles of consumer HDs, it is far more likely the OS or motherboard fails before a HD does. I can't put this strongly enough; if you use RAID for backing up your files, you are unwittingly taking on extra risk for zero benefit. RAID only makes sense when you have HD failures on a regular basis, or need faster concurrent data access in a multi-user environment through striping.

There is tremendous truth in this. Consider also that a RAID system may even propagate a data error further (an incorrect bit written to multiple locations), especially if there has been a write error or a corrupted bit ("bit rot") in one location which is then used as a reference to "scrub" another location. You now have two copies of a corrupted file and lost all the good ones. I believe that even the NAS systems mentioned here are susceptible to this.

There are RAID-based systems that have protections against this with the inclusion of things like ECC RAM and advanced filesystems such as ZFS and Btrfs, but now you're talking serious $$$ for something that you don't really need.

A couple of external USB disks might be a good idea. Even better if you can have the backup software write to them separately so that you're not just copying your backup onto a second drive (in case there was a write error the first time). At least that way you have another (hopefully uncorrupted) copy at an off-site location somewhere.

[PDL]

Just one or two things.
RAID is for fault tolerance and performance. If you need large continuous areas for storage you use RAID systems to span spindles and provide a cushion for hardware failure. RAID is not repeat NOT for backup. If you delete something off of a RAIDed storage it is just as deleted as if there were one drive.

External Hard Drives will fail just as quickly as internal HD's. Relying on a external HD sounds good, but they can be faulty if left alone for long periods of time.

Connections will age out. Just find a mother board that has a parallel port (remember ZIP drives? they were suggested as "Backup" in years past). Remember CD/DVD/BluRay? Some manufactures do not supply optical drives on their consumer products anymore (Apple for one, MS surface devices too.) We are at a change point on USB also, the new USB type C is coming. Apple devices ship with USB type C ports, how many years before the non-type-C ports disappear?

There are two types of Backup, Disaster Recovery and Archive.
Disaster Recovery is a way of keeping your data ready in case of massive hardware failure, theft of your device and things along those lines. Most of the backup solutions offered by HD manufactures, OS providers, third party SW etc. is Disaster Recovery. Even so called cloud backups are for all intents and purposes is DR.
Archive backups are where you back the data up to media that is stored off site and in a format/media that is not the same as the source. In my experience Archive is either optical (not CD/DVD/BluRay - but glass) and magnetic tape. Don't laugh tape is still here and is a big business - just ask Iron Mountain. For archive you pay, usually through the nose, for long term contracts.

To put my comments in context. I am a retired IT person with 25 years of experience with PC's and having worked for those 25 years in global corporations. My last paying gig was to manage 5 PetaBytes of disk spread across 56 servers providing DR services for 100K devices across the Continental US and a small site in Moscow. I know backups, hardware and storage solutions.

My personal method of image backup (data only, no metadata except that which I can embed using ExifTool basically GPS and LensID) is to copy the RAW files to another PC on my network and my 1 disk NAS. On my main box and the "other" device (a PC) are running RAID 1 disks for Fault Tollerance. So I have the RAW images in three places = three devices. For Archives I burn DVD's on a periodic basis. Not the best solution at all, since all three devices and my DVD storage is in the same room. (Man Cave).

I distrust External drives due to my experience with them at the corporate level. Each time I stored information to a external drive and stored it for later use (a year or two) the external drives failed to spin up when attached to, usually, new hardware. Digital images are extremely fragile, once the hardware fails for the most part they are gone, just as gone as a fire in the film vault.

The Elitist - formerly known as PDL