[Digitalis]

Hard drive failure is a rarity? What world do you live in?

I was referring to the drive electronics, Hard drive failure is almost invariably a result of physical components degrading to the point where it fails. I have rarely seen an electronic circuit fail. I have seen motors fail, heads crash from warped platters, platters scratched from micro-imperfections in the platter surface layer...so many failures due to hardware, but seldom do I see any drives taken out of commission due to the electronics.

a recovery service or someone handy with Linux may be able to recover your data ($$).

Fortunately it is quite easy to learn how to use Linux these days, I recommend it to anyone. Df,Isblk,Mdadm,Fsck are your friends.

[BarryE]

One note that most never think about is that some external backup drives are internally encrypted and even if they are a RAID design, without the enclosure (which has the encryption chip in it) your data is inaccessible. I have a WD 4TB RAID box like this. If something in the enclosure croaks, you cannot get to your data. Many smaller external USB drives also are not normal drives and have the USB circuits built-in. If something goes wrong with the USB part of the drive, you cannot just pop it out and get your data off of it.

As Ray says this is so true. Oddly WD and others don't mention this in their sales literature. They do have a process of recovery within the warranty process, but as I recall it requires a special new enclosure to be sent out. Don't believe RAID is foolproof...

---------- Post added 01-01-20 at 10:37 AM ----------

I was referring to the drive electronics, Hard drive failure is almost invariably a result of physical components degrading to the point where it fails. I have rarely seen an electronic circuit fail. I have seen motors fail, heads crash from warped platters, platters scratched from micro-imperfections in the platter surface layer...so many failures due to hardware, but seldom do I see any drives taken out of commission due to the electronics.



Fortunately it is quite easy to learn how to use Linux these days, I recommend it to anyone. Df,Isblk,Mdadm,Fsck are your friends.

Capacitors fail quite easily ...

[Digitalis]

I trust my own online storage system - mainly because I designed it, maintain it, and own it. I would have difficulty storing images or customer data on someone else's systems without having some sort of insurance against data loss and a comprehensive data recovery and security policy. I have looked up a number of services - Most of which are US based, not many local Australian data storage providers that cater to everyday consumers. I'm not sure what things are like in Europe or Asia.

Capacitors fail quite easily

Last hard drive controller board I looked at when I was working on my NAS yesterday featured 0 capacitors*. Complaining about caps is a bit of a weak argument - since there are places where a cap can fail that would be far more damaging to a system - such as in the CPU power regulators on the motherboard? With some skill in soldering, caps are rather bog standard devices and are easily replaced.

*though I'm not sure how consumer drives are constructed but there probably aren't any on those controller boards either.

[kaseki]

Likely capacitor failures are due to not designing for the actual ripple current the capacitor has to handle (Heathkit power amplifier power supply comes to long-term memory), or using capacitors incorrectly fabricated (for example, Chinese knock-offs of Japanese capacitors based on an incomplete fabrication process formula stolen from the Japanese). The latter was the cause of failures of at least one motherboard maker in the early 2000s, and became an advertising virtue for DFI selling its "no Chinese capacitors" motherboards back in the mid-2000s. This comes to mind because I was just into a 2005 DFI based PC of mine (where the battery needed replacement). No damage was visible on any electrolytic cap, unlike that of an Abit motherboard I once owned where many caps had barfed their contents onto the motherboard surface after 6 years.

Given that a NAS has an OS and a power supply that surely uses capacitors, no matter how rugged the drive controller, capacitor failures affecting the OS can lead to data corruption. In my case, the Abit randomly caused corruption of the HD stored OS (with several different HDs, and even when the HD was actually a hardware RAID 1 chassis) over a period of a few years before I became clued in to the capacitor problem.

Conversely, my Synology RS814 NAS has been running 24/7 for about 5 years using WD "Y" server drives without any issue. Similar hard drives have been operated 24/7 since 2010 in the PC I'm using to type this.

My old age inspired take-aways: You generally won't get more than you pay for. The cost of quality is modest when including one's time value for dealing with lower quality device failures.

[Kerrowdown]

Do you trust your online storage?

Not as much as multiple back ups on multiple hard disks, at least fifty percent of them held at another off site location... just call me paranoid.

[hypermodern]

Shortly I will be migrating computers and will be adding a NAS into my local mix so I still don't trust online fully.

I really like my little NAS setup. Took a while to get it working but now it auto-backs up my laptops I'm very happy with it. As you say, still need something off site.

[photoptimist]

If I were to pick a cloud service, it would be one that does not offer any "free" storage to anyone.

Flickr has the problem that they have too many leaches -- less than 1% of users pay. That means that each paying customer must cover the equipment, personnel, electricity, bandwidth, and service costs of themselves plus 99 other "free" customers. The fewer the paying customers, the more each paying customer must pay. But the more they charge each paying customer, the fewer paying customers they have. It's a recipe for a death spiral!

[Ray Pulley]

I was referring to the drive electronics, Hard drive failure is almost invariably a result of physical components degrading to the point where it fails. I have rarely seen an electronic circuit fail. I have seen motors fail, heads crash from warped platters, platters scratched from micro-imperfections in the platter surface layer...so many failures due to hardware, but seldom do I see any drives taken out of commission due to the electronics.

Yes, the mechanical parts of the drives are typically the main reason for hard drive failures, yet many that own WD RAID enclosures have reported failures of the non-drive control electronics, not drive failures.

Fortunately it is quite easy to learn how to use Linux these days, I recommend it to anyone. Df,Isblk,Mdadm,Fsck are your friends.

Your average photographer isn't going to learn Linux, and should not have to. It seems to me to be a poor design decision to sell a product as a backup tool where the failure of a non-drive part leads to unrecoverable data from a perfectly good set of RAID drives.

To get back to the main subject, you have made a good argument for redundant backup methods so one does not have to resort to a crash course in Linux just to restore files from a backup system.

[WorksAsIntended]

If you are going to learn any of this, just don't. MDADM is outdated technology that survives by its bare market share from old days. Next generation Filesystems with checksuming and build in raiding are the way to go these days. ZFS, maybe BTRFS or even Ceph are a whole different beast when it comes down to keeping a decent availabilty to your storage.

Data security can never be achieved by a one dimensional security layer like raid.

Than you have to ask yourself if you even need availabitly. If not, better invest in backups than in raiding.

I am in the luxery of owning a company building and running storage clusters, so for me it is very easy to find goog places to store my stuff.

[Digitalis]

At the risk of further veering off topic..

It seems to me to be a poor design decision to sell a product as a backup tool where the failure of a non-drive part leads to unrecoverable data from a perfectly good set of RAID drives.

Then don't buy WD desktop USB drives! There are a number of manufacturers such as Verbatim, Seagate, LaCie, Sandisk, Qnap and Samsung make Thunderbolt/USB-C DAS Hard drive chassis that are said to have good reliability and also support hardware RAID, all you need to do is bring your drives to the party.

If you are going to learn any of this, just don't. MDADM is outdated technology that survives by its bare market share from old days.

As someone who works in the IT industry you of all people should know, being flat out avoidant about different OS and the tools to recover partitions and data is the opposite of being helpful.

Next generation Filesystems with checksuming and build in raiding are the way to go these days. ZFS, maybe BTRFS or even Ceph are a whole different beast when it comes down to keeping a decent availabilty to your storage.

Unfortunately a lot of these new filesystems have been slow to be implemented due to stability, development time and support costs which also prevent these filing systems from being released in consumer products.

Data security can never be achieved by a one dimensional security layer like raid.

If someone relies purely on RAID to secure their data - they are doing it wrong.

Than you have to ask yourself if you even need availabitly. If not, better invest in backups than in raiding.

Conveniently, RAID 1 does precisely that. perfect mirror images of the host drive, if one drive fails simply replace it and re-sync. Providing there are enough duplicate drives...say an array of 4 drives in a RAID 1 reduce the chances of catastrophic data loss from a single drive drastically.

[Pablo Villegas]

If I were to pick a cloud service, it would be one that does not offer any "free" storage to anyone.

Flickr has the problem that they have too many leaches -- less than 1% of users pay. That means that each paying customer must cover the equipment, personnel, electricity, bandwidth, and service costs of themselves plus 99 other "free" customers. The fewer the paying customers, the more each paying customer must pay. But the more they charge each paying customer, the fewer paying customers they have. It's a recipe for a death spiral!

Yeah, Flickr is losing money because it’s a social network more than a photographer service. But that way my pictures can be seen by thousands instead of a few people on a private web site. I use it for my personal photos and to have a back up fo some, very few professional work.

[WorksAsIntended]

At the risk of further veering off topic..




Then don't buy WD desktop USB drives! There are a number of manufacturers such as Verbatim, Seagate, LaCie, Sandisk, Qnap and Samsung make Thunderbolt/USB-C DAS Hard drive chassis that are said to have good reliability and also support hardware RAID, all you need to do is bring your drives to the party.

Although this is a trend these days, there is no reason to dodge wd. Statistics show that they are just as reliable as other brands.

As someone who works in the IT industry you of all people should know, being flat out avoidant about different OS and the tools to recover partitions and data is the opposite of being helpful.

Choosing the correct technology for a specific task is one of the most important things in it, may it be hardware, a dev framework, an OS or a data replication system. Choosing mdadm is like choosing a manual 6×45 camera to shoot thr olympics. It simply fits the purpose that badly it is the wrong choice, although this has been different in the past.
If someone already owns this solution I am glad to help him out, but giving advce to get this solution or to learn it to use it in futire is just bad advice

Unfortunately a lot of these new filesystems have been slow to be implemented due to stability, development time and support costs which also prevent these filing systems from being released in consumer products.

Nothing of this stays true these days. There are plenty fully developed open source solutions available now.

If someone relies purely on RAID to secure their data - they are doing it wrong.


Conveniently, RAID 1 does precisely that. perfect mirror images of the host drive, if one drive fails simply replace it and re-sync. Providing there are enough duplicate drives...say an array of 4 drives in a RAID 1 reduce the chances of catastrophic data loss from a single drive drastically.

About 30% of data loss are due to harddrive failures. So you miss 70% of reasons by uaing any raid in first place.
Raid 1 has no data integrity check at all. If you got broken data on any drive it will be replicated into the new one. In about 20% of cases of a single broken disk in a 3 drive raid 1 there are broken files after recreating redundancy. Raid 1 wears all drives by identical pattern, incresing the chance of mutiple drive failure during raid recovery.
Raid recovery copies smashes data without even noticing, this might even be just some bad inoding which will result in a bad fs and possible loss of information.
The idea that Raid1 provides mutiple copies if the data is simply wrong as any change on data are done simultaniesly on all drives. A faulty ram stick will immidiatly kill your data.

Look at peojects like FreeNas if you dont feel confident setting up next gen fs your own. They provide zfs out of box with checksumming and desaster recovery without spare drives. This enables to check for silwnt data corruption and whipe it out as well as it recognizes bad sectors during recreation of redundancy and chooses another copy instead of copying the faulty one. Still it covers only 30% of data loss, but those at least in 98% instead of 80% of the cases.

[Digitalis]

Raid 1 has no data integrity check at all.

Fortunately most NAS devices have filesystem integrity checking capability of their own, Additionally the NAS devices I have experience with are capable of scanning unused space looking for faults and flagging the offending sectors. Though stand alone RAID enclosures lack these capabilities.. Buyer Beware.

Statistics show that they are just as reliable as other brands

Their drives are good, no one here is disputing that. Bear in mind there are people here who have other problems with WD products:

yet many that own WD RAID enclosures have reported failures of the non-drive control electronics, not drive failures.

There are plenty fully developed open source solutions available now.

And exactly how many of these alternate filesystems have been 100% supported by the manufacturer in a consumer level hardware data storage product? I know of some enterprise solutions that use ZFS straight out of the box, never heard of anything like that in consumer markets.

[WorksAsIntended]

Filesystem integrity != data integrity.
As a lot of those consumer market splutions are based on FreeNas a lot use zfs.

[MossyRocks]

And exactly how many of these alternate filesystems have been 100% supported by the manufacturer in a consumer level hardware data storage product? I know of some enterprise solutions that use ZFS straight out of the box, never heard of anything like that in consumer markets.

Look at FreeNAS.It has ZFS out of the box, runs on commodity hardware or if you want you can support them and buy their hardware. Any modern journaling file system has good file system integrity but that is nothing like having a raid setup with redundancy or redundancy from a properly setup ZFS pool where you are protected against a drive failure. As far as flagging bad sectors and writing elsewhere I know that has been done since at least FAT16 in the old MS DOS days (Windows 3.1 era is when I used it and dealt with it last) and probably even farther back than that so it isn't like it is exactly something new.