Is it safe to backup data to a hard drive and then leave it for a number of years?
Assuming the file system format can still be read, is this a safe thing to do. Or is it better to continually rewrite the data (every 6 months or so) to make sure it remains valid?
Or is this a stupid question?
-
I want to say probably (as long as you keep it away from magnets ;-), but I'm not sure. For long-term storage I would transfer the data to some archival format like a DVD - I think that like CDs, they're supposed to last 100 years. You can still keep the HD around for easy access, of course, but the DVD gives you a reliable backup.
Jeremy Huiskamp : Just curious, but can you point to studies that demonstrate the long term reliability of optical media? Are there certain materials/brands that last longer than others? I have heard (but not experienced) that regular consumer-writable optical disks can become unreadable in a matter of years.Martin C. : DVDs (especially the ones you use for burning yourself) have lower life-expecations as HDDs.patjbs : Most CD/DVD lifetime estimates I've seen point to more around 10-15 years, and that's in optimal conditions. Non-optimal storage of optical storage media tends to decrease their lifetime dramatically. And I've seen this reflected in my observations.Eddie : Backing up a 500GB drive to DVD is pretty painful. Optical media may or may not last as long as a hard drive, depending on many factors.duffbeer703 : There are "archival grade" writable optical media, but the actual lifespan is questionable AND require very specific environmental conditions.From David Zaslavsky -
I wouldn't trust important backups to any single device for any significant length of time.
I've had plenty of CDs that couldn't be read after a while. (Cheap ones, admittedly, but I'm leary of the longevity claims made.)
I've had hard disks silently corrupt data.
I seem to remember I've even had SSD failures, although with a low number of writes I'd expect them to be pretty reliable.
Aside from all of these things, using a single copy means you've got no protection against physical disasters: fire etc. If you have multiple copies, you can separate them physically. Ideally I'd take some number (e.g. 3) of copies and run a checksum (I usually use MD5) periodically over everything. If one of the copies becomes corrupt in some way, if you've got multiple other copies you should be able to trust the majority, and create a new backup to replace the corrupted one. (Of course, if you keep the correct checksums in a separate place, you could trust even a single backup which still gives the right checksums, as the canonical source for replacements.)
Of course, how much trouble you go to depends on the value of the data. My personal home data is only backed up on a RAIDed NAS. My work data is in Google datacenters, which I trust fairly strongly :)
From Jon Skeet -
I would say you should recycle the media every other year or so - that is, replacing the drive, disc or tape with whatever there is to replace it with and keep more than one copy.
Few things lasts forever, optical media can degrade rapidly depending on quality, method of writing to it and environment where it's stored. Mechanical parts can always fail or there could be bugs in the firmware that is related to time or to wear and tear.
I've pondered over your question often, it would be convenient with something that is guaranteed to stay working for say 5 years. There's tapes and other form sof backup media rated for 10 or more years but I'd never trust that, at least not without a decent amount of redundancy (several copies on different batches).
Keeping the data fresh and continually recycled seems to be the reliable way to go - that way you get to test it regularly as well.
From Oskar Duveborn -
From the article
What advice do you have for long-term storage of disk drives and other media?
Keep your hard drives in a climate controlled environment within an acceptable temperature and humidity range. Also, protect the drives from electrostatic discharge (ESD) and vibration -- this is normally done in their packaging, but it's important to prevent ESD, physical shock and excessive vibration when the drives are removed for storage.
All magnetic storage media has a finite life because magnetic fields start to decay as soon as they are written. This means a tape or drive will not retain its data forever. In a proper storage environment, it's reasonable to expect that the drive should remain readable for up to 10 years.
The concern is more about the drive's mechanical reliability; will it physically spin up? After a very long period of disuse, the spindle bearings or head actuator may be stiff, resulting in read/write errors. These considerations are particularly important for long-term archival storage systems, as well as the new class of removable hard-disk drives that are now appearing from ProStor Systems Inc., Imation Corp., Quantum Corp. and Iomega Corp.
From splattne -
HDDs have actually quite high life-expectations, at least from the magnetic side (setting external magnetic fields asside). The main problem with them is, they could eventually suffer mechanically, i.e. not spin up if they are not used regularely, as some oils and coplings could become a problem.
The safest approaches to really long-time storage in my opinion are:
- stream to one or more magnetic tapes
- print to paper and/or micro-film
- keep copies on operating (running) HDDs distributed over several physical machines and locations
- use an additional external backup space like Amazon S3
Optical media, especially the ones available for consumer use have unexpected low life-expectancy. You should at least check the quality of the raw data read every two years. You could have lost data in the meantime, though.
EDIT: An important aspect in this case is also that you should add checksums to the stored files (MD5, SHA1, etc.), so you'd be able to realize that some corruption occured (or not).
From Martin C. -
They will store data safely for a few years but you would be better to copy them every two years or so yes.
From Chopper3 -
Given your other options for backup, HDD is the safest way to go. Other options include Magnetic Tape, SSD and optical media.
Let's examine the pitfalls of each:
MT: More prone to erasure when exposed to a magnetic field than a HDD. Readers are also becoming harder and harder to find. You don't want to come back in 5 years and find that there's no way to remove the data from your medium.
SSD: Reliable in that there are no moving parts. They are prone to electrical degradation after several read/write cycles which is troublesome and potentially dangerous. The likelihood of losing data while the drive is not in use is slim, however.
Optical Media: The least reliable of the bunch. They're extremely prone to physical degradation (bending/warping) and it requires very little to throw them out of their deflection spec. Further, the encoding scheme used to write data to most optical media is rather complex, creating a greater likelihood of single element failure leading to unreadability.
HDDs: Solid, sealed devices. Can be damaged by physical shock more easily than most of the above devices. Has precise mechanical parts that can lead to failed read/writes if damaged.
The benefit of HDDs, however, is that they ARE sealed. All of the moving parts are stored in an air-filtered enclosure. The magnetic stability of the bits on the disk is quite high and unlikely to change.
Further, if the mechanical parts fail, it is possible to have the platters removed and the data recovered from them directly.
There's no perfect option, but of the imperfect ones, HDD would probably be your best bet.
From ParoX -
I've had hdds fail while "in storage", i.e. sitting in a climate-controlled room for a few years that, when called into duty again, refused to spin up or be booted from.
So no, I wouldn't say that this is a particularly good idea. As others have said, as part of a blunderbuss strategy it is one way of keeping a copy of your data, but it probably shouldn't be your only one.
From Lunatik -
One additional suggestion is that you should also always move data to the current format technology as soon as the previous technology is nearing its end of life. For example, currently I'd suggest moving away from IDE drives, as computers are starting to ship without IDE connectors and controllers.
Similarly in the audio/video archiving we have moved from Cassette (VHS) to CD (or LaserDisc) to DVD, and now to flash storage.
You might keep a USB-to-IDE adapter around, but along with regularly recycling your data between storage devices and locations, you should also keep in mind moving the data to current technology so as not to wait 10 years to find out you can no longer access the data on that 5.25" floppy disk.
From Steve Tranby -
If you want your data to survive or any period of time:
- Use tape if access is infrequent. Follow environmental guidelines and do the homework to determine how often you need to rotate the media.
- Use disk if you need access to the data. The disks should be "active". A disk in a closet is likely to either fail or get thrown out a few years down the road.
Using a third party provider is another alternative. Something like Amazon S3, Mozzy or a similar service gives you an ultra-low cost way to store stuff.
From duffbeer703 -
Do not store your hard drives for any length of time. They are designed to be on. If you don't let the HD spin up every now and then, they will go bad. I'm talking months or a year here.
They will break if not used. MTBF is "guaranteed" for drives in use, not in storage.
From Thomas -
It sounds like you're not so much concerned about hardware failures, but file corruption and bit rot. In this case, ZFS is your best ally. If data preservation is your goal, consider using RAIDZ2 if you can afford it, or at least RAIDZ1. RAIDZ is comparable to RAID5, except it uses a variable stripe width to eliminate the infamous RAID5 write hole. This is especially useful with a cheap NAS, because power failures likely won't corrupt the array. The file corruption and bit rot are taken care of by real-time disk scrubbing, in which the data is constantly being checksummed to verify it's accurate. Those are just the tip of the iceberg with how ZFS is THE filesystem of choice.
If you want an easy NAS setup at home with ZFS built in, check out http://freenas.org. The latest release candidate includes ZFS, and it's not that hard to set up.
It will be interesting to see the long term results of switching to ZFS simply for data preservation... it's too new at the moment. However, the facts are all there, and it's a no-brainer: the best file system for data integrity is ZFS.
From churnd -
Blu-ray seems to be nice solution for this problem.
From m1k4
