Part 1: Backups at Home
RAID & hard drives - Back it up!
People often ask me what my backup strategy is so I thought I would outline what I am doing with the many images I shoot these days. In the above photo on the very right, you can see a LaCie 2 TB RAID drive, set for the faster RAID 0 (striped) as opposed to the more secure RAID 1 (mirrored). I wanted the best performance possible for storing my Lightroom image library which is why it is set that way. You can also see a couple of WiebeTech drive enclosures to the left, and there is one more WiebeTech behind the speaker on the left that you cannot see.
In the foreground is a smaller enclosure with a 128GB SSD drive that I use on the road with my MacBook Air as a shared drive. My current working Lightroom image library and image files are on the SSD so I can work on stuff while away from home and then just plug the drive in at home and seamlessly resume where I left off without having to deal with library syncing etc.
This is how these drives are used:
2TB RAID --> working image and video files.
2TB External --> mirror backup for 2TB RAID.
1TB External --> mirror backup for iMac internal 1TB HD.
1TB External --> partitioned, containing a dedicated 150GB Photoshop "scratch" partition, a 128GB backup partition for the small SSD drive and two other partitions with miscellaneous non-critical files or files already backed up to optical media.
I use a program called SuperDuper to do automated nightly mirrored backups at 1:30am, after which the computer and hard drives automatically shut down. SuperDuper can do a "smart" clone, so only changed files are updated, making the process much faster of course. The scratch disk is an empty non-fragmented partition that Photoshop can use, which will not interfere with OS X's virtual memory access on the iMac's internal drive.
When I download images from a memory card, I use Adobe Lightroom and take advantage of its ability to not only download to the image library, but also to simultaneously create a backup copy on an external drive. So right away after downloading my cards, worst case, I already have my images on two hard drives. The first working set is either in my iMac's internal drive or on the 128GB SSD, and then a second set is on the external RAID. Then at 1:30am, those two sets get mirrored to two more drives, finally giving me copies on 4 different hard drives.
Can you tell I don't trust hard drives? I have seen far too many hard drives fail over the years to ever trust a single backup of my images, once even when a new drive was only a couple of weeks old! While I have never personally had a drive fail completely on me, my dad has, and he lost the better part of several years worth of images since he did not regularly backup to optical media like I suggested he do. I have had plenty of customers with failed hard drives over the years while working at Beau Photo as well. Since my dad's drive failure, I also have set him up with an automated hard drive backup, so hopefully that won't happen to him again! Actually, now that I think of it, I did have a hard drive fail in the old PowerMac G5 at work quite a few years ago now, but luckily nothing of consequence was lost, although I did have a lengthy re-install of OS X and all the applications and drivers to contend with.
Anyway, continuing with my backup procedures, once I have 25GB (or more) of raw downloaded images accumulated, the Lightroom "backup copy" versions, I will burn those to a 25GB single-layer Blu-Ray disc (BD-R) and once verified, will erase them off the hard drive. Realistically, I generally wait until I have 100GB plus of backup images and then will go and burn multiple BD-R discs in one sitting. After the backups are burned to BD-R, once my nightly backup kicks in, I'll have my working set of images on two hard drives and a backup set on Blu-Ray. Then, after a year or so has gone by, I will eventually rotate my working Lightroom images out onto a second set of Blu-Ray media, at which point I'll either delete the originals off my hard drives completely, or move the images onto a non-backed-up hard drive for continued convenient access.
Before burning the Lightroom working images to Blu-Ray, DVD-R, or whatever other medium you choose, you need to be sure to highlight all the images you are going to burn in Lightroom's grid view, then choose "Save Metadata to Files" from the Metadata menu, or just hit Command-S. This will ensure that all your Lightroom edits and IPTC info are stored along with the image files as .XMP sidecar files, so if you need to re-import those burned images at a later date, all the relevant info will be available to Lightroom again, or to Adobe Camera Raw if you were to just open a single image in Photoshop. A more detailed Lightroom tutorial is beyond the scope of this article, but hopefully that is enough to help you with the backup question.
I transitioned to using 25GB Blu-Ray optical media about a year ago, since using those "small" 4.7GB DVDs was a pain! Writeable BD-R discs have come way down in price, as have the burners, so they are now a viable alternative to DVD media, as far as being relatively affordable for the drives and the cost-per-GB. As of this article's writing, buying spindles of 10 or more BD-R discs (Verbatim in this example) will bring the price down to well under $4 per 25GB disc and a good LG 10x Blu-Ray internal burner is now only $160. That compares to $15 per disc and nearly $500 for the burner back when I bought mine!
One thing though, is that I have been able to find very little information on projected media longevity, so I would strongly suggest always backing up to two sets of discs, one working set and one that is kept in archival storage, ideally offsite. BD-R discs have very high data density and while all the good ones have hard, scratch-resistent coatings, if you do manage to scratch one, it is far more likely to affect data than a similar sized scratch on a DVD-R or CD-R disc. As far as overall longevity in archival storage, my gut feeling is that BD-R discs will certainly last long enough to be transitioned to whatever the next standard long-term backup media turns out to be.
What do I mean my "transition"? Well think about it: even though the discs might last 50+ years in storage (some gold DVD-R and CD-R discs are rated for 200+ years), will anyone be able to still read a BD-R disc 30 years into the future? With digital storage, long term preservation is a matter of continually copying files from old, obsolete media onto new, while there is still hardware around to read that old media! However that is a topic for a future blog entry, but I think you get the general idea...
One last point of backup drives is to not trust power supplies either! Say what? Well, let's presume you have a desktop tower, like a Mac Pro. It has four internal drive bays, so the easy solution for backup, is to have multiple internal hard drives set up either with RAID 1 (mirrored) or manually cloned using a program like SuperDuper. However what if there is a catastrophic power supply failure which fries all the drives at once! Unlikely? Maybe, but it actually happened to my dad, which is how he lost his drive mentioned earlier in the article. A power supply malfunction fried his hard drive, his internal CD burner and the motherboard on his computer. If he had multiple internal drives for backup, all of them would likely have suffered the same fate. Maybe a costly data-recovery could recover files, but having external hard drives as backup means that they would not be affected by a failure like that, which he unfortunately did not have. Of course a lightning strike or severe power company surge might kill everything at once, multiple drives or not, but hopefully a really good UPS/surge-protector will minimize that danger. You are using one, right? Anyway, that is one reason why I get stuff onto optical media as fast as is practical.
I have been somewhat lucky I suppose. Many years ago, I was not as paranoid about backing things up, but "got smart" before ever suffering a catastrophic data loss. There are too many people out there complacent about their data and don't implement a good backup strategy until after they experience a drive failure.
So, to summarize my at-home backup strategy, memory cards get downloaded simultaneously to two hard drives and those two drives get cloned to two other backup drives. At this point my images are on four drives. Eventually images from the initial "download backup" get transferred onto BD-R at which point they exist on one set of optical media and two hard drives. Lastly, my working set of images eventually gets stored onto a second set of BD-R media, so I have everything on two sets of optical media.
Once I transition to new media, I plan to keep all the old media for as long as they can still be read. For example, a few years ago I transitioned all my CD-R discs to DVD-R. I plan on doing the same thing with my DVD-R discs in a few years time as well, depending on what the next upcoming long-term standard turns out to be. Maybe it will be Blu-Ray, maybe not...
By the way, when you buy optical media, for Pete's sake make sure you are buying quality discs! When I did my CD-R transition a few years ago, I was able to read every single disc I ever burned, hundreds of them, stretching all the way back to the first one from 1993. I only used quality Kodak or Fuji CD-R discs over the years. I have seen "no-name" spindles of cheap CD-R discs fail only a matter of months after burning, even though they were kept in archival storage. The writeable layer started delaminating from the discs rendering them unusable! Surely your images are worth storing on good media?
At present I am either using Fuji or Verbatim Blu-Ray discs. In the absence of any concrete test data (or long-term experience), I'm really not sure if there are significant differences in the quality of Blu-Ray media. I became an early adopter of Blu-Ray, back when it was still substantially more expensive than it is now, it part to be a guinea pig for testing this new optical media. So far so good: my earliest discs are still 100% readable. Do keep in mind, however, that since most people do not yet have Blu-Ray drives in their computers, when it comes to sharing files, DVDs or CDs are still the way to go.
So that's it for this long blog entry, explaining my at-home backup strategy in detail. Knock on wood, but so far I have never lost a single scan or digital image in all the years I've been working with digital photography! Stay tuned for Part 2 in this series, coming soon, where I will cover my on-the-road backup strategy when shooting stills and video with my digital cameras while traveling...
In the foreground is a smaller enclosure with a 128GB SSD drive that I use on the road with my MacBook Air as a shared drive. My current working Lightroom image library and image files are on the SSD so I can work on stuff while away from home and then just plug the drive in at home and seamlessly resume where I left off without having to deal with library syncing etc.
This is how these drives are used:
2TB RAID --> working image and video files.
2TB External --> mirror backup for 2TB RAID.
1TB External --> mirror backup for iMac internal 1TB HD.
1TB External --> partitioned, containing a dedicated 150GB Photoshop "scratch" partition, a 128GB backup partition for the small SSD drive and two other partitions with miscellaneous non-critical files or files already backed up to optical media.
I use a program called SuperDuper to do automated nightly mirrored backups at 1:30am, after which the computer and hard drives automatically shut down. SuperDuper can do a "smart" clone, so only changed files are updated, making the process much faster of course. The scratch disk is an empty non-fragmented partition that Photoshop can use, which will not interfere with OS X's virtual memory access on the iMac's internal drive.
When I download images from a memory card, I use Adobe Lightroom and take advantage of its ability to not only download to the image library, but also to simultaneously create a backup copy on an external drive. So right away after downloading my cards, worst case, I already have my images on two hard drives. The first working set is either in my iMac's internal drive or on the 128GB SSD, and then a second set is on the external RAID. Then at 1:30am, those two sets get mirrored to two more drives, finally giving me copies on 4 different hard drives.
Can you tell I don't trust hard drives? I have seen far too many hard drives fail over the years to ever trust a single backup of my images, once even when a new drive was only a couple of weeks old! While I have never personally had a drive fail completely on me, my dad has, and he lost the better part of several years worth of images since he did not regularly backup to optical media like I suggested he do. I have had plenty of customers with failed hard drives over the years while working at Beau Photo as well. Since my dad's drive failure, I also have set him up with an automated hard drive backup, so hopefully that won't happen to him again! Actually, now that I think of it, I did have a hard drive fail in the old PowerMac G5 at work quite a few years ago now, but luckily nothing of consequence was lost, although I did have a lengthy re-install of OS X and all the applications and drivers to contend with.
Anyway, continuing with my backup procedures, once I have 25GB (or more) of raw downloaded images accumulated, the Lightroom "backup copy" versions, I will burn those to a 25GB single-layer Blu-Ray disc (BD-R) and once verified, will erase them off the hard drive. Realistically, I generally wait until I have 100GB plus of backup images and then will go and burn multiple BD-R discs in one sitting. After the backups are burned to BD-R, once my nightly backup kicks in, I'll have my working set of images on two hard drives and a backup set on Blu-Ray. Then, after a year or so has gone by, I will eventually rotate my working Lightroom images out onto a second set of Blu-Ray media, at which point I'll either delete the originals off my hard drives completely, or move the images onto a non-backed-up hard drive for continued convenient access.
Before burning the Lightroom working images to Blu-Ray, DVD-R, or whatever other medium you choose, you need to be sure to highlight all the images you are going to burn in Lightroom's grid view, then choose "Save Metadata to Files" from the Metadata menu, or just hit Command-S. This will ensure that all your Lightroom edits and IPTC info are stored along with the image files as .XMP sidecar files, so if you need to re-import those burned images at a later date, all the relevant info will be available to Lightroom again, or to Adobe Camera Raw if you were to just open a single image in Photoshop. A more detailed Lightroom tutorial is beyond the scope of this article, but hopefully that is enough to help you with the backup question.
Blu-Ray burner nestled under left side of iMac
I transitioned to using 25GB Blu-Ray optical media about a year ago, since using those "small" 4.7GB DVDs was a pain! Writeable BD-R discs have come way down in price, as have the burners, so they are now a viable alternative to DVD media, as far as being relatively affordable for the drives and the cost-per-GB. As of this article's writing, buying spindles of 10 or more BD-R discs (Verbatim in this example) will bring the price down to well under $4 per 25GB disc and a good LG 10x Blu-Ray internal burner is now only $160. That compares to $15 per disc and nearly $500 for the burner back when I bought mine!
One thing though, is that I have been able to find very little information on projected media longevity, so I would strongly suggest always backing up to two sets of discs, one working set and one that is kept in archival storage, ideally offsite. BD-R discs have very high data density and while all the good ones have hard, scratch-resistent coatings, if you do manage to scratch one, it is far more likely to affect data than a similar sized scratch on a DVD-R or CD-R disc. As far as overall longevity in archival storage, my gut feeling is that BD-R discs will certainly last long enough to be transitioned to whatever the next standard long-term backup media turns out to be.
What do I mean my "transition"? Well think about it: even though the discs might last 50+ years in storage (some gold DVD-R and CD-R discs are rated for 200+ years), will anyone be able to still read a BD-R disc 30 years into the future? With digital storage, long term preservation is a matter of continually copying files from old, obsolete media onto new, while there is still hardware around to read that old media! However that is a topic for a future blog entry, but I think you get the general idea...
One last point of backup drives is to not trust power supplies either! Say what? Well, let's presume you have a desktop tower, like a Mac Pro. It has four internal drive bays, so the easy solution for backup, is to have multiple internal hard drives set up either with RAID 1 (mirrored) or manually cloned using a program like SuperDuper. However what if there is a catastrophic power supply failure which fries all the drives at once! Unlikely? Maybe, but it actually happened to my dad, which is how he lost his drive mentioned earlier in the article. A power supply malfunction fried his hard drive, his internal CD burner and the motherboard on his computer. If he had multiple internal drives for backup, all of them would likely have suffered the same fate. Maybe a costly data-recovery could recover files, but having external hard drives as backup means that they would not be affected by a failure like that, which he unfortunately did not have. Of course a lightning strike or severe power company surge might kill everything at once, multiple drives or not, but hopefully a really good UPS/surge-protector will minimize that danger. You are using one, right? Anyway, that is one reason why I get stuff onto optical media as fast as is practical.
I have been somewhat lucky I suppose. Many years ago, I was not as paranoid about backing things up, but "got smart" before ever suffering a catastrophic data loss. There are too many people out there complacent about their data and don't implement a good backup strategy until after they experience a drive failure.
So, to summarize my at-home backup strategy, memory cards get downloaded simultaneously to two hard drives and those two drives get cloned to two other backup drives. At this point my images are on four drives. Eventually images from the initial "download backup" get transferred onto BD-R at which point they exist on one set of optical media and two hard drives. Lastly, my working set of images eventually gets stored onto a second set of BD-R media, so I have everything on two sets of optical media.
Once I transition to new media, I plan to keep all the old media for as long as they can still be read. For example, a few years ago I transitioned all my CD-R discs to DVD-R. I plan on doing the same thing with my DVD-R discs in a few years time as well, depending on what the next upcoming long-term standard turns out to be. Maybe it will be Blu-Ray, maybe not...
By the way, when you buy optical media, for Pete's sake make sure you are buying quality discs! When I did my CD-R transition a few years ago, I was able to read every single disc I ever burned, hundreds of them, stretching all the way back to the first one from 1993. I only used quality Kodak or Fuji CD-R discs over the years. I have seen "no-name" spindles of cheap CD-R discs fail only a matter of months after burning, even though they were kept in archival storage. The writeable layer started delaminating from the discs rendering them unusable! Surely your images are worth storing on good media?
At present I am either using Fuji or Verbatim Blu-Ray discs. In the absence of any concrete test data (or long-term experience), I'm really not sure if there are significant differences in the quality of Blu-Ray media. I became an early adopter of Blu-Ray, back when it was still substantially more expensive than it is now, it part to be a guinea pig for testing this new optical media. So far so good: my earliest discs are still 100% readable. Do keep in mind, however, that since most people do not yet have Blu-Ray drives in their computers, when it comes to sharing files, DVDs or CDs are still the way to go.
So that's it for this long blog entry, explaining my at-home backup strategy in detail. Knock on wood, but so far I have never lost a single scan or digital image in all the years I've been working with digital photography! Stay tuned for Part 2 in this series, coming soon, where I will cover my on-the-road backup strategy when shooting stills and video with my digital cameras while traveling...
(Just a point to note: Beau Photo Supplies sells hard drives, WiebeTech drive enclosures, LaCie drives and RAID systems, as well as Blu-Ray burners and BD-R media too!)
Updated (January 3, 2011) - Added more details on hard drive failures, backing up Lightroom library images with XMP metadata and mentioned current pricing on BD-R media and burners.
Updated (January 3, 2011) - Added more details on hard drive failures, backing up Lightroom library images with XMP metadata and mentioned current pricing on BD-R media and burners.
0 comments:
Post a Comment