KDE / Git Prompted Review of Personal Backup Strategy
-
I do have one but something about it has been bothering me for a while. Since, apart from the obviously good reasons for having backups, any failure to carefully guard pictures and videos of our wee little man will cause me to involuntarily lose body parts courtesy of my good lady I've had everything 'backed up' for the last couple of years, it even survived moving country. The KDE story finally prompted me to kick-off the process and then I hit a problem. Since backup didn't seem to fit into any of the other forums I've dropped it in here and also here since it seems to be of general interest to most. Back to the problem... Currently I use a NAS for everything, all documents, photos etc. All the “My xxxx” folders point at the NAS, including “Public xxxx”. This is copied to a USB HDD every day, in full. I’ve been moderately bothered for a while about the obvious failure mode, corruption just gets copied to the ‘backup’. The KDE Git story finally made me think that I really ought to do something and I’m after recommendations. I want it to be fire and forget and I guess I’m looking at doing something like keeping full copies longer term but I don’t want to pre-judge. The NAS is a QNAP 2 bay SOHO device (I think it’s the TS 210 rather than the current TS 212). It has one 2TB disk (I bought a 2 bay for later expansion). The USB that’s attached to it is also 2TB. I am currently accepting of the risk that both disks are in the same house and that I don’t have one offsite (fire, meteor strike etc.). I am not enamoured by the idea of sending 1TB of data up an ADSL line. My first thought was: 1) Add the second disk and put it into mirror mode as an alternative to nightly copies (loss of expansion capacity but that’s acceptable) 2) Have nightly copies that are over-written seven days later on an external disk - i.e. I can reverse anything in the last 7 days 3) Keep one nightly copy (Sunday for talks sake) from each week for 4 or 5 weeks at which point the oldest one gets written – i.e. I can revert to any given Sunday in the last 4 or 5 4) Keep one copy per 4 / 5 weeks for 6 months Sounds great until you realise what that adds up to in volume, without compression it turns 1 byte into 16!! Assume I have 1TB and it doesn’t compress well (videos and photos). Option 2 requires 7 TB Option 3 adds a further 4 TB (5 week cycle) Option 4 adds a further 5 TB So then I think, versioning, searching t’internet for QNAP versioning brings up lots of links for the offsite storage capability of the dev
Mike-MadBadger wrote:
I am currently accepting of the risk that both disks are in the same house and that I don’t have one offsite (fire, meteor strike etc.). I am not enamoured by the idea of sending 1GB of data up an ADSL line.
If you only have 1 GB of data, just suck it up and upload it. Even at crappy slow DSL speeds it shouldn't take more than a few nights. Amazon's cloud uploader is fully pausable so you can stop it when you need to use your bandwidth during the day. If you meant a TB of data then I'd agree you've got a problem with initial population of a cloud backup. I know some will let you do initial imports by mailing external drives back and forth; but don't know if that's available for consumer oriented services or not. Do you have any friends with fast cable/fiber pipes to jam the initial upload through?
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
-
http://www.codeproject.com/Messages/4525349/KDEs-Git-meltdown.aspx[^] :doh:
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
yeah, that's the link I found with google.
If your actions inspire others to dream more, learn more, do more and become more, you are a leader.-John Q. Adams
You must accept one of two basic premises: Either we are alone in the universe, or we are not alone in the universe. And either way, the implications are staggering.-Wernher von Braun
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.-Albert Einstein -
Mike-MadBadger wrote:
I am currently accepting of the risk that both disks are in the same house and that I don’t have one offsite (fire, meteor strike etc.). I am not enamoured by the idea of sending 1GB of data up an ADSL line.
If you only have 1 GB of data, just suck it up and upload it. Even at crappy slow DSL speeds it shouldn't take more than a few nights. Amazon's cloud uploader is fully pausable so you can stop it when you need to use your bandwidth during the day. If you meant a TB of data then I'd agree you've got a problem with initial population of a cloud backup. I know some will let you do initial imports by mailing external drives back and forth; but don't know if that's available for consumer oriented services or not. Do you have any friends with fast cable/fiber pipes to jam the initial upload through?
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason? Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful? --Zachris Topelius Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies. -- Sarah Hoyt
-
What you're probably looking for is a versioning file system, that like Subversion maintains the changes to files (or rather the current state, and a delta to the 'previous' states), that way if a file is accidentally deleted etc, you can go back to a previous version, without the overhead of having multiple snapshot copies of all the data. I'm not sure if any exist however.
Storm-blade wrote:
What you're probably looking for is a versioning file system, that like Subversion maintains the changes to files (or rather the current state, and a delta to the 'previous' states), that way if a file is accidentally deleted etc, you can go back to a previous version, without the overhead of having multiple snapshot copies of all the data.
Well actually, git does that - for text files, most (all?) version control systems maintain an original file and deltas between subsequent states (binary files are typically stored as entire copies of each file version). Some VCSs will handle unicode as binary, some not.
-- Harvey
-
Storm-blade wrote:
What you're probably looking for is a versioning file system, that like Subversion maintains the changes to files (or rather the current state, and a delta to the 'previous' states), that way if a file is accidentally deleted etc, you can go back to a previous version, without the overhead of having multiple snapshot copies of all the data.
Well actually, git does that - for text files, most (all?) version control systems maintain an original file and deltas between subsequent states (binary files are typically stored as entire copies of each file version). Some VCSs will handle unicode as binary, some not.
-- Harvey
Ideally I'd want something like Subversion, however it is not suitable for this task. I don't know how git handles it's data, but with subversion you effectively have three copies of the same data, the repository, the working copy and the 'pristine' copy as part of the working copy, so the data takes at least 3 times the space. Also, in the repository, it uses forward deltas, the current version isn't stored, only the initial version and changes since that version, which means any attempt to read from the repository without using a working copy, is a lot slower than a normal file. All this is fine for a revision control system, but not for a versioning file system.
-
I do have one but something about it has been bothering me for a while. Since, apart from the obviously good reasons for having backups, any failure to carefully guard pictures and videos of our wee little man will cause me to involuntarily lose body parts courtesy of my good lady I've had everything 'backed up' for the last couple of years, it even survived moving country. The KDE story finally prompted me to kick-off the process and then I hit a problem. Since backup didn't seem to fit into any of the other forums I've dropped it in here and also here since it seems to be of general interest to most. Back to the problem... Currently I use a NAS for everything, all documents, photos etc. All the “My xxxx” folders point at the NAS, including “Public xxxx”. This is copied to a USB HDD every day, in full. I’ve been moderately bothered for a while about the obvious failure mode, corruption just gets copied to the ‘backup’. The KDE Git story finally made me think that I really ought to do something and I’m after recommendations. I want it to be fire and forget and I guess I’m looking at doing something like keeping full copies longer term but I don’t want to pre-judge. The NAS is a QNAP 2 bay SOHO device (I think it’s the TS 210 rather than the current TS 212). It has one 2TB disk (I bought a 2 bay for later expansion). The USB that’s attached to it is also 2TB. I am currently accepting of the risk that both disks are in the same house and that I don’t have one offsite (fire, meteor strike etc.). I am not enamoured by the idea of sending 1TB of data up an ADSL line. My first thought was: 1) Add the second disk and put it into mirror mode as an alternative to nightly copies (loss of expansion capacity but that’s acceptable) 2) Have nightly copies that are over-written seven days later on an external disk - i.e. I can reverse anything in the last 7 days 3) Keep one nightly copy (Sunday for talks sake) from each week for 4 or 5 weeks at which point the oldest one gets written – i.e. I can revert to any given Sunday in the last 4 or 5 4) Keep one copy per 4 / 5 weeks for 6 months Sounds great until you realise what that adds up to in volume, without compression it turns 1 byte into 16!! Assume I have 1TB and it doesn’t compress well (videos and photos). Option 2 requires 7 TB Option 3 adds a further 4 TB (5 week cycle) Option 4 adds a further 5 TB So then I think, versioning, searching t’internet for QNAP versioning brings up lots of links for the offsite storage capability of the dev
-
I do have one but something about it has been bothering me for a while. Since, apart from the obviously good reasons for having backups, any failure to carefully guard pictures and videos of our wee little man will cause me to involuntarily lose body parts courtesy of my good lady I've had everything 'backed up' for the last couple of years, it even survived moving country. The KDE story finally prompted me to kick-off the process and then I hit a problem. Since backup didn't seem to fit into any of the other forums I've dropped it in here and also here since it seems to be of general interest to most. Back to the problem... Currently I use a NAS for everything, all documents, photos etc. All the “My xxxx” folders point at the NAS, including “Public xxxx”. This is copied to a USB HDD every day, in full. I’ve been moderately bothered for a while about the obvious failure mode, corruption just gets copied to the ‘backup’. The KDE Git story finally made me think that I really ought to do something and I’m after recommendations. I want it to be fire and forget and I guess I’m looking at doing something like keeping full copies longer term but I don’t want to pre-judge. The NAS is a QNAP 2 bay SOHO device (I think it’s the TS 210 rather than the current TS 212). It has one 2TB disk (I bought a 2 bay for later expansion). The USB that’s attached to it is also 2TB. I am currently accepting of the risk that both disks are in the same house and that I don’t have one offsite (fire, meteor strike etc.). I am not enamoured by the idea of sending 1TB of data up an ADSL line. My first thought was: 1) Add the second disk and put it into mirror mode as an alternative to nightly copies (loss of expansion capacity but that’s acceptable) 2) Have nightly copies that are over-written seven days later on an external disk - i.e. I can reverse anything in the last 7 days 3) Keep one nightly copy (Sunday for talks sake) from each week for 4 or 5 weeks at which point the oldest one gets written – i.e. I can revert to any given Sunday in the last 4 or 5 4) Keep one copy per 4 / 5 weeks for 6 months Sounds great until you realise what that adds up to in volume, without compression it turns 1 byte into 16!! Assume I have 1TB and it doesn’t compress well (videos and photos). Option 2 requires 7 TB Option 3 adds a further 4 TB (5 week cycle) Option 4 adds a further 5 TB So then I think, versioning, searching t’internet for QNAP versioning brings up lots of links for the offsite storage capability of the dev
The KDE story makes me wonder about the risk when your backup app deletes files from the backup, especially if it's automated... accidental deletions from the source would cause loss of backup data too.
"For fifty bucks I'd put my face in their soup and blow." - George Costanza
CP article: SmartPager - a Flickr-style pager control with go-to-page popup layer.
-
I have a file server at with 24TB of disk space on it. This is made up of 11x2TB plus 2x1TB drives connected through a combination of eSata and USB 2.0 (via a couple of powered USB hubs). The file server runs XP because it is an old machine that already had XP Professional on it and it sufficed. I keep my data on the first 12TB and back it up (deltas) every night to the other 12TB. I also have a 2TB drive on my main development machine and another on my wife's machine which are also backed-up. At the moment I am well below capacity (I got carried away with a Tiger Direct Black Something sale last year) so space isn't an issue. For movies, music, books and my own huge image files (another story) and keep just the one extra backup copy of each. For code I use SVN so I can keep umpteen copies back to 2006 (when I first created the current SVN depository) and corruption isn't an issue as I can go back as far as necessary to get original code. The depositories are, of course, backed-up. I have a batch file set up that I can run at any extra time in case I want an extra delta or full backup run but normally I just use the nightly process. For copies of "official" documents I keep extra backups on a couple of USB keys and a portable USB drive as well as an encrypted archive of them on each of Dropbox, Google Drive and MS SkyDrive, as well as another copy on my laptop (along with an occasional manual code copy) - am I paranoid? Only a little, once I have set up the scheduled jobs for all this it is Fire and Forget as you wanted with reasonable peace of mind.
- I would love to change the world, but they won’t give me the source code.
Forogar wrote:
This is made up of 11x2TB plus 2x1TB drives connected
The extra weight makes it harder to steal too... giving you added security :-D Personally, I'm starting to wonder how well my backup strategy would fare in event of fire or theft... not too well at this stage :doh:
"For fifty bucks I'd put my face in their soup and blow." - George Costanza
CP article: SmartPager - a Flickr-style pager control with go-to-page popup layer.
-
I do have one but something about it has been bothering me for a while. Since, apart from the obviously good reasons for having backups, any failure to carefully guard pictures and videos of our wee little man will cause me to involuntarily lose body parts courtesy of my good lady I've had everything 'backed up' for the last couple of years, it even survived moving country. The KDE story finally prompted me to kick-off the process and then I hit a problem. Since backup didn't seem to fit into any of the other forums I've dropped it in here and also here since it seems to be of general interest to most. Back to the problem... Currently I use a NAS for everything, all documents, photos etc. All the “My xxxx” folders point at the NAS, including “Public xxxx”. This is copied to a USB HDD every day, in full. I’ve been moderately bothered for a while about the obvious failure mode, corruption just gets copied to the ‘backup’. The KDE Git story finally made me think that I really ought to do something and I’m after recommendations. I want it to be fire and forget and I guess I’m looking at doing something like keeping full copies longer term but I don’t want to pre-judge. The NAS is a QNAP 2 bay SOHO device (I think it’s the TS 210 rather than the current TS 212). It has one 2TB disk (I bought a 2 bay for later expansion). The USB that’s attached to it is also 2TB. I am currently accepting of the risk that both disks are in the same house and that I don’t have one offsite (fire, meteor strike etc.). I am not enamoured by the idea of sending 1TB of data up an ADSL line. My first thought was: 1) Add the second disk and put it into mirror mode as an alternative to nightly copies (loss of expansion capacity but that’s acceptable) 2) Have nightly copies that are over-written seven days later on an external disk - i.e. I can reverse anything in the last 7 days 3) Keep one nightly copy (Sunday for talks sake) from each week for 4 or 5 weeks at which point the oldest one gets written – i.e. I can revert to any given Sunday in the last 4 or 5 4) Keep one copy per 4 / 5 weeks for 6 months Sounds great until you realise what that adds up to in volume, without compression it turns 1 byte into 16!! Assume I have 1TB and it doesn’t compress well (videos and photos). Option 2 requires 7 TB Option 3 adds a further 4 TB (5 week cycle) Option 4 adds a further 5 TB So then I think, versioning, searching t’internet for QNAP versioning brings up lots of links for the offsite storage capability of the dev
I'd also keep a copy on optical media. Do some research and find the one with the best longevity rating (that's all voodoo anyway, but who knows, maybe they got it right). The biggest problem with long term backups isn't actually the media, its the data format. It can also be the media format if there's more that one way to save bits onto a media (e.g. Amiga floppy). No matter what you do, if you really want the data for long term, you will need to periodically convert it to a new media and perhaps data format too. It sounds like you're aproaching this with the mindeset of IT backup strategies. In reality, your every digital utterance doesn't need to be protected equally. Preserve the stuff you can't recreate and absolutely don't want to loose (like pictures) with more redundancy than the stuff you could easily recreate (like your browser cookies). When you distill it down to that, even with redundancy in media, file format, duplicate copies on the media, and storage locations, it just isn't that much data.
We can program with only 1's, but if all you've got are zeros, you've got nothing.
-
Forogar wrote:
This is made up of 11x2TB plus 2x1TB drives connected
The extra weight makes it harder to steal too... giving you added security :-D Personally, I'm starting to wonder how well my backup strategy would fare in event of fire or theft... not too well at this stage :doh:
"For fifty bucks I'd put my face in their soup and blow." - George Costanza
CP article: SmartPager - a Flickr-style pager control with go-to-page popup layer.