Archival Methods (TAPE, IDE DISK DVD-R)...
-
Here is my problem... We do medical imaging research which generates several GB of data per day. We can not possibly store this online (which we would like to) but we currently archive everything to tape and store as much as we can on disk. We have several TB of IDE drives scattered in a few servers. The thing is that tape now costs more per GB than IDE drives cost. With our ADR drives tapes cost > $2 GB for 25 GB native, IDE drives run for less than $1 per GB for 120GB models... Tape is much slower and much harder to get your data off it. And also it is debatable on how reliable tapes are since we have lost more data on tapes than IDE drives... We are thinking of buying a bunch of 120GB or larger IDE drives and giving up on tape in the future. An other thought is to use DVD-R disks which can be purchased for $2 to $3 in bulk but 4.7GB would require a lot of DVD disks and would require more work to store and find the correct disk with the data on. Also we would probably not want cases to span disks so this will cause some waste and with the other two methods compression is easily available and transparent to the user but it seems it would be more difficult to use compression on a DVD disk unless we use packet writing and NTFS... My question is what are your thoughts on the matter? John
-
Here is my problem... We do medical imaging research which generates several GB of data per day. We can not possibly store this online (which we would like to) but we currently archive everything to tape and store as much as we can on disk. We have several TB of IDE drives scattered in a few servers. The thing is that tape now costs more per GB than IDE drives cost. With our ADR drives tapes cost > $2 GB for 25 GB native, IDE drives run for less than $1 per GB for 120GB models... Tape is much slower and much harder to get your data off it. And also it is debatable on how reliable tapes are since we have lost more data on tapes than IDE drives... We are thinking of buying a bunch of 120GB or larger IDE drives and giving up on tape in the future. An other thought is to use DVD-R disks which can be purchased for $2 to $3 in bulk but 4.7GB would require a lot of DVD disks and would require more work to store and find the correct disk with the data on. Also we would probably not want cases to span disks so this will cause some waste and with the other two methods compression is easily available and transparent to the user but it seems it would be more difficult to use compression on a DVD disk unless we use packet writing and NTFS... My question is what are your thoughts on the matter? John
-
Hi, you can check 'Enterprise Automated Tape Library' from IBM - or any other storage solution offered by IBM Bye, Orbital^ ...the night is long ... but not long enought to do some real coding ...
I thought about automated tape libraries but the drives are many thousands of dollars and I am not sure what the cost per GB for storage is.. Our thoughts was $2 / GB for tape was too expensive... [EDIT] Ok a quick check on pricewatch I see LTO tapes are about 0.50 / GB for 100GB or 200GB native models... [/EDIT] John
-
Here is my problem... We do medical imaging research which generates several GB of data per day. We can not possibly store this online (which we would like to) but we currently archive everything to tape and store as much as we can on disk. We have several TB of IDE drives scattered in a few servers. The thing is that tape now costs more per GB than IDE drives cost. With our ADR drives tapes cost > $2 GB for 25 GB native, IDE drives run for less than $1 per GB for 120GB models... Tape is much slower and much harder to get your data off it. And also it is debatable on how reliable tapes are since we have lost more data on tapes than IDE drives... We are thinking of buying a bunch of 120GB or larger IDE drives and giving up on tape in the future. An other thought is to use DVD-R disks which can be purchased for $2 to $3 in bulk but 4.7GB would require a lot of DVD disks and would require more work to store and find the correct disk with the data on. Also we would probably not want cases to span disks so this will cause some waste and with the other two methods compression is easily available and transparent to the user but it seems it would be more difficult to use compression on a DVD disk unless we use packet writing and NTFS... My question is what are your thoughts on the matter? John
As cheap as drives are it is not a bad option. A factor is also storage space, with 24 DVD's per drive equivalent is not that much different. A question back that I think is very important for you to answer prior to finding a solution. How are you indexing all of this data and how does that index get saved with the metadata for each case you are archiving? You need some automated method of labeling each device no matter what you choose and getting that label saved with the rest of the case information. If that is not done in a fool proof manner you are kidding your self that you have an archival system. You might as well print copies out and save them in boxes. Just my 2 cents. "For as long as I can remember, I have had memories. Colin Mochrie."
-
As cheap as drives are it is not a bad option. A factor is also storage space, with 24 DVD's per drive equivalent is not that much different. A question back that I think is very important for you to answer prior to finding a solution. How are you indexing all of this data and how does that index get saved with the metadata for each case you are archiving? You need some automated method of labeling each device no matter what you choose and getting that label saved with the rest of the case information. If that is not done in a fool proof manner you are kidding your self that you have an archival system. You might as well print copies out and save them in boxes. Just my 2 cents. "For as long as I can remember, I have had memories. Colin Mochrie."
The data is medical images which can be anywhere from 36MB / case to almost 1 GB per case depending on what type of case it is digital mammograpgy, chest CT, MRI... Right now there are several databases that store the filenames associated with the images. A quick search on Veritas Backup Executive tells us what tape to insert... This system is in the process of change with new government regulations all of our images and data must be scrubbed to remove patient identification so at some point we must read all the tapes (over 100 now) anonymize them into a new honest broker database and write the data back out. I know this will be a pain... John
-
Here is my problem... We do medical imaging research which generates several GB of data per day. We can not possibly store this online (which we would like to) but we currently archive everything to tape and store as much as we can on disk. We have several TB of IDE drives scattered in a few servers. The thing is that tape now costs more per GB than IDE drives cost. With our ADR drives tapes cost > $2 GB for 25 GB native, IDE drives run for less than $1 per GB for 120GB models... Tape is much slower and much harder to get your data off it. And also it is debatable on how reliable tapes are since we have lost more data on tapes than IDE drives... We are thinking of buying a bunch of 120GB or larger IDE drives and giving up on tape in the future. An other thought is to use DVD-R disks which can be purchased for $2 to $3 in bulk but 4.7GB would require a lot of DVD disks and would require more work to store and find the correct disk with the data on. Also we would probably not want cases to span disks so this will cause some waste and with the other two methods compression is easily available and transparent to the user but it seems it would be more difficult to use compression on a DVD disk unless we use packet writing and NTFS... My question is what are your thoughts on the matter? John
Sounds like you need to invest in a high density SAN or NAS type system. http://www.dell.com/us/en/biz/topics/segtopic_storageHome.htm[^]
Paul Watson wrote: "At the end of the day it is what you produce that counts, not how many doctorates you have on the wall." George Carlin wrote: "Don't sweat the petty things, and don't pet the sweaty things." Jörgen Sigvardsson wrote: If the physicists find a universal theory describing the laws of universe, I'm sure the asshole constant will be an integral part of that theory.
-
Here is my problem... We do medical imaging research which generates several GB of data per day. We can not possibly store this online (which we would like to) but we currently archive everything to tape and store as much as we can on disk. We have several TB of IDE drives scattered in a few servers. The thing is that tape now costs more per GB than IDE drives cost. With our ADR drives tapes cost > $2 GB for 25 GB native, IDE drives run for less than $1 per GB for 120GB models... Tape is much slower and much harder to get your data off it. And also it is debatable on how reliable tapes are since we have lost more data on tapes than IDE drives... We are thinking of buying a bunch of 120GB or larger IDE drives and giving up on tape in the future. An other thought is to use DVD-R disks which can be purchased for $2 to $3 in bulk but 4.7GB would require a lot of DVD disks and would require more work to store and find the correct disk with the data on. Also we would probably not want cases to span disks so this will cause some waste and with the other two methods compression is easily available and transparent to the user but it seems it would be more difficult to use compression on a DVD disk unless we use packet writing and NTFS... My question is what are your thoughts on the matter? John
We ran into a similar problem a few months ago. A typical mapping project for us will generate between 1 and 4 GB of imagery and data files. We've been buying 120 GB IDE drives, but this obviously has its limitations. We decided to buy DVD writers and now burn DVD-R's for completed projects. I start the burning just before I'm done for the day. We've now turned our IDE drives into backup drives that are updated each night. First thing in the morning I burn a duplicate copy of each DVD-R for offsite storage. I resist the temptation to put nice labels on them - I've read some articles that suggest the glue can seriously reduce the lifespan of CD's and DVD's. I write on them with a Sharpie pen. Works great so far. We use an incremental year/month based project numbering system that keeps things simple but effective. Drew.
-
The data is medical images which can be anywhere from 36MB / case to almost 1 GB per case depending on what type of case it is digital mammograpgy, chest CT, MRI... Right now there are several databases that store the filenames associated with the images. A quick search on Veritas Backup Executive tells us what tape to insert... This system is in the process of change with new government regulations all of our images and data must be scrubbed to remove patient identification so at some point we must read all the tapes (over 100 now) anonymize them into a new honest broker database and write the data back out. I know this will be a pain... John
I did not make myself clear. Will your Veritas Backup Executive also handle which drive to insert if you swap them out? "For as long as I can remember, I have had memories. Colin Mochrie."
-
I did not make myself clear. Will your Veritas Backup Executive also handle which drive to insert if you swap them out? "For as long as I can remember, I have had memories. Colin Mochrie."
No it won't. John
-
We ran into a similar problem a few months ago. A typical mapping project for us will generate between 1 and 4 GB of imagery and data files. We've been buying 120 GB IDE drives, but this obviously has its limitations. We decided to buy DVD writers and now burn DVD-R's for completed projects. I start the burning just before I'm done for the day. We've now turned our IDE drives into backup drives that are updated each night. First thing in the morning I burn a duplicate copy of each DVD-R for offsite storage. I resist the temptation to put nice labels on them - I've read some articles that suggest the glue can seriously reduce the lifespan of CD's and DVD's. I write on them with a Sharpie pen. Works great so far. We use an incremental year/month based project numbering system that keeps things simple but effective. Drew.
I seriously thought of this method but if I can fit 24 DVDs on a single hard disk it seems that it would be much less work to use hard drives but your method makes the idea interesting again. The problem I have with this is our data comes from several different and unrealted sources. Mamographic images, chest CT, MRI ... John
-
Here is my problem... We do medical imaging research which generates several GB of data per day. We can not possibly store this online (which we would like to) but we currently archive everything to tape and store as much as we can on disk. We have several TB of IDE drives scattered in a few servers. The thing is that tape now costs more per GB than IDE drives cost. With our ADR drives tapes cost > $2 GB for 25 GB native, IDE drives run for less than $1 per GB for 120GB models... Tape is much slower and much harder to get your data off it. And also it is debatable on how reliable tapes are since we have lost more data on tapes than IDE drives... We are thinking of buying a bunch of 120GB or larger IDE drives and giving up on tape in the future. An other thought is to use DVD-R disks which can be purchased for $2 to $3 in bulk but 4.7GB would require a lot of DVD disks and would require more work to store and find the correct disk with the data on. Also we would probably not want cases to span disks so this will cause some waste and with the other two methods compression is easily available and transparent to the user but it seems it would be more difficult to use compression on a DVD disk unless we use packet writing and NTFS... My question is what are your thoughts on the matter? John
We are now thinking of adding a 1 TB RAID 5 system which we can put together for about $1000 US. I checked price watch and 120GB WD drives are about $100 US. Using our DC (Win2k Server) which has a 10 open bays and support for 12 HDs on the MOBO we should not need any additional hw or software because win2k has software raid. To me this is kind of a delay tactic but it will solve the immediate need. John
-
We are now thinking of adding a 1 TB RAID 5 system which we can put together for about $1000 US. I checked price watch and 120GB WD drives are about $100 US. Using our DC (Win2k Server) which has a 10 open bays and support for 12 HDs on the MOBO we should not need any additional hw or software because win2k has software raid. To me this is kind of a delay tactic but it will solve the immediate need. John
problem with lots of hard drives, is that you better be able to keep them cool.. otherwise they will kill each other.
-
problem with lots of hard drives, is that you better be able to keep them cool.. otherwise they will kill each other.
I make sure that the drives don't get too hot.. The server room is absolutely freezing and the pc that these are going into has 7 fans on it already... John
-
I make sure that the drives don't get too hot.. The server room is absolutely freezing and the pc that these are going into has 7 fans on it already... John
:cool:
-
Here is my problem... We do medical imaging research which generates several GB of data per day. We can not possibly store this online (which we would like to) but we currently archive everything to tape and store as much as we can on disk. We have several TB of IDE drives scattered in a few servers. The thing is that tape now costs more per GB than IDE drives cost. With our ADR drives tapes cost > $2 GB for 25 GB native, IDE drives run for less than $1 per GB for 120GB models... Tape is much slower and much harder to get your data off it. And also it is debatable on how reliable tapes are since we have lost more data on tapes than IDE drives... We are thinking of buying a bunch of 120GB or larger IDE drives and giving up on tape in the future. An other thought is to use DVD-R disks which can be purchased for $2 to $3 in bulk but 4.7GB would require a lot of DVD disks and would require more work to store and find the correct disk with the data on. Also we would probably not want cases to span disks so this will cause some waste and with the other two methods compression is easily available and transparent to the user but it seems it would be more difficult to use compression on a DVD disk unless we use packet writing and NTFS... My question is what are your thoughts on the matter? John
How about geting 200GB drives and putting them in removable carriers (many RAID controllers support hot swap now)? And yes, use the best compression you can ! Elaine :rose: The tigress is here :-D
-
We ran into a similar problem a few months ago. A typical mapping project for us will generate between 1 and 4 GB of imagery and data files. We've been buying 120 GB IDE drives, but this obviously has its limitations. We decided to buy DVD writers and now burn DVD-R's for completed projects. I start the burning just before I'm done for the day. We've now turned our IDE drives into backup drives that are updated each night. First thing in the morning I burn a duplicate copy of each DVD-R for offsite storage. I resist the temptation to put nice labels on them - I've read some articles that suggest the glue can seriously reduce the lifespan of CD's and DVD's. I write on them with a Sharpie pen. Works great so far. We use an incremental year/month based project numbering system that keeps things simple but effective. Drew.
We have now decided to do both. We purchased 1 TB of 120GB WD drives for < $1000 US which we are going to use with software RAID5 in our main server which has 12 open drive bays and we are also going to backup the data with inexpensive DVD+R ($1 to $3 per disk) disks using our Veritas Backup Executive program. The two solutions together will still cost less than $2 / GB so it is a win win situation... Thanks for your help. John