How to store huge binary files without Database
-
Mercurius84 wrote:
I just want to compress the files and package them into one container for a configurable size.
Windows has compressed drives. Pretty sure every major OS does as well. However your requirements still don't meet that need. Again your requirements don't require a specialized system. Current file systems are more than capable of handling that trivial amount of data.
Yes, However, we need to further shrinking down the file size and better file handling. Instead of OS handling.
-
Many thanks. But this process only works at the end of the document status. I do not think it would be able to append more files after been compressed and segmented the files I have a requirement too, which additional files can be added to after the above compression and segmentation.
Normally, I don't think further modification of the multi-volume is possible, but I don't have the insight to tell you so. There might be one archive tool that can do it. Kind Regards, Keld Ølykke
-
Yes, However, we need to further shrinking down the file size and better file handling. Instead of OS handling.
Mercurius84 wrote:
However, we need to further shrinking down the file size and better file handling.
And you base that on what exactly? What is your criteria? What is your desired improvement? How did you measure the criteria using the file system. How do you escape the fact that any such solution will STILL rely on the file system?
Mercurius84 wrote:
Instead of OS handling.
The OS has been optimized to handle files given that OS file systems are a key component of desktop OSes.
-
Hi, What do you mean by overkill? This HDFS has its limitation and does not locate the processing logic power? I have no personal experience to this product As per summarized: Hadoop is an Apache Software Foundation distributed file system and data management project with goals for storing and managing large amounts of data. Hadoop uses a storage system called HDFS to connect commodity personal computers, known as nodes, contained within clusters over which data blocks are distributed. You can access and store the data blocks as one seamless file system using the MapReduce processing model. HDFS shares many common features with other distributed file systems while supporting some important differences. One significant difference is HDFS's write-once-read-many model that relaxes concurrency control requirements, simplifies data coherency, and enables high-throughput access. In order to provide an optimized data-access model, HDFS is designed to locate processing logic near the data rather than locating data near the application space. It sounds promising.
Mercurius84 wrote:
What do you mean by overkill...Hadoop is an Apache Software Foundation distributed file system and data management project with goals for storing and managing large amounts of data.
Your stated requirements do not meet the definition of "large amounts of data". Let me give you some examples of large data - 2000 transactions a second sustained with a expected lifetime of 7 years and a real time need of 6 to 18 months immediate availability. Each transaction has a 1k size. - Each originator will produce several 100 meg downloads several times a month. Sizing must expect up to 10,000 originators with a lifetime of 5 years.
-
Mercurius84 wrote:
What do you mean by overkill...Hadoop is an Apache Software Foundation distributed file system and data management project with goals for storing and managing large amounts of data.
Your stated requirements do not meet the definition of "large amounts of data". Let me give you some examples of large data - 2000 transactions a second sustained with a expected lifetime of 7 years and a real time need of 6 to 18 months immediate availability. Each transaction has a 1k size. - Each originator will produce several 100 meg downloads several times a month. Sizing must expect up to 10,000 originators with a lifetime of 5 years.
I get what you mean : )
-
Mercurius84 wrote:
However, we need to further shrinking down the file size and better file handling.
And you base that on what exactly? What is your criteria? What is your desired improvement? How did you measure the criteria using the file system. How do you escape the fact that any such solution will STILL rely on the file system?
Mercurius84 wrote:
Instead of OS handling.
The OS has been optimized to handle files given that OS file systems are a key component of desktop OSes.
I had an assumption that programming could do 'almost' wonderful things. for example: I have a file sized 10 MB. This special program shrinks down the size by ~30% (7MB for example) and then store it in to a container. What i mean by OS handling is unlike a normal compression and store them into a folder.
-
I have found the solution by this product: Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data. Thanks :)
-
I had an assumption that programming could do 'almost' wonderful things. for example: I have a file sized 10 MB. This special program shrinks down the size by ~30% (7MB for example) and then store it in to a container. What i mean by OS handling is unlike a normal compression and store them into a folder.
Mercurius84 wrote:
This special program shrinks down the size by ~30% (7MB for example)
and then store it in to a container.Fine - but why to you need to do that? What is the business or technical need that requires this?
Mercurius84 wrote:
What i mean by OS handling is unlike a normal compression and store them into a folder.
Not sure what you mean by that - as I already said desktop OSes already support compression. So that point by itself is moot.
-
Mercurius84 wrote:
This special program shrinks down the size by ~30% (7MB for example)
and then store it in to a container.Fine - but why to you need to do that? What is the business or technical need that requires this?
Mercurius84 wrote:
What i mean by OS handling is unlike a normal compression and store them into a folder.
Not sure what you mean by that - as I already said desktop OSes already support compression. So that point by itself is moot.
Hi, Actually our company wants to develop something similar to COLD system. But in generically .... we are building a new product based to this concept....
-
Hi, Actually our company wants to develop something similar to COLD system. But in generically .... we are building a new product based to this concept....
Mercurius84 wrote:
similar to COLD system.
No idea what that is but if a static storage system then your stated requirements still are not anywhere close to being reasonable for creating and marketing such a system. Just to make it clear again, your stated requirements do not present a need for anything. I can only presume that either you work in an highly unusual environment (an unstated requirement) or the stated file count\size does not span that actual need (another unstated requirement.)
-
Mercurius84 wrote:
similar to COLD system.
No idea what that is but if a static storage system then your stated requirements still are not anywhere close to being reasonable for creating and marketing such a system. Just to make it clear again, your stated requirements do not present a need for anything. I can only presume that either you work in an highly unusual environment (an unstated requirement) or the stated file count\size does not span that actual need (another unstated requirement.)
understand. thanks