How to store huge binary files without Database

jschell

Mercurius84 wrote:

Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.

Based only what you described as your needs this is overkill.

Keld Olykke

Hi Merc, It just hit me when reading this post that your problem is similar to to the problem of splitting a large file into chunks. As an example if you have big archieve on disk and want to store it on removable devices e.g. floppy, cdrom or dvd. In the old days we used ARJ to split a compressed file into a number of volumes (.arj, .a01, .a02, etc.). Each volume had a fixed maximum size e.g. 1.44 MB for a floppy. A limitation was that you could not add/remove stuff from .a02 without breaking the big file, so if you need to do this, splitting big files into compressed volumes might not be your solution. If you want to play with this approach you can use rar. According to http://acritum.com/software/manuals/winrar/html/helparcvolumes.htm[^] this feature is called multivolume archieves. Kind Regards, Keld Ølykke

jschell

Mercurius84 wrote:

I just want to compress the files and package them into one container for a configurable size.

Windows has compressed drives. Pretty sure every major OS does as well. However your requirements still don't meet that need. Again your requirements don't require a specialized system. Current file systems are more than capable of handling that trivial amount of data.

Mercurius84

Hi, What do you mean by overkill? This HDFS has its limitation and does not locate the processing logic power? I have no personal experience to this product As per summarized: Hadoop is an Apache Software Foundation distributed file system and data management project with goals for storing and managing large amounts of data. Hadoop uses a storage system called HDFS to connect commodity personal computers, known as nodes, contained within clusters over which data blocks are distributed. You can access and store the data blocks as one seamless file system using the MapReduce processing model. HDFS shares many common features with other distributed file systems while supporting some important differences. One significant difference is HDFS's write-once-read-many model that relaxes concurrency control requirements, simplifies data coherency, and enables high-throughput access. In order to provide an optimized data-access model, HDFS is designed to locate processing logic near the data rather than locating data near the application space. It sounds promising.

Mercurius84

Many thanks. But this process only works at the end of the document status. I do not think it would be able to append more files after been compressed and segmented the files I have a requirement too, which additional files can be added to after the above compression and segmentation.

Mercurius84

Yes, However, we need to further shrinking down the file size and better file handling. Instead of OS handling.

Keld Olykke

Normally, I don't think further modification of the multi-volume is possible, but I don't have the insight to tell you so. There might be one archive tool that can do it. Kind Regards, Keld Ølykke

jschell

Mercurius84 wrote:

However, we need to further shrinking down the file size and better file handling.

And you base that on what exactly? What is your criteria? What is your desired improvement? How did you measure the criteria using the file system. How do you escape the fact that any such solution will STILL rely on the file system?

Mercurius84 wrote:

Instead of OS handling.

The OS has been optimized to handle files given that OS file systems are a key component of desktop OSes.

jschell

Mercurius84 wrote:

What do you mean by overkill...Hadoop is an Apache Software Foundation distributed file system and data management project with goals for storing and managing large amounts of data.

Your stated requirements do not meet the definition of "large amounts of data". Let me give you some examples of large data - 2000 transactions a second sustained with a expected lifetime of 7 years and a real time need of 6 to 18 months immediate availability. Each transaction has a 1k size. - Each originator will produce several 100 meg downloads several times a month. Sizing must expect up to 10,000 originators with a lifetime of 5 years.

Mercurius84

I get what you mean : )

Mercurius84

I had an assumption that programming could do 'almost' wonderful things. for example: I have a file sized 10 MB. This special program shrinks down the size by ~30% (7MB for example) and then store it in to a container. What i mean by OS handling is unlike a normal compression and store them into a folder.

Lost User

Why not use http://msdn.microsoft.com/en-us/library/system.io.compression.ziparchive.aspx[^]?

Use the best guess

jschell

Mercurius84 wrote:

This special program shrinks down the size by ~30% (7MB for example)
and then store it in to a container.

Fine - but why to you need to do that? What is the business or technical need that requires this?

Mercurius84 wrote:

What i mean by OS handling is unlike a normal compression and store them into a folder.

Not sure what you mean by that - as I already said desktop OSes already support compression. So that point by itself is moot.

Mercurius84

Hi, Actually our company wants to develop something similar to COLD system. But in generically .... we are building a new product based to this concept....

jschell

Mercurius84 wrote:

similar to COLD system.

No idea what that is but if a static storage system then your stated requirements still are not anywhere close to being reasonable for creating and marketing such a system. Just to make it clear again, your stated requirements do not present a need for anything. I can only presume that either you work in an highly unusual environment (an unstated requirement) or the stated file count\size does not span that actual need (another unstated requirement.)

Mercurius84

understand. thanks