Storing huge numbers of files
-
Why would the size of the files matter? Very few are small enough to fit in the available space of the directory entry. Yes, they are files, by definition. Mostly, new files are added to the directory. This is the most common access. File access is far more infrequent.
Why would the size of the files matter? -- Because if the "files" are small enough, sticking them in some other cataloging system might be a better idea. Maybe a database, maybe a custom archiving system. Think of things like version control systems. File access is far more infrequent. -- Then just do whatever you want, it won't matter.
-
Can you provide an explanation of why it would be that way? Or is it at the "gut feeling" level?
-
I can tell you from experience that Windows does not do well with thousands of files in a single directory. You will be much better off distributing them over many sub-dirs. Off Topic: I think it's time for you to choose a user name instead of Member 7989122. :)
The difficult we do right away... ...the impossible takes slightly longer.
Richard Andrew x64 wrote:
I think it's time for you to choose a user name instead of Member 7989122.
I don't know; see Peanuts for [30-Sep-1963 to 04-Oct-1963](https://peanuts.fandom.com/wiki/September\_1963\_comic\_strips). :)
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows. -- 6079 Smith W.
-
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
Member 7989122 wrote:
If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage?
If performance downgrade with number of files in a directory, there is only 1 explanation: The directory is organized as a flat list of files, unsorted. This imply that to find a file, you have to scan the list/directory sequentially. In O(n). If an OS can have the directory sorted in the order you look for (file name), cost of finding a file is in O(log(n))
Patrice “Everything should be made as simple as possible, but no simpler.” Albert Einstein
-
I can tell you from experience that Windows does not do well with thousands of files in a single directory. You will be much better off distributing them over many sub-dirs. Off Topic: I think it's time for you to choose a user name instead of Member 7989122. :)
The difficult we do right away... ...the impossible takes slightly longer.
Don't rush him. It's only been a little more than 9 years. He probably needs time to think of one... ;)
Anything that is unrelated to elephants is irrelephant
Anonymous
-----
The problem with quotes on the internet is that you can never tell if they're genuine
Winston Churchill, 1944
-----
Never argue with a fool. Onlookers may not be able to tell the difference.
Mark Twain -
Can you provide an explanation of why it would be that way? Or is it at the "gut feeling" level?
I can't explain why that is, but it's quite simple to test. Write a small piece of code that copies an image file into the same directory multiple times. Doesn't have to be 100.000, I think 10-20.000 will suffice. Then try to open that directory with Explorer. That'll give you an idea about the problem. :sigh:
Anything that is unrelated to elephants is irrelephant
Anonymous
-----
The problem with quotes on the internet is that you can never tell if they're genuine
Winston Churchill, 1944
-----
Never argue with a fool. Onlookers may not be able to tell the difference.
Mark Twain -
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
Maximum number of files on disk: 4,294,967,295 As already mentioned, the problems will start when you try to browse the disk in question with pretty much any existing application. A better option would be to put the files in a database as blobs. At that point, you'll only have one file on the disk for the database itself. It wouldf also be easier to organize and manage than a complex folder hierarchy.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
We have some directories that contain that big number of files, the record I can remember right now is around 450k files in a folder. They come from long time meassurements that trigger a data file a between 3 and 5 in a minute, each between 1 and 5 Mb. Accessing the directory is slow, changing the order from name to timestamp is slow, moving the directory to another place is slow, getting the properties of the folder is slow, deleting the folder once is not needed anymore is slow. Windwos 10 even slower specially the "folder properties" it needs over 15 minutes to count the files and give the size of the folder. Windows 7 did it in 30 or 40 seconds. We can't move that to FAT drives, due to number limitations as other said. Need to be NFTS.
M.D.V. ;) If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about? Help me to understand what I'm saying, and I'll explain it better to you Rating helpful answers is nice, but saying thanks can be even nicer.
-
I hope to persue most users to go for NTFS rather than FAT32. The most common access will be through an application, which will read the directory programmatically. Windows Explorer access can be considered an exception (although not that exceptional!).
Windows explorer will be your bottleneck ... while you sit and wait while it "builds" a 100k tree view. Odds are, it will "hang". "Reading" directories is not a big deal; how you "display" them is.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it. ― Confucian Analects: Rules of Confucius about his food
-
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
What are you saving the files for? How will you access them? And how will you search for them? One at a time, sequential, by date, by name...?
Wrong is evil and must be defeated. - Jeff Ello Never stop dreaming - Freddie Kruger
-
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
Neither Windows nor Linux do well when putting too many files in a single folder. I've tried it with a million files, it is very painful. Some operations, like simply listing the directory, or even trying to delete the files take absurdly long. It seems to be doing some operations that are simply not designed for large numbers of files. Like said, around 10,000 files in a folder is a reasonable max. I simply make it 1,000. So for a million files, spread them across 1,000 folders. There is a nice symmetry here, and it works like a charm.
-
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
A few years ago I worked on a system that generates around 50.000 to 100.000 files a day. We ran in trouble right away. Storing the files was not a problem, but retrieving them was impossible. And a second problem was that we needed to search the contents of the files to find all files with a certain string in the text. We eventually choose to store all files in a database. This was quite easy because the files were small. (Less than 10K) We choose an Oracle database because of the CLOB datatype. (it allows for indexing and searching) We had no problems since and have more the 200 million files.:cool:
-
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
I worked on a system that had to stream 1MB images to disk at 75fps. I found that once there were about 700 files in a directory, creating new files suddenly became slower and the required transfer rate was unachievable. I ended up creating a new subdirectory every 500 files. Of course this won't be a problem if your system is purely for archive.
-
Windows explorer will be your bottleneck ... while you sit and wait while it "builds" a 100k tree view. Odds are, it will "hang". "Reading" directories is not a big deal; how you "display" them is.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it. ― Confucian Analects: Rules of Confucius about his food
-
Can you provide an explanation of why it would be that way? Or is it at the "gut feeling" level?
Probably because a windows folder isn't designed to contained 10,000 files, unlike database table which is expected to contain millions of rows. Or spreadsheets. When we browse into a folder using windows explorer, it tried to read all file names inside that folder. There's no virtualization or partial loading. Reading 10,000 file names and extensions is surely detrimental. EDIT : it's probably fine as long as you don't browse it using any explorer view.
-
Can you provide an explanation of why it would be that way? Or is it at the "gut feeling" level?
-
Why would the size of the files matter? Very few are small enough to fit in the available space of the directory entry. Yes, they are files, by definition. Mostly, new files are added to the directory. This is the most common access. File access is far more infrequent.
-
File-access; so, mostly reading "files"? A database would give you the most flexibility and performance. --edit You can easily expand Sql Server over multiple servers if need be, with more control over sharding and backups than with a regular filesystem.
Bastard Programmer from Hell :suss: If you can't read my code, try converting it here[^] "If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
-
People don’t relate well to numbers and this is a place where camaraderie is important. A name - even an obvious alias will make the interactions more personable. ;)
If you can't laugh at yourself - ask me and I will do it for you.
-
This is about file systems in general, although with a primary emphasis on NTFS: If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?` If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage? Is this differnent e.g among differnt FAT variants, and with NTFS?
I don't know about access issues for a large number of files in a directory, but you might also consider security issues. If, for example, you have several different users whose files should not be accessible by the others, creating a subfolder for each user might allow you to secure them such that only their user has access to their subfolder (plus maybe some 'admin' user that you use which can see all directories). Obvious organizational advantages as well.
