File copy buffer size?
-
I am creating an application that copies files from one location to another sometimes locally and sometimes over a network. I am using buffered streams to accomplish desired functionality and it is working as expected but I am unsure on what size my buffer should be. I have experimented with different sizes and I have had varied results. Recap: What is an optimal size for the buffer when copying files? Note: File size varies from 1 byte to 1.5Gb never more.
When the game is over the Pawn and the King go into the same box.
-
I am creating an application that copies files from one location to another sometimes locally and sometimes over a network. I am using buffered streams to accomplish desired functionality and it is working as expected but I am unsure on what size my buffer should be. I have experimented with different sizes and I have had varied results. Recap: What is an optimal size for the buffer when copying files? Note: File size varies from 1 byte to 1.5Gb never more.
When the game is over the Pawn and the King go into the same box.
MicealG wrote:
What is an optimal size for the buffer when copying files?
Depends. Optimal size for reading/writing to/from Disk is based on the OS and hardware I/O specifications. Optimal size for transporting network packets is based on network protocols and hardware however the network stack you are using, like TCP/IP does that buffering for you so you don't deal with that issue.
led mike
-
MicealG wrote:
What is an optimal size for the buffer when copying files?
Depends. Optimal size for reading/writing to/from Disk is based on the OS and hardware I/O specifications. Optimal size for transporting network packets is based on network protocols and hardware however the network stack you are using, like TCP/IP does that buffering for you so you don't deal with that issue.
led mike
-
Thanks for the reply. Is there any way I could work out an optimal buffer size to use depending on the file size?
When the game is over the Pawn and the King go into the same box.
MicealG wrote:
Is there any way I could work out an optimal buffer size to use depending on the file size?
No. It has nothing to do with the file size. The networking software/hardware determines the optimal buffer size. For Disk I/O it is dependent on the Hardware / OS File System combination. Always use an exponent of 2 since the systems do. When testing Disks I have found that 2k or 4k will prove out as the optimal size. I have never looked at network transfers but if you are doing Disk I/O it seems that should be your determining factor.
led mike
-
I am creating an application that copies files from one location to another sometimes locally and sometimes over a network. I am using buffered streams to accomplish desired functionality and it is working as expected but I am unsure on what size my buffer should be. I have experimented with different sizes and I have had varied results. Recap: What is an optimal size for the buffer when copying files? Note: File size varies from 1 byte to 1.5Gb never more.
When the game is over the Pawn and the King go into the same box.
The streams are buffered by the file system and the network layer, so a buffer smaller than one cluster or smaller than one network packet, doesn't have any buffering effect at all. Other than that, the buffer size shouldn't matter that much. Have you tried using two buffers and asynchronous reading and writing, so that you read to one buffer while writing from the other?
Despite everything, the person most likely to be fooling you next is yourself.
-
The streams are buffered by the file system and the network layer, so a buffer smaller than one cluster or smaller than one network packet, doesn't have any buffering effect at all. Other than that, the buffer size shouldn't matter that much. Have you tried using two buffers and asynchronous reading and writing, so that you read to one buffer while writing from the other?
Despite everything, the person most likely to be fooling you next is yourself.
-
I am creating an application that copies files from one location to another sometimes locally and sometimes over a network. I am using buffered streams to accomplish desired functionality and it is working as expected but I am unsure on what size my buffer should be. I have experimented with different sizes and I have had varied results. Recap: What is an optimal size for the buffer when copying files? Note: File size varies from 1 byte to 1.5Gb never more.
When the game is over the Pawn and the King go into the same box.
What the others have said is true - it depends on many things. However - even though an optimal buffer size can only be determined by implementing a complex (and thus error-prone - just look at Vista) self-tuning algorithm of some kind, I guess you're just asking for an arbitrary but "reasonable" value. Correct? I've used 64K (65536) bytes for the last five years or so. Equals the maximum stripe size of popular RAID controllers, the maximum block size of popular tape devices and the default maximum window size of the Windows TCP stack (pre-RSS). The world is changing, though, so I'm considering reconsidering that. Whenever I have a budget allowing finer points, I make buffer sizes configurable (still with a 64K default, though). On occasion I've experimented with 256K or 512K and it seems to work OK as well. Being old enough to remember computers with less RAM than that, it always gives me an eerie feeling - but given the RAM in computers of today, 512KB is really not a lot of memory to set aside for a buffer. In any case: Like someone said, powers of two still feel like the right thing to use. Actually moot when it comes to network traffic (TCP couldn't care less), but disks still stick to that for block sizes, allocation units and stripe sizes. Later,
-- Peter
-
What the others have said is true - it depends on many things. However - even though an optimal buffer size can only be determined by implementing a complex (and thus error-prone - just look at Vista) self-tuning algorithm of some kind, I guess you're just asking for an arbitrary but "reasonable" value. Correct? I've used 64K (65536) bytes for the last five years or so. Equals the maximum stripe size of popular RAID controllers, the maximum block size of popular tape devices and the default maximum window size of the Windows TCP stack (pre-RSS). The world is changing, though, so I'm considering reconsidering that. Whenever I have a budget allowing finer points, I make buffer sizes configurable (still with a 64K default, though). On occasion I've experimented with 256K or 512K and it seems to work OK as well. Being old enough to remember computers with less RAM than that, it always gives me an eerie feeling - but given the RAM in computers of today, 512KB is really not a lot of memory to set aside for a buffer. In any case: Like someone said, powers of two still feel like the right thing to use. Actually moot when it comes to network traffic (TCP couldn't care less), but disks still stick to that for block sizes, allocation units and stripe sizes. Later,
-- Peter
PeterTheSwede wrote:
On occasion I've experimented with 256K or 512K and it seems to work OK as well.
On consideration here is that an object larger than 85 kB will be allocated on the large objects heap instead of the regular heap. The large objects heap is not compacted in the way that the regular heap is, so the application might not be able to return that memory to the system when you have released the buffer. Not a very serious concern as it's not very much memory, but it might be good to know.
PeterTheSwede wrote:
Being old enough to remember computers with less RAM than that
Those were the days... My first programming was done on a Sinclair ZX81 with 4 kB of memory. My first own computer was an Atari 600XL with 16 kb of memory. After that I got an Atari 130XE with massive 128 kB of memory. It had a special chip for swapping memory banks, as the CPU wasn't able to address more than 64 KB of memory... :)
Despite everything, the person most likely to be fooling you next is yourself.
-
What the others have said is true - it depends on many things. However - even though an optimal buffer size can only be determined by implementing a complex (and thus error-prone - just look at Vista) self-tuning algorithm of some kind, I guess you're just asking for an arbitrary but "reasonable" value. Correct? I've used 64K (65536) bytes for the last five years or so. Equals the maximum stripe size of popular RAID controllers, the maximum block size of popular tape devices and the default maximum window size of the Windows TCP stack (pre-RSS). The world is changing, though, so I'm considering reconsidering that. Whenever I have a budget allowing finer points, I make buffer sizes configurable (still with a 64K default, though). On occasion I've experimented with 256K or 512K and it seems to work OK as well. Being old enough to remember computers with less RAM than that, it always gives me an eerie feeling - but given the RAM in computers of today, 512KB is really not a lot of memory to set aside for a buffer. In any case: Like someone said, powers of two still feel like the right thing to use. Actually moot when it comes to network traffic (TCP couldn't care less), but disks still stick to that for block sizes, allocation units and stripe sizes. Later,
-- Peter
-
PeterTheSwede wrote:
On occasion I've experimented with 256K or 512K and it seems to work OK as well.
On consideration here is that an object larger than 85 kB will be allocated on the large objects heap instead of the regular heap. The large objects heap is not compacted in the way that the regular heap is, so the application might not be able to return that memory to the system when you have released the buffer. Not a very serious concern as it's not very much memory, but it might be good to know.
PeterTheSwede wrote:
Being old enough to remember computers with less RAM than that
Those were the days... My first programming was done on a Sinclair ZX81 with 4 kB of memory. My first own computer was an Atari 600XL with 16 kb of memory. After that I got an Atari 130XE with massive 128 kB of memory. It had a special chip for swapping memory banks, as the CPU wasn't able to address more than 64 KB of memory... :)
Despite everything, the person most likely to be fooling you next is yourself.
Hi, On the "large objects heap": I have managed to miss that somehow. Thanks a lot for the enlightenment! Will keep it in mind for the future. On RAM sizes: My first programming was done on a Norsk Data ND-100 minicomputer with 512KB RAM (IIRC) and an 80MB disk running the SINTRAN-III OS. It had an ND-50 number-cruncher for a side-car (same one that did the F-16 simulations a few years earlier). We were approximately 200 student groups writing, compiling and testing BASIC and Pascal programs on it (50-60 groups at a time, on DEC VT-100 and ND terminals using current loop cables from the terminal room to the computer room). I ended up being an assistant teacher there for a few years... we're talking 1981 or so. My first experience with a PC: - Hmmm... wonder if it's like Unix: C:\>ls - No, let's try SINTRAN III syntax: C:\>LIST-FILES - Nope, not that. Let's do VMS then: C:\>DIR - WOW, it worked. Let's run it in the background: C:\>BG DIR - Nope. Maybe some Unix influence after all: C:\>DIR & - Not that. Now let's see... where's the manual. WHAT?!?! No multi-tasking?!?!? Are you f*ing serious?!?! And I had just promised writing actual software for the thing... I have never felt so lost in my life... Later,
-- Peter