Appropriate file buffer size
-
I'm writing a file manipulation program, and was wondering in your experience what is the most efficient buffer size to load at a time and manipulate. Would this simply be based on the largest chunk of memory I am willing to allocate on the system, or can smaller buffers produce faster results? Speed is important because I could be converting several thousand files at a time. Thanks, Dustin
-
I'm writing a file manipulation program, and was wondering in your experience what is the most efficient buffer size to load at a time and manipulate. Would this simply be based on the largest chunk of memory I am willing to allocate on the system, or can smaller buffers produce faster results? Speed is important because I could be converting several thousand files at a time. Thanks, Dustin
How much overhead would it be to first iterate the list of files, look for the largest file, and allocate that much memory?
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
How much overhead would it be to first iterate the list of files, look for the largest file, and allocate that much memory?
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
The biggest problem would not neccesarily be the overhead, but the allocation itself. What happens if I run accross a 2GB file? I'm actually attemting to write an encryption program, more as an experiment than anything, and do not know the make up of the files I will be opening.
-
The biggest problem would not neccesarily be the overhead, but the allocation itself. What happens if I run accross a 2GB file? I'm actually attemting to write an encryption program, more as an experiment than anything, and do not know the make up of the files I will be opening.
Dustin Henry wrote:
What happens if I run accross a 2GB file?
Indeed that would be a problem.
Dustin Henry wrote:
I'm actually attemting to write an encryption program...
What does the encryption algorithm require (in terms of input)?
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
Dustin Henry wrote:
What happens if I run accross a 2GB file?
Indeed that would be a problem.
Dustin Henry wrote:
I'm actually attemting to write an encryption program...
What does the encryption algorithm require (in terms of input)?
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
I am actually using my own random number generator to randomly convert each byte of data based on an initial seed, so the only requirement is that I process the data in order.
-
I am actually using my own random number generator to randomly convert each byte of data based on an initial seed, so the only requirement is that I process the data in order.
Then I don't see why you couldn't allocate a couple of megabytes for that.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
Then I don't see why you couldn't allocate a couple of megabytes for that.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
So basically the bigger the better then?
-
So basically the bigger the better then?
To a point, but you don't want to get so big that you introduce more disk thrashing. Allocating several MB of memory is not usually noticeable with semi-modern machines.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
To a point, but you don't want to get so big that you introduce more disk thrashing. Allocating several MB of memory is not usually noticeable with semi-modern machines.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
Great, thanks for your help David.