read a percent from a file
-
Hi people, Could anyone point me to some code that loads any file in 10% pieces. i.e. load first 25% of file content, then the next 25% and so on till 100%? I'm sure this has been done before...... TIA.
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdev -
Hi people, Could anyone point me to some code that loads any file in 10% pieces. i.e. load first 25% of file content, then the next 25% and so on till 100%? I'm sure this has been done before...... TIA.
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdevYou mean something like:
DWORD dwFileSize = ::GetFileSize( hTheFile, NULL ); DWORD dw10Percent = ( dwFileSize *.10 ); BYTE *pBuffer = new BYTE[ dw10Percent ]; if( !pBuffer ) { return( ERROR_NOT_ENOUGH_MEMORY ); } DWORD dwStatus = 0; DWORD dwRead = 0; dwStatus = ::ReadFile( hTheFile, pBuffer, dw10Percent, &dwRead, NULL ); if( dwStatus != ERROR_SUCCESS ) // // ... //Peace! -=- James (Sonork:100.21837) "There is nothing worse than being oblivious to the fact that you do not know what you are doing." [Get Check Favorites 1.5 Now!] -
You mean something like:
DWORD dwFileSize = ::GetFileSize( hTheFile, NULL ); DWORD dw10Percent = ( dwFileSize *.10 ); BYTE *pBuffer = new BYTE[ dw10Percent ]; if( !pBuffer ) { return( ERROR_NOT_ENOUGH_MEMORY ); } DWORD dwStatus = 0; DWORD dwRead = 0; dwStatus = ::ReadFile( hTheFile, pBuffer, dw10Percent, &dwRead, NULL ); if( dwStatus != ERROR_SUCCESS ) // // ... //Peace! -=- James (Sonork:100.21837) "There is nothing worse than being oblivious to the fact that you do not know what you are doing." [Get Check Favorites 1.5 Now!]Thanks,James. But for large files, what if enough memory cannot be allocated? Can a smaller buffer(say 5000 chars) be reused?
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdev -
Thanks,James. But for large files, what if enough memory cannot be allocated? Can a smaller buffer(say 5000 chars) be reused?
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdevIf you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files. My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
-
If you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files. My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
-
If you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files. My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
Thomas George wrote: If you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files. A small note: using memory mapped files alone does not guarantee that the entire file will (or will not) be mapped into your address space: the amount of the file mapped in can be specified by the user, so that you can map 10KB chunks into your address space as needed, if desired. If you try to map a large enough file (or part of a file), you can still encounter a out-of-memory-type situation, because you need a contigous range of addresses large enough for the size of the mapping. Paging, however, will still occur as-needed. Peace! -=- James (Sonork:100.21837) "There is nothing worse than being oblivious to the fact that you do not know what you are doing." [Get Check Favorites 1.5 Now!]
-
Thomas George wrote: If you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files. A small note: using memory mapped files alone does not guarantee that the entire file will (or will not) be mapped into your address space: the amount of the file mapped in can be specified by the user, so that you can map 10KB chunks into your address space as needed, if desired. If you try to map a large enough file (or part of a file), you can still encounter a out-of-memory-type situation, because you need a contigous range of addresses large enough for the size of the mapping. Paging, however, will still occur as-needed. Peace! -=- James (Sonork:100.21837) "There is nothing worse than being oblivious to the fact that you do not know what you are doing." [Get Check Favorites 1.5 Now!]
As I understand it, if the mapping is unnamed, it is mapped into the local address space of the application and is not in the global space. So, the chances of running out of memory is remote - unless you have a 2 GB file. In that case, you are not in a place where generic solutions work. I am not sure whether I am 100% correct on this. Please feel free to correct me if I am wrong. :-) My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
-
As I understand it, if the mapping is unnamed, it is mapped into the local address space of the application and is not in the global space. So, the chances of running out of memory is remote - unless you have a 2 GB file. In that case, you are not in a place where generic solutions work. I am not sure whether I am 100% correct on this. Please feel free to correct me if I am wrong. :-) My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
[Edit: this had an undesired aggressive tone] I do not believe so: specifying a name only makes the name of the file mapping object (a kernel object) available for others processes to open. Regardless of it bring named or unnamed, it is always mapped into the address space of the calling process. However, in the case of a named object, the two (or more) different processes share the same physical memory (pages), and physical disk space (the file). You are correct in saying that (in general) you need a pretty large file to fail, unless your physical address range is fragmented, or you specified a starting address that does not have the contigous range available. BTW: a process' address space is normally 2GB (unless you booted with /3GB). Peace! -=- James (Sonork:100.21837) "There is nothing worse than being oblivious to the fact that you do not know what you are doing." [Get Check Favorites 1.5 Now!]
-
[Edit: this had an undesired aggressive tone] I do not believe so: specifying a name only makes the name of the file mapping object (a kernel object) available for others processes to open. Regardless of it bring named or unnamed, it is always mapped into the address space of the calling process. However, in the case of a named object, the two (or more) different processes share the same physical memory (pages), and physical disk space (the file). You are correct in saying that (in general) you need a pretty large file to fail, unless your physical address range is fragmented, or you specified a starting address that does not have the contigous range available. BTW: a process' address space is normally 2GB (unless you booted with /3GB). Peace! -=- James (Sonork:100.21837) "There is nothing worse than being oblivious to the fact that you do not know what you are doing." [Get Check Favorites 1.5 Now!]
you are right. The file is always mapped into the process address space, which is 2 GB. But since the mapping is to a logical process address space, i do not think that there will be a failure to map a view, if the physical address space is fragmented. AFAIK, It will just cause more page faults, because each logical page can be mapped to a different physical page. I am saying this because I have mapped 512 MB on a machine with 512 MB RAM, obviously, there was not enough physical memory to do that. In any case, the idea is to be able to use more memory than available physically. Thomas My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
-
you are right. The file is always mapped into the process address space, which is 2 GB. But since the mapping is to a logical process address space, i do not think that there will be a failure to map a view, if the physical address space is fragmented. AFAIK, It will just cause more page faults, because each logical page can be mapped to a different physical page. I am saying this because I have mapped 512 MB on a machine with 512 MB RAM, obviously, there was not enough physical memory to do that. In any case, the idea is to be able to use more memory than available physically. Thomas My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
Thomas George wrote: [...] if the physical address space is fragmented. AFAIK, It will just cause more page faults, because each logical page can be mapped to a different physical page Correct. I said "physical address", when I should have said "virtual address", because you need a contigious range of addresses in your virtual address range to satisfy the size of the view. Peace! -=- James (Sonork:100.21837) [Tip for SUV winter driving survival: "Professional Driver on Closed Course" does not mean "your Dumb Ass on a Public Road"!]
[Get Check Favorites 1.5 Now!] -
If you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files. My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
Supposing our discussion was only limited to text files, moderately large to small text files, would such buffer allocation at one shot be of some advantage? what are the general rules for handling text files?
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdev -
You mean something like:
DWORD dwFileSize = ::GetFileSize( hTheFile, NULL ); DWORD dw10Percent = ( dwFileSize *.10 ); BYTE *pBuffer = new BYTE[ dw10Percent ]; if( !pBuffer ) { return( ERROR_NOT_ENOUGH_MEMORY ); } DWORD dwStatus = 0; DWORD dwRead = 0; dwStatus = ::ReadFile( hTheFile, pBuffer, dw10Percent, &dwRead, NULL ); if( dwStatus != ERROR_SUCCESS ) // // ... //Peace! -=- James (Sonork:100.21837) "There is nothing worse than being oblivious to the fact that you do not know what you are doing." [Get Check Favorites 1.5 Now!]Supposing our discussion was only limited to text files, moderately large to small text files, would such buffer allocation at one shot be of some advantage? what are the general rules for handling text files?
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdev -
Supposing our discussion was only limited to text files, moderately large to small text files, would such buffer allocation at one shot be of some advantage? what are the general rules for handling text files?
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdevI would think that you can handle small files by allocating memory in the heap. You will need some representation of the file in memory, if the program is like a text editor, which will be written to the file, when saved. My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
-
I would think that you can handle small files by allocating memory in the heap. You will need some representation of the file in memory, if the program is like a text editor, which will be written to the file, when saved. My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers
What if my files were small to large emails? I would guess such allocation would be fine way of doing things?
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdev -
Supposing our discussion was only limited to text files, moderately large to small text files, would such buffer allocation at one shot be of some advantage? what are the general rules for handling text files?
_'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.
- Marvin, the robot.
_Amit Dey sonork: 100:18407 msn: visualcdevIMHO, only if you were making changes to the file. Peace! -=- James (Sonork:100.21837) [Tip for SUV winter driving survival: "Professional Driver on Closed Course" does not mean "your Dumb Ass on a Public Road"!]
[Get Check Favorites 1.5 Now!] -
IMHO, only if you were making changes to the file. Peace! -=- James (Sonork:100.21837) [Tip for SUV winter driving survival: "Professional Driver on Closed Course" does not mean "your Dumb Ass on a Public Road"!]
[Get Check Favorites 1.5 Now!]