File Size Limits
-
Im trying to read a large text file with a CStdioFile object (the file's around 2.3 Gb, OS Windows XP, File System NTFS). After reading for a while an exception is thrown during a CStdioFile::ReadString call which says something like "Controler Invalid" I simplified the code so that its doing nothing more than loopìng around the ReadString call, and the error persists. Could there be an inherent limit on the size of a file readable using CStdioFile (I dont belive that 2.3 Gb is too big for NTFS ?!?) Thanks, Dave
-
You may be running into some unexpected 32bit dword limitations, not necessarily a file os problem, but a file pointer limitation.
-
Im trying to read a large text file with a CStdioFile object (the file's around 2.3 Gb, OS Windows XP, File System NTFS). After reading for a while an exception is thrown during a CStdioFile::ReadString call which says something like "Controler Invalid" I simplified the code so that its doing nothing more than loopìng around the ReadString call, and the error persists. Could there be an inherent limit on the size of a file readable using CStdioFile (I dont belive that 2.3 Gb is too big for NTFS ?!?) Thanks, Dave
Are you using VC6? if so then the file pointer used is a 32 bit value, with a limit of 2GB. AFAIK if you are using VC7+ then the file pointer is a 64 bit value. If you want to get around this limit in VC6 then do not use the MFC classes (which wrap the fopen, fread, etc CRT functions) and use the SDK functions CreateFile, ReadFile etc.
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04 "There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05 Within you lies the power for good - Use it!
-
Are you using VC6? if so then the file pointer used is a 32 bit value, with a limit of 2GB. AFAIK if you are using VC7+ then the file pointer is a 64 bit value. If you want to get around this limit in VC6 then do not use the MFC classes (which wrap the fopen, fread, etc CRT functions) and use the SDK functions CreateFile, ReadFile etc.
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04 "There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05 Within you lies the power for good - Use it!
-
Are you using VC6? if so then the file pointer used is a 32 bit value, with a limit of 2GB. AFAIK if you are using VC7+ then the file pointer is a 64 bit value. If you want to get around this limit in VC6 then do not use the MFC classes (which wrap the fopen, fread, etc CRT functions) and use the SDK functions CreateFile, ReadFile etc.
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04 "There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05 Within you lies the power for good - Use it!
PJ Arends wrote: do not use the MFC classes (which wrap the fopen, fread, etc CRT functions) Seriously? I always though that MFC didn't wrap CRT functions, but rather the WIN API directly! The CRT file API's (I don't know about the others) are just a layer on top of the OS's API's, which are CreateFile, ReadFile, etc.. I don't know about the MFC shipped with VC6, but the newer MFC version 8, does wrap the window's API's... Behind every great black man... ... is the police. - Conspiracy brother Blog[^]
-
PJ Arends wrote: do not use the MFC classes (which wrap the fopen, fread, etc CRT functions) Seriously? I always though that MFC didn't wrap CRT functions, but rather the WIN API directly! The CRT file API's (I don't know about the others) are just a layer on top of the OS's API's, which are CreateFile, ReadFile, etc.. I don't know about the MFC shipped with VC6, but the newer MFC version 8, does wrap the window's API's... Behind every great black man... ... is the police. - Conspiracy brother Blog[^]
From VC 6.0 MFC Source (FILECORE.CPP Line 111) Yes it wraps Win32 API, not standard C runtime calls...
BOOL CFile::Open(LPCTSTR lpszFileName, UINT nOpenFlags, CFileException* pException) { ... // attempt file creation HANDLE hFile = ::CreateFile(lpszFileName, dwAccess, dwShareMode, &sa, dwCreateFlag, FILE_ATTRIBUTE_NORMAL, NULL); if (hFile == INVALID_HANDLE_VALUE) { if (pException != NULL) { pException->m_lOsError = ::GetLastError(); pException->m_cause = CFileException::OsErrorToException(pException->m_lOsError); // use passed file name (not expanded vesion) when reporting // an error while opening pException->m_strFileName = lpszFileName; } return FALSE; } m_hFile = (HFILE)hFile; m_bCloseOnDelete = TRUE; -
From VC 6.0 MFC Source (FILECORE.CPP Line 111) Yes it wraps Win32 API, not standard C runtime calls...
BOOL CFile::Open(LPCTSTR lpszFileName, UINT nOpenFlags, CFileException* pException) { ... // attempt file creation HANDLE hFile = ::CreateFile(lpszFileName, dwAccess, dwShareMode, &sa, dwCreateFlag, FILE_ATTRIBUTE_NORMAL, NULL); if (hFile == INVALID_HANDLE_VALUE) { if (pException != NULL) { pException->m_lOsError = ::GetLastError(); pException->m_cause = CFileException::OsErrorToException(pException->m_lOsError); // use passed file name (not expanded vesion) when reporting // an error while opening pException->m_strFileName = lpszFileName; } return FALSE; } m_hFile = (HFILE)hFile; m_bCloseOnDelete = TRUE; -
I guess I should read the MFC source instead of MSDN[^] [quote] A CStdioFile object represents a C run-time stream file as opened by the run-time function fopen. [/quote]
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04 "There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05 Within you lies the power for good - Use it!
-
I guess I should read the MFC source instead of MSDN[^] [quote] A CStdioFile object represents a C run-time stream file as opened by the run-time function fopen. [/quote]
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04 "There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05 Within you lies the power for good - Use it!
-
I guess I should read the MFC source instead of MSDN[^] [quote] A CStdioFile object represents a C run-time stream file as opened by the run-time function fopen. [/quote]
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04 "There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05 Within you lies the power for good - Use it!
Well, I DID look at the CFile code. Who knows, maybe the CStdioFile object does something different? Not likely, but this IS MFC ... :omg:
-
I guess I should read the MFC source instead of MSDN[^] [quote] A CStdioFile object represents a C run-time stream file as opened by the run-time function fopen. [/quote]
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04 "There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05 Within you lies the power for good - Use it!
Generally what I think that this is referring to is that this object *IS* the C run time and behaves the same. In the open it does call CFile::Open:
0:000> u MFC42!CStdioFile::Open
MFC42!CStdioFile::Open:
73e2c96c 8bff mov edi,edi
73e2c96e 55 push ebp
73e2c96f 8bec mov ebp,esp
73e2c971 53 push ebx
73e2c972 8b5d0c mov ebx,[ebp+0xc]
73e2c975 56 push esi
73e2c976 57 push edi
73e2c977 ff7510 push dword ptr [ebp+0x10]
0:000> u
MFC42!CStdioFile::Open+0xe:
73e2c97a 8bc3 mov eax,ebx
73e2c97c 25ffbfffff and eax,0xffffbfff
73e2c981 50 push eax
73e2c982 ff7508 push dword ptr [ebp+0x8]
73e2c985 8bf9 mov edi,ecx
73e2c987 33f6 xor esi,esi
73e2c989 897710 mov [edi+0x10],esi
73e2c98c e8e043fbff call MFC42!CFile::Open (73de0d71)And that calls CreateFile, however right afterwards they do create a standard handle:
73e2c9db 75d2 jnz MFC42!CStdioFile::Open+0x43 (73e2c9af)
73e2c9dd ebd7 jmp MFC42!CStdioFile::Open+0x4a (73e2c9b6)
73e2c9df c644050c74 mov byte ptr [ebp+eax+0xc],0x74
73e2c9e4 40 inc eax
73e2c9e5 51 push ecx
0:000>
MFC42!CStdioFile::Open+0x7a:
73e2c9e6 ff7704 push dword ptr [edi+0x4]
73e2c9e9 c644050c00 mov byte ptr [ebp+eax+0xc],0x0
73e2c9ee ff150467e773 call dword ptr [MFC42!_imp___open_osfhandle (73e76704)]
73e2c9f4 83f8ff cmp eax,0xffffffff
73e2c9f7 59 pop ecx
73e2c9f8 59 pop ecxLook, most other functions are wrappers around C runtime, as an example, look here:
0:000> u MFC42!CStdioFile::WriteString
MFC42!CStdioFile::WriteString:
73e2cb01 8bff mov edi,edi
73e2cb03 56 push esi
73e2cb04 8bf1 mov esi,ecx
73e2cb06 ff7610 push dword ptr [esi+0x10] <-- FILE *
73e2cb09 ff74240c push dword ptr [esp+0xc]
73e2cb0d ff151867e773 call dword ptr [MFC42!_imp__fputs (73e76718)]
73e2cb13 83f8ff cmp eax,0xffffffffIn WriteString, fputs is called and it gets the FILE * from the "this" pointer at offset 16. So, CStdioFile calls some standard APIs, some C runtime APIs and it in itself essentially re-implements some of the functionality that would have been done
-
Im trying to read a large text file with a CStdioFile object (the file's around 2.3 Gb, OS Windows XP, File System NTFS). After reading for a while an exception is thrown during a CStdioFile::ReadString call which says something like "Controler Invalid" I simplified the code so that its doing nothing more than loopìng around the ReadString call, and the error persists. Could there be an inherent limit on the size of a file readable using CStdioFile (I dont belive that 2.3 Gb is too big for NTFS ?!?) Thanks, Dave
Yes there is a limit. We can normaly ignore those, but we can not ignore the amount of memory we have for storage. Where are you copying the data to? INTP "The more help VB provides VB programmers, the more miserable your life as a C++ programmer becomes." Andrew W. Troelsen