::FindFirstFile problem [modified]

Loreia

I am writing a small article about file enumeration, and in one of my CppUnit test I *discovered* that FindFirstFile/FindNextFile won't correctly report file attributes on FAT32 and FAT16. Specifically, FILE_ATTRIBUTE_TEMPORARY attribute is the problem here: Here is what my code does:

// first I set desired attribute
SetFileAttributes(sTestFile.c_str(), FILE_ATTRIBUTE_TEMPORARY);

// then I read attributes with GetFileAttributes
DWORD attr_1 = GetFileAttributes(sTestFile.c_str());
// attr_1 is FILE_ATTRIBUTE_TEMPORARY on all three FS tested (NTFS, FAT32 and FAT16)

// then I read attributes with FindFirstFile
FindFirstFile(sTestFile.c_str(), &fd);

// now I get that fd.dwFileAttributes equals FILE_ATTRIBUTE_NORMAL on FAT32 and FAT16, but FILE_ATTRIBUTE_TEMPORARY on NTFS !!!!

It took me nearly a day to realize that problem is due to File system and not my code. And now after spending hours going through on-line documentation I decided to ask here: 1. Is this documented behavior or a bug? 2. Am I doing something wrong in my code. Maybe it is something about temporary files that I failed to take into account. Any help would be highly appreciated. I have tested this on two computers, both running XP, and I get same results on both. Finally, here is a simple console project that demonstrates the problem:

#include "tchar.h"
#include "Windows.h"
#include
#include

using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{
basic_string sPath = _T("");
basic_string sTestFile = _T("\\test_file.txt");
TCHAR root[256] = {0};
DWORD size = 256;
WIN32_FIND_DATA fd;

if (::GetCurrentDirectory(size, root) != 0)
    sPath = root;
else return 1;

sPath += sTestFile;

CreateFile(
			sPath.c\_str(), 
			GENERIC\_READ | GENERIC\_WRITE, 
			FILE\_SHARE\_READ, 
			NULL, 
			CREATE\_ALWAYS, 
			FILE\_ATTRIBUTE\_TEMPORARY, 
			NULL);	// creates file with two attributes set: FILE\_ATTRIBUTE\_ARCHIVE and FILE\_ATTRIBUTE\_TEMPORARY

\_tprintf(\_T("\\nA file %s is created...\\n"), sPath.c\_str());
\_tprintf(\_T("\\nFile attributes (as reported by GetFileAttributes() ) are now: %x \\n"), GetFileAttributes(sPath.c\_str()));

\_tprintf(\_T("\\nExplicitly set FILE\_ATTRIBUTE\_TEMPORARY attribute...\\n"));
SetFileAttributes(sPath.c\_str(), FILE\_ATTRIBUTE\_TEMPORARY);
// this line ALWAYS prints: 100 (code for FILE\_ATTRIBUTE\_TEMPORARY)
\_tprintf(\_T("\\nFile attr

Luc Pattyn

Hi, it has been a long time I used it, but here is what I remember: 1. CreateFile creates a file, opens it and returns a handle to it; 2. when you are done, you must call CloseHandle; 3. file data and metadata (such as last write time, and the flags) is only guaranteed to be committed to disk when the file got closed. Issue 3 is what allows different file systems to behave slightly differently. Also I don't expect FindFile to keep track exactly of what is happening with open files. Hence I suggest you set the files the way you want them, make sure they are all closed, and only then observe in any way you see fit, including enumeration through FindFile. :)

Luc Pattyn [Forum Guidelines] [My Articles]

Love, happiness and fewer bugs for 2009!

Loreia

Thank you for your answer, I guess I was to hasty to post my question (and the fact that I went to bed at 4:30 AM last night certainly didn't help). I forgot to say that I removed all error checking and file handles because it did not affect my problem, and I wanted a simplest/shortest possible test. The same thing happens even if file handle is closed (I even tried calling ::Sleep(5000) right after CloseHandle, just to be sure (I was also desperate :-)). Basically, my Unit test goes through a loop of all possible file attributes, it creates a new file, sets one Attribute and checks if my enumeration class finds the file. If it does, test is successful, and loops goes to test another Attribute. And all is fine until FILE_ATTRIBUTE_TEMPORARY is tested, and only if I start tests on my USB drive (which is FAT32 based), on my hard drive test works fine. You had a very good idea, but unfortunately that is not it :-( Best regards, loreia

Luc Pattyn

Fair enough. This page on CreateFile[^] holfd the following sentence: "Specifying the FILE_ATTRIBUTE_TEMPORARY attribute causes file systems to avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data. Although it doesn't directly control data caching in the same way as the previously mentioned flags, the FILE_ATTRIBUTE_TEMPORARY attribute does tell the system to hold as much as possible in the system cache without writing and therefore may be of concern for certain applications." which seems to hint temporary files may be treated in a very lazy way; maybe you shouldn't really worry too much if the only condition under which your tests fail is with the TEMP bit set. :)

Luc Pattyn [Forum Guidelines] [My Articles]

Love, happiness and fewer bugs for 2009!

Loreia

Thanks again :-D I've read that sentence like 300 times in last few days, and I must admit that I don't fully understand it. OK, English is my second language (third in fact) and I may be missing something, but this is how I see it: When application sets FILE_ATTRIBUTE_TEMPORARY attribute this information *should* be written somewhere in file system, and GetFileAttributes and FindFirstFile *should* read the same thing from file system. I don't understand why these two functions report different Attributes on FAT. Now, I can implement my class in such a way that when searching for FILE_ATTRIBUTE_TEMPORARY attribute, my class would additionally call GetFileAttributes within FindNextFile block. But I find this workaround a bit clumsy, and I was hoping to avoid it. Best regards loreia

Luc Pattyn

Well, one way one could interpret the sentence is: a temporary file is private, it belongs to the process and will not survive the process, so once you decide to make a file temporary, there is no need for further file operations as long as it keeps holding the data for the owning process (so the only thing it may have to write to disk is data that exceeds the cache capacity). Once the file gets closed, it should be deleted (not sure by whom: the OS or the app, probably the OS, maybe only when the app exits, but then your tests have finished too). (I know you can disagree on all that, the file system should always tell the truth, etc) You currently create the file as a regular file (despite your comment in the source), and later set the TEMP flag; so the file gets created on disk as a regular file, and once you turn it into a TEMP file, it may or may not reflect that on disk. This is a little experiment you could do: create file, write data, make temp, show current time, wait 10 seconds, write more data, close file, now use Explorer to see modified datetime: did it update after you turned it into temp? if not, it supports the "TEMP means no valid metadata" hypothesis. And maybe, just maybe, if you create it the start as a TEMP file right away, it will show that everywhere, and later on you could clear the TEMP flag and all would be good... :)

Luc Pattyn [Forum Guidelines] [My Articles]

Love, happiness and fewer bugs for 2009!

Loreia

Thanks, you last suggestion helped me better understand how TEMP files work. When I run test program on my hard drive, "test_file.txt" is not created at all, Explorer simply doesn't show it. I can see file only when I run program on my USB drive. After realizing that, I re-read this sentence from MSDN: "Specifying the FILE_ATTRIBUTE_TEMPORARY attribute causes file systems to avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data." So, TEMP file is held in memory as long enough cache memory is available. And after seeing that file is NOT deleted on my USB stick, I can verify that application is the one responsible for deleting file (unless FILE_FLAG_DELETE_ON_CLOSE is set in CreateFile) Now, at least I fully understand what MSDN documentation says about FILE_ATTRIBUTE_TEMPORARY :) . Off course, my problem still remains. GetFileAttributes and FindFirstFile *should* see the same set of file attributes, and should report the same DWORD value. And BTW, you really have a "hawk eye", I meant to call CreateFile with FILE_ATTRIBUTE_TEMPORARY parameter but I made a mistake when copied source code here (in original that parameter is "*iter"). But it doesn't change much, no matter how I create file, once I set TEMP attribute, FindFirstFile chokes on it(on FAT32).

Luc Pattyn

To complicate matters, I think removable devices such as USB sticks, work or may work slightly differently from regular disks; you (sometimes?) can set a device attribute that makes it safe to remove them at all times, which tells me they can only cache on read, not on write; if you don't, they may operate faster (on writes that is) but you officially need to "safely remove hardware" telling Windows it should flush the cache to the device. I don't always find how to control this feature though, it should be on one of the Device Property tabs but sometimes seems missing; I use both XP and Vista, and it may depend on the specific USB stick (some tend to install and use their own driver). As a last remark: USB sticks really contain a microcontroller and a flash EEPROM, so there is software involved there too; they get to execute FAT commands, so your TEMP problem could even be stick-specific! If you ever discover the finer details on the TEMP matter, feel free to write and publish an article on the subject! :)

Luc Pattyn [Forum Guidelines] [My Articles]

Love, happiness and fewer bugs for 2009!

Loreia

This post brought smile to my face, I too am an electronics engineer and I program microcontrollers in my spare time (Microchip's PIC family) and my next project (if I ever find time to do it, that is) is to implement procedure to save data (collected from various sensors) on USB stick. Back to topic... I think Windows simply flushes data on USB stick because they are removable, and that is the only reason I can see the file with TEMP attribute. So it is probably not specific to my UBS stick. But I was thinking along the same lines, so I tested on three different USB stick and two computers, and then I created a small FAT32 (and FAT) partition on my hard drive and tested it there too. I got the same results everywhere, FindFirstFile identifies TEMP file only on NTFS partitions :(

Lost User

loreia wrote:

FindFirstFile identifies TEMP file only on NTFS partitions :(

FindFirstFile is returning the FAT attributes from reading the disk because it was designed to also work with 16-bit programs. FILE_ATTRIBUTE_TEMPORARY is an attribute which does not exist in the FAT specification[^] but does in NTFS. Would you expect to see FILE_ATTRIBUTE_ENCRYPTED on a FAT partition? No because only NTFS supports encryption. GetFileAttributes is different and checks internel objects such as the cache manager. There are *many* attributes in the GetFileAttributes Function[^] which are not present on a FAT based disk. Best Wishes, -David Delaune

Loreia

David, thank you a lot for your answer. I know from experience that FAT32 doesn't support encryption, so answer to your question is: no. LOL So, TEMP attribute is NTFS specific, and it is not supported on FAT32. Thanks for clarifying that, but what I don't understand is why SetFileAttributes returns TRUE when setting TEMP attribute to a file on FAT32. It SetFileAttributes function returned FALSE I wouldn't be in this mess in the first place. My understanding is: if FAT32 doesn't support TEMP files, than SetFileAttributes should fail to set TEMP attribute. I was wrong in presuming that "error" (for the lack of better description) was in FindFirstFile. But in fact FindFirstFile correctly reports that TEMP attribute is not set on FAT32 (as it cannot be set on FAT32). And the real problem is in SetFileAttributes/GetFileAttributes functions that *claim* that file has a TEMP attribute set even on FAT32 (and this is in direct contradiction to FAT32 File System Specification that you posted a link for). I you know why are SetFileAttributes/GetFileAttributes acting in this way, your explanation would help me a lot. Best regards loreia

Luc Pattyn

OK, that seems to explain it all. Thanks. :)

Luc Pattyn [Forum Guidelines] [My Articles]

Love, happiness and fewer bugs for 2009!

Lost User

loreia wrote:

My understanding is: if FAT32 doesn't support TEMP files, than SetFileAttributes should fail to set TEMP attribute.

You need to seperate FAT from the operating system within your mind. When you call SetFileAttributes and set the file as FILE_ATTRIBUTE_TEMPORARY what happens internally is that the NT kernel calls CcInitializeCacheMap[^] and CcSetAdditionalCacheAttributes[^] which disables write-behind and essentially enables a "lazy write" which means that the file may or may not be written to disk depending on an internal algorithm based on critera such as i/o priority, filesize, media type ect... The difference is that FAT32 does not have anywhere to store this extra information as it is an older technology. NTFS on the other hand was designed to store additional file attributes for both now and in the future as it can be extended. FindFirstFile[^] has many other problematic surprises[^]. Some of which stem from supporting 16 bit applications. GetFileAttributes is not lying to you when it says the file is FILE_ATTRIBUTE_TEMPORARY. Best Wishes, -David Delaune

Loreia

What a great explanation, Now I fully understand what happened in my program. Thanks a lot, I owe you one :) Best regards, loreia

David Crow

GetFileAttributes() is used to retrieve a set of FAT-style attributes. See if GetFileAttributesEx() behaves any differently.

"Love people and use things, not love things and use people." - Unknown

"The brick walls are there for a reason...to stop the people who don't want it badly enough." - Randy Pausch

Loreia

Hi David, thanks for your suggestion. GetFileAttributesEx() won't help me. I had problems understanding why ::FindFirstFile and GetFileAttributes() report different set of file attributes on FAT32. And now I get it, TEMP files are not flushed to HDD, and therefore limitations of FAT can be see with ::FindFirstFile and not with GetFileAttributes(). I made a small test in which I created a file on USB stick (with FAT32 file system), then added a TEMP attribute to it and: a) read file attributes right after creating file b) then I safely removed USB drive (forcing windows to flush file to FAT) c) re-read file attributes, now both ::FindFirstFile and GetFileAttributes() reported correct values Now I can safely continue developing my enumeration class. One more time, a BIG thank you to all three of you for your help, suggestions and explanations. Best regards loreia