pasztorpisti wrote:

Today its not the only choice

So what language would you choose today to write a new OS in ?

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

I'd suggest starting with LZ77: http://en.wikipedia.org/wiki/LZ78[^] - It is one of the earlier most widespread lossless algo. - It has been the base for many other algo. - The compress and decompress code can be done in less than 100 lines of code. It might take a while, but get a copy of the code and study it until you understand what every single line is doing. Then try to modify it and understand how all the parameters are inter-related.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Sadly it's first-come-first-use. That's why you see everyone with status lining-up for priority boarding - to make sure they get a spot for their carry-on.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Here's some old code that may help:

double Vk05Pi = 1.5707963267948966192313216916398;

#define MTH_ASM_IS0(LABEL_IS0) __asm \
{ \
__asm ftst \
__asm fnstsw ax \
__asm test ah, 40h \
__asm jne LABEL_IS0 \
}
#define MTH_ASM_IS1(LABEL_IS1) __asm \
{ \
__asm fld1 \
__asm fcomp \
__asm fnstsw ax \
__asm test ah, 40h \
__asm jne LABEL_IS1 \
}

// arcsine: atan( sqrt( A*A / (1-A*A) ) )
//
double FkASinR( double A )
{
if( A > 1 ) return(0);
#ifndef MTH_ASM_USE
A *= A;
if( FkIs1(A, DBL_MIN) ) return(Vk05Pi);
return( atan( sqrt(A/(1-A)) ) );
#else
__asm {
fld A
fmul A
MTH_ASM_IS1(isone)
fld st(0)
fld1
fsubr // 1 - A*A
fdiv // A*A / (1-A*A)
fsqrt
fld1
fpatan
jmp end
isone:
fstp A
fld Vk05Pi
end:
fstp A
}
return(A);
#endif
}

// arccosine: atan( sqrt( (1-A*A) / (A*A) ) )
//
double FkACosR( double A )
{
#ifndef MTH_ASM_USE
A *= A;
if( FkIs0(A, DBL_MIN) ) return(Vk05Pi);
return( atan( sqrt((1-A)/A) ) );
#else
__asm {
fld A
fmul A
MTH_ASM_IS0(iszero)
fld st(0)
fld1
fsubr // 1 - A*A
fdivr // (1-A*A) / A*A
fsqrt
fld1
fpatan
jmp end
iszero:
fstp A
fld Vk05Pi
end:
fstp A
}
return(A);
#endif
}

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Good summary. As you mention, the fast code option can can cause a cache miss. Additionally, earlier versions at least, could result in enough bloat to create extra pages. Related code, that under 'small' sits in the same page, could be split into different pages under 'fast'. This can result in more page faults. The performance impact due to page faults can exceed the speed improvements made from 'fast'. So, based on past experience, and that these days the difference in optimization strategies is generally minor, I still use 'small'. I haven't experienced the compiler ignoring my inline settings yet, have to check that, thanks. The memset issue ... I ran into the same things years ago and found the following: http://msdn2.microsoft.com/en-us/library/ms972826.aspx[^]

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Optimization: Minimize Size Inline Function Expansion: Only those explicitly marked Enable Intrinsic Functions: Yes Favor Size or Speed: Favor small code Omit Frame Pointers: No Enable Fiber-safe Optimizations: Yes Whole Program Optimization: Yes Favour size or speed is a result of previous issues with speed optimizations causing problems. Also, smaller code can result in less page swapping, which swamps any benefits from 'faster' code. I had also done a compare of a number of apps compiled both ways. I was rarely able to find significant differences in speed, but often in size - the size opt had more effect. Inline expansion 'Any' has caused significant module bloating for me before. This usually happens with template code. I like this part to be deterministic. Fiber/whole optimizations, never had a reason to turn them off.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Do you mean like writing your own gina.dll (or the replacement process post-vista) See:http://msdn.microsoft.com/en-ca/magazine/cc163489.aspx[^]

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

It sounds like there are more issue than can be easily solved via a forum post. I'd suggest you look at some of the IOCP posts here. In particular, Len Holgate wrote a good series: A reusable, high performance, socket server class - Part 1[^] A reusable, high performance, socket server class - Part 2[^] A reusable, high performance, socket server class - Part 3[^] Handling multiple pending socket read and write operations[^]

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

1. Agree. 2. Agree. Or maybe ... So on other end ... Which is how you can use an IOCP to be a job/task manager like you could also create using the Win32^ Job Object / Thread Pool functions. But I am thinking now ... wsarecv(OL) will not block your posting thread. When GetQueuedCompletionStatus returns the results of the wsarecv, the specified buffer will be filled with the received data ... of course this means the buffer specified in OL must persist from time wsarecv(OL) is called until you are done with it after GetQueuedCompletionStatus. If you did not get all data (i.e. only posted recv to get a header) then either post another async recv, or (depending on how you handle things) you may be able to call a sync recv to get rest of data (e.g. payload).

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Depending on your needs, the following may be sufficient: CryptProtectData() CryptUnprotectData() They encrypt using either machine or user specific key and help you avoid having to mess with the Crypto API.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

One framework I created that used ICOP had the following classes:

IoObj - Base class for I/O
IoHnd - IFS handle based
File - OS file access
NetSocket - Network socket
NetIpClnt/NetIpSvc, NetTCP*, NetUdp*, NetIrda*, NetBth*, ...
NetFPT*, NetNNTP*, ...

WSAOVERLAPPED
IoOL - Base overlapped; adds flags, timestamps, type, control, buffer
NetIoOLAddr - specialied for sendto/recfrom, contains remote address
NetIoOLAccept - specialized for handling AcceptEx

IoCP - I/O completion port

The IoCP had the main posting functions and GetQueuedCompletionStatus loop. To post a msg on an IoCP you'd pass an IoObj* and IoOL*. The IoObj* would be the key, the IoOL* the overlapped (per call). The IoCP GetQueuedCompletionStatus loop would call (something like) IoObj->Msg(IoOL*). With the above you can use one or more IoCP instances to handle any combination of files, sockets, or other IoObj based objects (e.g. jobs/tasks). I found both having key and OL convenient. Without an explicit key param you'd have to add it as state to your extended OL ... which you may want anyways.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

The fact that it's open source alleviated any concerns about installing as admin ... or do you mean concerns for deploying to your customers ... yes, I could see that being an issue. I don't think it uses much when idle ... on my machine the mounter service is using 928KB, 71 handles, and 4 threads with no FS mounted. I am more concerned about the fact that it's open source means I have no idea how rigorous the testing has been on the code; however, it seems to be used by others who are providing commercial products that use it, so it can't be that bad. For me it's a good starting point. I can focus on the FS part of the code. Later if I find bugs in dokan, want more performance, want more control, then I can look at writing the kernal part myself.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

As far as I can tell, you have to write code to handle file CRUD requests no matter how you do it ... six of one, half-dozen of the other. So far the Dokan still looks easiest, it looks like the main dev points are: 1. Write thread-safe functions specified in DOKAN_OPERATIONS e.g. CreateFile, ReadFile, ... 2. Pass an instance of DOKAN_OPERATIONS with pointers set to your functions, and DOKAN_OPTIONS with info on how you want to mount your FS, to the driver via DokanMain. This will also mount your FS. 3. Call DokanUnmount when done with FS. I still have to work through the nuances of mounting/unmounting via: my app, dokanctl, dokan mounter, ... Notes: - Although Dokan comes with both x86 and x64 drivers, it looks like the user dll is x86. This means your app must be compiled for x86. - The Dokan API does support files > 2GB. - Test your app FS in a virtual machine, if you made a mistake it can totally hose your system.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

I believe I understand, it's similar to what I'm looking to do - be able to mount new drives that expose filesystems stored in a database. I think Doken will be the easiest and most flexible solution. If you look through the source of the ones implimented on their site you'll see it's not too bad.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

I've been looking into a similar thing. The best I've come up with so far is Doken[^]. They have developed the kernel driver and expose a user API to develop your own filesystem apps. Others have already developed both ram and file based filesystems[^] for it. The minifilter option provides filesystem access for _all_ applications, shell extensions are only for accessing files through, well, the shell e.g. explorer. It comes down to how will the client need to interact with your filesystem ?

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Don't be a tease, where is the setting ?

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Huh, intersting, thanks.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Google's new _forced_ location aware search results - FAIL! I'm often searching for (hardware) reviews and such. I don't want the result list biased for Toronto/Ontario/Canada. I want results from everywhere in the world, even ones in a different language. Seriously, maybe I'm odd, but here's how I shop: 1. Search worldwide for all products meeting requirements. 2. Review worldwide the top candidate products. 3. Locate best place to buy chosen product - does not have to be local. At least Bing still gives the option to get all results. My top 3 current peeves where companies think they are doing the consumer a favour, but really end up making life more difficult: 1. Rogers Cable (Canada): 'new' menu that pops up instead of guide. Doubles the time to get to the program guide, which is what I want to do 100% of the time and quite frequently. 2. Google forced location aware search results. 3. MS following Apple and tethering Win7 Phone installs to marketplace.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

I've avoided developing for the i* for specifically this reason. I was going to get a Win Phone 7, but it looks like I'm going to go to Android. Stupid mistake MS.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack

Options in increasing order of complexity/robustness: - Those mentioned, uncontrolled access by both apps/threads. - LockFileEx() / UnlockFileEx() - OS level read/write locking, critical if multiple threads will be appending. - Transactional NTFS - TxF API to perform transacted reads/writes, critical if multiple writes to different data sources (file, registry, database) may need to be rolled back and readers need consistant view across all.

...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack