I thought .NET was supposed to make things easier, if anything, than unmanaged code.
-
Yes, though it was added to .NET after my initial attempt at using mem mapped files from C#. Besides all that is is a wrapper like the one i had written years ago. It doesn't change the basic problem which is: var foo = new int[1000000]; //backed by disk, paged automatically, in C/C++ it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
honey the codewitch wrote:
it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
Yes, and that's by design.
-
I know this is a very special case but still i ran headlong into it. The easiest way to implement a B+ tree on disk is using a memory mapped file. I think this is what SQL Server does, but don't quote me. However, the only way you can access memory mapped files in C# is through .NET interop which makes it useless. Because one of the points of a memory mapped file is that you can do memory allocations that are backed by disk. There's no way in hell .NET can give you that in its current incarnation, even if one were to write a custom host, because of the way a GC system works. What I'd like var foo = new int[1000000]; // backed by disk, paged automatically What I'd have to do. somepointer = VirtualAlloc(...) Write(somepointer, data) etc etc basically it works like file i/o which defeats essentially the whole purpose. =(
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
Environments designated to making simple things simple tend to make complicated things more complicated. Or maybe not, in the case of .NET, there's the unsafe keyword which allows you to work on raw pointers. However, that indeed doesn't help much with disk stuff. See it this way, C# is the polar opposite of C here. In C, complex things and simple things work about the same way at the cost of simple things being ridiculously complicated. I remember a project of mine where I combined the strengths of two worlds, I did my low-level in C (technically C++, but I had raw pointers at work), exposed a C interface and did my business logic in C#. Such a mixed design is of course too easy to get wrong, but I think I've managed to make it make sense.
-
Environments designated to making simple things simple tend to make complicated things more complicated. Or maybe not, in the case of .NET, there's the unsafe keyword which allows you to work on raw pointers. However, that indeed doesn't help much with disk stuff. See it this way, C# is the polar opposite of C here. In C, complex things and simple things work about the same way at the cost of simple things being ridiculously complicated. I remember a project of mine where I combined the strengths of two worlds, I did my low-level in C (technically C++, but I had raw pointers at work), exposed a C interface and did my business logic in C#. Such a mixed design is of course too easy to get wrong, but I think I've managed to make it make sense.
I completely agree, and I've done similar. It's just that here, it's not really my goal to make something tied to the windows platform specifically, and pretty much no matter the route i take as soon as I'm relying on that mixed managed and unmanaged code I'm also inherently tying it to a platform. Sure I can cross compile the unmanaged code, but even then, it's one package per platform - not what i want.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
honey the codewitch wrote:
it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
Yes, and that's by design.
I'm aware of that. I worked on the Visual Studio team at microsoft. I know a little about .NET
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
honey the codewitch wrote:
I'm fine with looking at complexity as for example O(log N), and that's fine, without worrying about the .NET overhead on top of the base functionality.
Doing a full u-turn, are we? This was your response to another person on this thread:
honey the codewitch wrote:
but it's not efficient and doesn't solve the basic issue, which is persistent, rapidly searchable storage.
No I'm not, because in this case, there's an order of magnitude of difference in performance between using the windows memory paging system, and seeking around a disk file all the time. It changes the complexity. I'm not concerned about 10% improvements.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I completely agree, and I've done similar. It's just that here, it's not really my goal to make something tied to the windows platform specifically, and pretty much no matter the route i take as soon as I'm relying on that mixed managed and unmanaged code I'm also inherently tying it to a platform. Sure I can cross compile the unmanaged code, but even then, it's one package per platform - not what i want.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
Don't quote me, but I am rather sure that you can't do that in a platform-agnostic manner. Going with P/Invoke or a low-level library in C will require platform ties as the C API doesn't know of the concept you need (hey, the C standard doesn't even have a concept of alignment). From what I know, the C++ standard library doesn't have this concept either and the moment your standard libraries don't provide something, you have to work with IFDEFs for platforms. Be it with .NET, C++ or any other language.
-
Don't quote me, but I am rather sure that you can't do that in a platform-agnostic manner. Going with P/Invoke or a low-level library in C will require platform ties as the C API doesn't know of the concept you need (hey, the C standard doesn't even have a concept of alignment). From what I know, the C++ standard library doesn't have this concept either and the moment your standard libraries don't provide something, you have to work with IFDEFs for platforms. Be it with .NET, C++ or any other language.
yeah but i think you can do p/invoke in mono to access ELF binaries (or whatever they'll called these days), etc. worst case I'd use #if in C# to plug in the right p/invoke attributes for the platform So like i said, it might be that I could cross compile the unmanaged bits, but in any case it doesn't give me what i want here.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
yeah but i think you can do p/invoke in mono to access ELF binaries (or whatever they'll called these days), etc. worst case I'd use #if in C# to plug in the right p/invoke attributes for the platform So like i said, it might be that I could cross compile the unmanaged bits, but in any case it doesn't give me what i want here.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
As I was saying, I am darn sure that it's not realistic to get this rather special use case of yours in a platform-agnostic manner. You don't need to cross-compile though. Of course can you P/Invoke in Mono and Mono has it's own share of IFDEFs to tell platforms apart. So compile ONE executable to run on Mono (or .NET Core which also has ways to tell platforms apart) but with your NativeMethods class being full of IFDEFs.
-
As I was saying, I am darn sure that it's not realistic to get this rather special use case of yours in a platform-agnostic manner. You don't need to cross-compile though. Of course can you P/Invoke in Mono and Mono has it's own share of IFDEFs to tell platforms apart. So compile ONE executable to run on Mono (or .NET Core which also has ways to tell platforms apart) but with your NativeMethods class being full of IFDEFs.
I'm pretty sure I'd need to cross compile the C code as Virtual memory management is completely different under windows than it is in linux for example. I don't think i can even map arbitrary files to process address space in linux but i could be wrong.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I'm pretty sure I'd need to cross compile the C code as Virtual memory management is completely different under windows than it is in linux for example. I don't think i can even map arbitrary files to process address space in linux but i could be wrong.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
You would have to cross-compile if you go the route of compiling a separate library (in, let's say, C) for this stuff. But you can also go with P/Invoke which you "cross compile" via runtime checks. .NET Core supports this scenario, I haven't checked on Mono (yet). https://stackoverflow.com/questions/38790802/determine-operating-system-in-net-core
-
You would have to cross-compile if you go the route of compiling a separate library (in, let's say, C) for this stuff. But you can also go with P/Invoke which you "cross compile" via runtime checks. .NET Core supports this scenario, I haven't checked on Mono (yet). https://stackoverflow.com/questions/38790802/determine-operating-system-in-net-core
yeah in the general sense, but in this case, i'd need two binaries per platform (one managed, one unmanaged) probably. #ifs require compilation of course, in C# as well as C
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
yeah in the general sense, but in this case, i'd need two binaries per platform (one managed, one unmanaged) probably. #ifs require compilation of course, in C# as well as C
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
Why do you insist on an unmanaged binary? Why don't you want to do your unmanaged ABI calls with P/Invoke?
-
Why do you insist on an unmanaged binary? Why don't you want to do your unmanaged ABI calls with P/Invoke?
Because it's impossible to map a file to process address space in C#. You can only marshal calls to the vmem system which makes it exactly like reading and writing a file instead of reading and writing a pointer. Defeating the primary reason I'd use it. what I want (and can't have)
var foo = new int[100000]; // backed by disk automatically
and i don't want an unmanaged binary. I'm speaking in the hypothetical. I will not use vmem in this project because of .NET limitations.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
Because it's impossible to map a file to process address space in C#. You can only marshal calls to the vmem system which makes it exactly like reading and writing a file instead of reading and writing a pointer. Defeating the primary reason I'd use it. what I want (and can't have)
var foo = new int[100000]; // backed by disk automatically
and i don't want an unmanaged binary. I'm speaking in the hypothetical. I will not use vmem in this project because of .NET limitations.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
Write an "array" class which looks like an array from the outside but works with ABI calls internally. Using container classes is often recommended over using plain arrays in C# anyway.
-
I'm aware of that. I worked on the Visual Studio team at microsoft. I know a little about .NET
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
I'm not sure why you feel the need to tell me that you worked at Microsoft. If you were trying to imply that it adds credibility to what you say, I've go to tell you that I've met some real nimrods who came from Microsoft to those MVP summits I used to go to. I'm sure that a number of exceptionally talented people work or have worked at Microsoft, but there are also these nimrods (who thought they were exceptionally talented). A random sample could belong to either camp. Because you had to say that, I had to say this.
-
Yes, though it was added to .NET after my initial attempt at using mem mapped files from C#. Besides all that is is a wrapper like the one i had written years ago. It doesn't change the basic problem which is: var foo = new int[1000000]; //backed by disk, paged automatically, in C/C++ it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
So you can't pin?: whatabout [fixed Statement - C# Reference | Microsoft Docs](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/fixed-statement) So I think it's time to use the latest .NET Version and have a look into what has changed...
-
So you can't pin?: whatabout [fixed Statement - C# Reference | Microsoft Docs](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/fixed-statement) So I think it's time to use the latest .NET Version and have a look into what has changed...
pinning does not solve this problem. I don't need a random pointer .NET produced from the GC heap. The only thing i could do with that is throw it away. It's useless for this
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I'm not sure why you feel the need to tell me that you worked at Microsoft. If you were trying to imply that it adds credibility to what you say, I've go to tell you that I've met some real nimrods who came from Microsoft to those MVP summits I used to go to. I'm sure that a number of exceptionally talented people work or have worked at Microsoft, but there are also these nimrods (who thought they were exceptionally talented). A random sample could belong to either camp. Because you had to say that, I had to say this.
No I'm telling you i don't need your lectures about the very basics of how .NET works. I don't care if you find me credible because your opinion doesn't matter to me. If that were to change, we'll talk.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
Write an "array" class which looks like an array from the outside but works with ABI calls internally. Using container classes is often recommended over using plain arrays in C# anyway.
it has been done. It's called List/IList and it doesn't help me. It does zero to reduce the complexity of what I'm trying to do. I don't actually need an int array. I need a B+tree. Try wrapping that in something fun and you'll have done my job. :)
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
it has been done. It's called List/IList and it doesn't help me. It does zero to reduce the complexity of what I'm trying to do. I don't actually need an int array. I need a B+tree. Try wrapping that in something fun and you'll have done my job. :)
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
http://csharptest.net/projects/bplustree From the description "BPlusTree is a implementation of the generic IDictionary interface backed by a disk-based B+Tree".